Lpad webgui broken, "TypeError: cannot pickle '_thread.lock' object"

The lpad webgui command no longer works for me. It’s been a while since I used it and I don’t know what changed.

$ lpad webgui
Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/borealis/bin/lpad", line 8, in <module>
  File "/usr/local/var/pyenv/versions/3.8.3/envs/borealis/lib/python3.8/site-packages/fireworks/scripts/lpad_run.py", line 1209, in lpad
  File "/usr/local/var/pyenv/versions/3.8.3/envs/borealis/lib/python3.8/site-packages/fireworks/scripts/lpad_run.py", line 561, in webgui
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

This happens in FireWorks==1.9.5 on both Python 3.8.0 and Python 3.8.3. lpad reset doesn’t change this.

lpad get_fws works fine. So does running workflows with this my_launchpad.yaml file and this MongoDB server.

I could try running the lpad command in the PyCharm debugger, but in the stack trace it looks like the webgui() function calls p1.start() on a Process object that contains a Thread lock.

I tried creating a fresh Python 3.8.3 virtualenv and running pip install FireWorks>=1.9.5. pip list prints:

Package          Version
---------------- -------
click            7.1.2
FireWorks        1.9.5
Flask            1.1.2
flask-paginate   0.7.0
gunicorn         20.0.4
itsdangerous     1.1.0
Jinja2           2.11.2
MarkupSafe       1.1.1
monty            3.0.4
pip              20.1.1
pymongo          3.10.1
python-dateutil  2.8.1
ruamel.yaml      0.16.10
ruamel.yaml.clib 0.2.0
setuptools       49.2.0
six              1.15.0
tabulate         0.8.7
tqdm             4.47.0
Werkzeug         1.0.1
wheel            0.34.2

I narrowed this down: The exception occurs in Python 3.8.0 and 3.8.3; not in Python 3.7.7.

Steps to reproduce:

pyenv install 3.8.0
pyenv virtualenv 3.8.0 fireworks-3.8 && pyenv local fireworks-3.8
pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel
pip install FireWorks==1.9.5
pyenv rehash
lpad webgui

This assumes there’s a my_launchpad.yaml file with info to connect to a LaunchPad server.

This usually happens when trying to pickle a multiprocessing object

Pickling a _thread.lock does sound risky but nothing in What’s New In Python 3.8 suggests a relevant change from 3.7.

Is webgui even trying to pickle an object containing a _thread.lock?

This Stack Overflow answer suggests it’s a bug in Python 3.8.

Python 3.8.5 is now available via pyenv, and the problem is still there, at least on macOS.

Does the same problem occur with FWS 1.9.6 (just released). Some problems in gunicorn were fixed

Alas, upgrading to FireWorks==1.9.6 does not fix the problem, nor does rebuilding this virtualenv from scratch using it.

FWIW, these virtualenvs already had gunicorn==20.0.4.

I remember having this issue in the past and unfortunately didn’t go digging for its root cause. So while I don’t have a real solution to offer, I recall that running the web GUI in server mode (i.e. lpad webgui -s) worked around the pickling issue. HTH

Thanks, @tschaume! lpad webgui -s succeeds inside the minimal-test-case virtualenv, but in my regular project virtualenv it says “Gunicorn is required” even though it’s already installed!

$ lpad webgui -s
Gunicorn is required for server mode. Install using `pip install gunicorn`.
$ pip freeze | grep gunicorn
$ pip install gunicorn
Requirement already satisfied: gunicorn in /usr/local/var/pyenv/versions/3.8.3/envs/wcEcoli3/lib/python3.8/site-packages (20.0.4)
Requirement already satisfied: setuptools>=3.0 in /usr/local/var/pyenv/versions/3.8.3/envs/wcEcoli3/lib/python3.8/site-packages (from gunicorn) (49.2.0)
$ lpad webgui -s
Gunicorn is required for server mode. Install using `pip install gunicorn`.

Aha! After upgrading that virtualenv to FireWorks==1.9.6, lpad webgui -s works.

That’s a sufficient workaround for us.

It turns out this test in lpad_run.py:

            from fireworks.flask_site.gunicorn import (
                StandaloneApplication, number_of_workers)
        except ImportError:
            import sys
            sys.exit("Gunicorn is required for server mode. "
                     "Install using `pip install gunicorn`.")

was misled by the ModuleNotFoundError: No module named 'gunicorn.six' problem.

1 Like