Lpad webgui broken, "TypeError: cannot pickle '_thread.lock' object"

FireWorks
1

The lpad webgui command no longer works for me. It’s been a while since I used it and I don’t know what changed.

$ lpad webgui
Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/borealis/bin/lpad", line 8, in <module>
    sys.exit(lpad())
  File "/usr/local/var/pyenv/versions/3.8.3/envs/borealis/lib/python3.8/site-packages/fireworks/scripts/lpad_run.py", line 1209, in lpad
    args.func(args)
  File "/usr/local/var/pyenv/versions/3.8.3/envs/borealis/lib/python3.8/site-packages/fireworks/scripts/lpad_run.py", line 561, in webgui
    p1.start()
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/var/pyenv/versions/3.8.3/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

This happens in FireWorks==1.9.5 on both Python 3.8.0 and Python 3.8.3. lpad reset doesn’t change this.

lpad get_fws works fine. So does running workflows with this my_launchpad.yaml file and this MongoDB server.

I could try running the lpad command in the PyCharm debugger, but in the stack trace it looks like the webgui() function calls p1.start() on a Process object that contains a Thread lock.

I tried creating a fresh Python 3.8.3 virtualenv and running pip install FireWorks>=1.9.5. pip list prints:

Package          Version
---------------- -------
click            7.1.2
FireWorks        1.9.5
Flask            1.1.2
flask-paginate   0.7.0
gunicorn         20.0.4
itsdangerous     1.1.0
Jinja2           2.11.2
MarkupSafe       1.1.1
monty            3.0.4
pip              20.1.1
pymongo          3.10.1
python-dateutil  2.8.1
ruamel.yaml      0.16.10
ruamel.yaml.clib 0.2.0
setuptools       49.2.0
six              1.15.0
tabulate         0.8.7
tqdm             4.47.0
Werkzeug         1.0.1
wheel            0.34.2
2

I narrowed this down: The exception occurs in Python 3.8.0 and 3.8.3; not in Python 3.7.7.

Steps to reproduce:

pyenv install 3.8.0
pyenv virtualenv 3.8.0 fireworks-3.8 && pyenv local fireworks-3.8
pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel
pip install FireWorks==1.9.5
pyenv rehash
lpad webgui

This assumes there’s a my_launchpad.yaml file with info to connect to a LaunchPad server.

3

This usually happens when trying to pickle a multiprocessing object

4

Pickling a _thread.lock does sound risky but nothing in What’s New In Python 3.8 suggests a relevant change from 3.7.

Is webgui even trying to pickle an object containing a _thread.lock?

This Stack Overflow answer suggests it’s a bug in Python 3.8.

Python 3.8.5 is now available via pyenv, and the problem is still there, at least on macOS.

5

Does the same problem occur with FWS 1.9.6 (just released). Some problems in gunicorn were fixed

6

Alas, upgrading to FireWorks==1.9.6 does not fix the problem, nor does rebuilding this virtualenv from scratch using it.

FWIW, these virtualenvs already had gunicorn==20.0.4.

7

I remember having this issue in the past and unfortunately didn’t go digging for its root cause. So while I don’t have a real solution to offer, I recall that running the web GUI in server mode (i.e. lpad webgui -s) worked around the pickling issue. HTH

8

Thanks, @tschaume! lpad webgui -s succeeds inside the minimal-test-case virtualenv, but in my regular project virtualenv it says “Gunicorn is required” even though it’s already installed!

$ lpad webgui -s
Gunicorn is required for server mode. Install using `pip install gunicorn`.
$ pip freeze | grep gunicorn
gunicorn==20.0.4
$ pip install gunicorn
Requirement already satisfied: gunicorn in /usr/local/var/pyenv/versions/3.8.3/envs/wcEcoli3/lib/python3.8/site-packages (20.0.4)
Requirement already satisfied: setuptools>=3.0 in /usr/local/var/pyenv/versions/3.8.3/envs/wcEcoli3/lib/python3.8/site-packages (from gunicorn) (49.2.0)
$ lpad webgui -s
Gunicorn is required for server mode. Install using `pip install gunicorn`.

Aha! After upgrading that virtualenv to FireWorks==1.9.6, lpad webgui -s works.

That’s a sufficient workaround for us.

It turns out this test in lpad_run.py:

        try:
            from fireworks.flask_site.gunicorn import (
                StandaloneApplication, number_of_workers)
        except ImportError:
            import sys
            sys.exit("Gunicorn is required for server mode. "
                     "Install using `pip install gunicorn`.")

was misled by the ModuleNotFoundError: No module named 'gunicorn.six' problem.

1 Like