Parallel processing on Windows 10

[agrav]

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

[Anubhav_Jain]

Hi Audun,

Unfortunately, I don’t have a Windows box and won’t be able to debug / fix this error. I’ll pass your ticket along to the person that wrote most of the multiprocessing to see if he has any thoughts.

Best,

Anubhav

···

On Monday, August 7, 2017 at 6:21:35 AM UTC-7, agrav wrote:

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

[agrav]

Ok, great!

Thanks for the quick reply.

Audun

mandag 7. august 2017 18.53.51 UTC+2 skrev Anubhav Jain følgende:

···

Hi Audun,

Unfortunately, I don’t have a Windows box and won’t be able to debug / fix this error. I’ll pass your ticket along to the person that wrote most of the multiprocessing to see if he has any thoughts.

Best,

Anubhav

On Monday, August 7, 2017 at 6:21:35 AM UTC-7, agrav wrote:

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

[Xiaohui_Qu]

I can’t reproduce error since I am not using Windows. I did some search on google. I agree that there is some difference between Linux and Windows.

“Windows doesn’t have Unix sockets. Instead, …”

http://stupidpythonideas.blogspot.com/2014/09/sockets-and-multiprocessing.html

A possible solution is discussed in this thread:

https://github.com/uqfoundation/pathos/issues/67

“Second item to note is that Windows works differently than non-windows systems. Much of the time, for the ProcessPool, Windows requires from pathos.helpers import freeze_support, then freeze_support(). It’s unnecessary for non-windows systems. Windows often requires freeze_support to be provided in main.”

···

On Monday, August 7, 2017 at 6:21:35 AM UTC-7, agrav wrote:

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

[Martin_B]

Hello,

I have the exact same problem (with the same error message) and I was wondering if anyone had found a solution to it ?
I tried to follow Xiaohui_Qu’s answer but I’m afraid it’s too techy for me, I didn’t quite understand what I was supposed to import and where.
I would also like to report that I had many problems with queue adapters on Windows, I suspect because Fireworks is expecting /bin/bash to be a thing on this OS, which of course it isn’t.
Will Fireworks be supported on Windows anytime soon ? Would you recommend any alternative that would work on Windows ? Switching to Linux isn’t quite possible in my case, the main reason being that the computers from the lab I’m working with are running on that OS and this decision isn’t mine to take (and the transition would certainly be a very long and difficult one, since they have been used for years by now).
Thank you in advance for your help.

[Martin_B]

Update : Using WSL 1 seems to be a good compromise. It doesn’t require installing a VM or a dual-boot, and it fixes rlaunch multi, but not queue adapters, apparently. Something with it not finding files like ‘qstat’ and ‘sbatch’, no idea what it means. No idea what’s the difference between each queue adapter and what anything means in there, by the way. It would be much appreciated if we could get some extra documentation on these.
Anyway, if anyone finds themselves in the same situation with rlaunch multi, I’d definitely recommend checking WSL out to see whether it can fix it for you. It is also very easy to setup the whole thing. Everything is detailed on Microsoft’s website.
PS : You don’t even have to reinstall MongoDB on WSL1, surprisingly. I only had to reinstall VSCode, pip, and Fireworks (with its prerequisites).

[HugoDGD]

Hello,

This error is due to windows not supporting the fork system call. Therefore, the children don’t have access to the environment of the parent etc. For this reason, the arguments of the processes are shared by pickling and unpickling them. In multi_launcher.py, the launchpad is shared via a lambda function which is not picklable. To solve this problem, you can replace the pickle library by dill import dill as pickle in multiprocessing or use multiprocess instead (fork of multiprocessing).

There is actually another problem that occurs after the first fix. This one is actually an error in how the managers are used. In this example (multiprocessing — Process-based parallelism — Python 3.11.4 documentation), you can see that the clients have to register an empty callable (get_queue) before connecting. Which is not done in Fireworks.

By fixing these two things, i can successfully use rlaunch multi and launch_multiprocess() on windows.

I hope this can help some people avoid having to read the multiprocessing documentation and source code for hours (like i did ).