The specific bug is that when unpickling a Firetask with required parameters, the instance parameter test in FiretaskMeta.call fails, fatally ending the unpickle operation.
The code is here:
The use case is in the context of Jupyter interactive workflows, where I want to have fine-grained controls and monitoring over each Firetask, without requiring that users of existing workflows rewrite everything according to a new spec. Each Firetask is being run by a service in a separate process from the rocket launcher.
Here is a simple example to replicate this:
import pickle
from fireworks import ScriptTask
task = ScriptTask.from_str(‘ls -l > listing.txt’)
message = pickle.dumps(task)
obj = pickle.loads(message)
This will produce an exception traceback similar to the following:
<details class='elided'>
<summary title='Show trimmed content'>···</summary>
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-a8ca0b9807d0> in <module>()
5 task = ScriptTask.from_str('ls -l > listing.txt')
6 message = pickle.dumps(task)
----> 7 obj = pickle.loads(message)
~/miniconda3/lib/python3.6/site-packages/fireworks/core/firework.py in __call__(cls, *args, **kwargs)
49 for k in cls.required_params:
50 if k not in o:
---> 51 raise ValueError("{}: Required parameter {} not specified!".format(cls, k))
52 return o
53
ValueError: <class 'fireworks.user_objects.firetasks.script_task.ScriptTask'>: Required parameter script not specified!
My current workaround is using a locally modified version of FiretaskMeta.call that uses an if statement to prevent reaching the parameter check when call has no inputs.
The inherent problem is that unpickle does not call init first, so those required parameters will never be filled out when the check happens. I can submit a PR if you’d like, or you may want to take a different approach.
class FiretaskMeta(abc.ABCMeta):
def call(cls, *args, **kwargs):
o = abc.ABCMeta.call(cls, *args, **kwargs)
if len(args) == 0 and len(kwargs) == 0:
return o
for k in cls.required_params:
if k not in o:
raise ValueError("{}: Required parameter {} not specified!".format(cls, k))
return o