(bug) picke/unpickle Firetask with required parameters

The specific bug is that when unpickling a Firetask with required parameters, the instance parameter test in FiretaskMeta.call fails, fatally ending the unpickle operation.

The code is here:

The use case is in the context of Jupyter interactive workflows, where I want to have fine-grained controls and monitoring over each Firetask, without requiring that users of existing workflows rewrite everything according to a new spec. Each Firetask is being run by a service in a separate process from the rocket launcher.

Here is a simple example to replicate this:

import pickle

from fireworks import ScriptTask

task = ScriptTask.from_str(‘ls -l > listing.txt’)

message = pickle.dumps(task)

obj = pickle.loads(message)

This will produce an exception traceback similar to the following:


<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-a8ca0b9807d0> in <module>()
      5 task = ScriptTask.from_str('ls -l > listing.txt')
      6 message = pickle.dumps(task)
----> 7 obj = pickle.loads(message)

~/miniconda3/lib/python3.6/site-packages/fireworks/core/firework.py in __call__(cls, *args, **kwargs)
     49         for k in cls.required_params:
     50             if k not in o:
---> 51                 raise ValueError("{}: Required parameter {} not specified!".format(cls, k))
     52         return o
     53

ValueError: <class 'fireworks.user_objects.firetasks.script_task.ScriptTask'>: Required parameter script not specified!

My current workaround is using a locally modified version of FiretaskMeta.call that uses an if statement to prevent reaching the parameter check when call has no inputs.

The inherent problem is that unpickle does not call init first, so those required parameters will never be filled out when the check happens. I can submit a PR if you’d like, or you may want to take a different approach.

class FiretaskMeta(abc.ABCMeta):

def call(cls, *args, **kwargs):

o = abc.ABCMeta.call(cls, *args, **kwargs)

if len(args) == 0 and len(kwargs) == 0:

return o

for k in cls.required_params:

if k not in o:

raise ValueError("{}: Required parameter {} not specified!".format(cls, k))

return o

Hi Matt,

We never used pickle serialization (always used the to_dict() and from_dict()) so hadn’t run into this.

I think the solution you propose is fine. If you can submit a PR I can merge and release a new version. The only modification I’d make is just to add a small comment either above or in-line with your added lines to say that this is to enable pickling support.

e.g.

if len(args) == 0 and len(kwargs) == 0: # to support pickle / unpickle

return o

···

On Mon, Mar 5, 2018 at 12:42 PM, Matt Henderson [email protected] wrote:

The specific bug is that when unpickling a Firetask with required parameters, the instance parameter test in FiretaskMeta.call fails, fatally ending the unpickle operation.

The code is here:

https://github.com/materialsproject/fireworks/blob/master/fireworks/core/firework.py#L46-L52

The use case is in the context of Jupyter interactive workflows, where I want to have fine-grained controls and monitoring over each Firetask, without requiring that users of existing workflows rewrite everything according to a new spec. Each Firetask is being run by a service in a separate process from the rocket launcher.

Here is a simple example to replicate this:

import pickle

from fireworks import ScriptTask

task = ScriptTask.from_str(‘ls -l > listing.txt’)

message = pickle.dumps(task)

obj = pickle.loads(message)

This will produce an exception traceback similar to the following:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-a8ca0b9807d0> in <module>()
      5 task = ScriptTask.from_str('ls -l > listing.txt')
      6 message = pickle.dumps(task)
----> 7 obj = pickle.loads(message)

~/miniconda3/lib/python3.6/site-packages/fireworks/core/firework.py in __call__(cls, *args, **kwargs)
     49         for k in cls.required_params:
     50             if k not in o:
---> 51                 raise ValueError("{}: Required parameter {} not specified!".format(cls, k))
     52         return o
     53

ValueError: <class 'fireworks.user_objects.firetasks.script_task.ScriptTask'>: Required parameter script not specified!

My current workaround is using a locally modified version of FiretaskMeta.call that uses an if statement to prevent reaching the parameter check when call has no inputs.

The inherent problem is that unpickle does not call init first, so those required parameters will never be filled out when the check happens. I can submit a PR if you’d like, or you may want to take a different approach.

class FiretaskMeta(abc.ABCMeta):

def call(cls, *args, **kwargs):

o = abc.ABCMeta.call(cls, *args, **kwargs)

if len(args) == 0 and len(kwargs) == 0:

return o

for k in cls.required_params:

if k not in o:

raise ValueError(“{}: Required parameter {} not specified!”.format(cls, k))

return o

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f856a88d-7dc2-48a1-8d78-dfdf0458a590%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Best,
Anubhav