Question about dynamic WF

Hi,

I am trying to use “additions” and “detours” of FWAction, but I have encountered two unexpected behaviors.

I am not sure that I’m doing correct things, so please correct or advise me if I’m wrong.

My test script is as follows;

my_fw_for_fwaction.py:

import os

from fireworks import explicit_serialize, FiretaskBase, FWAction, Firework

from fireworks.features.dupefinder import DupeFinderBase

path = [os.path.dirname(file)]

@explicit_serialize

class PrintTask(FiretaskBase):

required_params = [“name”]

def run_task(self, fw_spec):

   print("---self---\n")

   for k in self.keys():

       print("{}: {}\n".format(k, self[k]))

   print("---fw_spec---\n")

   for k in fw_spec.keys():

       print("{}: {}\n".format(k, fw_spec[k]))

   return FWAction()

@explicit_serialize

class AdditionsTask(FiretaskBase):

required_params = ["name", "parents"]

def run_task(self, fw_spec):

   new_spec = fw_spec if fw_spec else {}

   new_spec["added"] = True

   print(new_spec)

   tasks = [PrintTask(name="Additions")]

   return FWAction(additions=[Firework(tasks,

                                       spec=new_spec,

                                       parents=self["parents"])])

class MyDupeFinder(DupeFinderBase):
_fw_name = ‘MyDupeFinder’

def verify(self, spec1, spec2):

   return True

def query(self, spec):

   print("query called!")

   return {"$and": [{"launches": {"$ne": []}}]}

class FirstFW(Firework):

_fw_name = “FirstFW”

def init(self, parents=None):

   self._task = [PrintTask(name="fw1_print"),

                 AdditionsTask(name="add1", parents=self)]

   super().__init__(self._task, parents=parents, name=self._fw_name,

                    spec={"_dupefinder": MyDupeFinder()})

class SecondFW(Firework):

_fw_name = “SecondFW”

def init(self, parents=None):

   self._task = [PrintTask(name="fw2-print")]

   super().__init__(self._task, parents=parents, name=self._fw_name)

``

add_wf_to_lpad.py:

from future import division, unicode_literals, print_function

from fireworks import LaunchPad, Workflow

from my_fw_for_fwaction import FirstFW, SecondFW

lpad = LaunchPad.auto_load()

def add_wf(name):

first = FirstFW()

second = SecondFW(parents=first)

wf = [first, second]

wf = Workflow(wf, name=name)

lpad.add_wf(wf)

if name == “main”:

add_wf(“test”)

``

Command I executed on terminal:

lpad reset

python add_wf_to_lpad.py
rlaunch singleshot
rlaunch singleshot

``

1st problem:

When I wrote script and executed add_wf_to_lpad.py, error as following occurs.

Traceback (most recent call last):

File “add_wf_to_lpad.py”, line 20, in

add_wf(“test”)

File “add_wf_to_lpad.py”, line 16, in add_wf

lpad.add_wf(wf)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/launchpad.py”, line 320, in add_wf

old_new = self._upsert_fws(list(wf.id_fw.values()), reassign_all=reassign_all)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/launchpad.py”, line 1352, in _upsert_fws

self.fireworks.insert_many((fw.to_db_dict() for fw in fws))

File “/home/common/virtualenv/python3.6/lib/python3.6/site-packages/pymongo/collection.py”, line 752, in insert_many

blk.ops = [doc for doc in gen()]

File “/home/common/virtualenv/python3.6/lib/python3.6/site-packages/pymongo/collection.py”, line 752, in

blk.ops = [doc for doc in gen()]

File “/home/common/virtualenv/python3.6/lib/python3.6/site-packages/pymongo/collection.py”, line 742, in gen

for document in documents:

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/launchpad.py”, line 1352, in

self.fireworks.insert_many((fw.to_db_dict() for fw in fws))

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/firework.py”, line 301, in to_db_dict

m_dict = self.to_dict()

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 141, in _decorator

m_dict = func(self, *args, **kwargs)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/firework.py”, line 257, in to_dict

spec[’_tasks’] = [t.to_dict() for t in self.tasks]

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/firework.py”, line 257, in

spec[’_tasks’] = [t.to_dict() for t in self.tasks]

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 168, in _decorator

m_dict = func(self, *args, **kwargs)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 142, in _decorator

m_dict = recursive_dict(m_dict)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 83, in recursive_dict

for k, v in obj.items()}

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 83, in

for k, v in obj.items()}

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 76, in recursive_dict

return recursive_dict(obj.as_dict(), preserve_unicode)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 212, in as_dict

return self.to_dict()

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 141, in _decorator

m_dict = func(self, *args, **kwargs)

… (very long error message, but simply repeated)

RecursionError: maximum recursion depth exceeded while calling a Python object

``

When I specified parents=None in FirstFW, instead of parents=self, this problem was solved.

Don’t I need to specify added task’s “parents”?

2nd Problem:

Even I rewirted FirstFW as above (parents=self -> parents=None), the second rlaunch fails.

The output is as follows;

rlaunch singleshot
2018-12-15 19:28:37,542 INFO Hostname/IP lookup (this will take a few seconds)

2018-12-15 19:28:37,543 INFO Launching Rocket

query called!

2018-12-15 19:28:37,596 INFO Duplicate found! fwids 3 and 2

query called!

2018-12-15 19:28:37,623 INFO Duplicate found! fwids 4 and 3

query called!

2018-12-15 19:28:37,654 INFO Duplicate found! fwids 5 and 4

query called!

(…infinite loop)

``

Even I used “detours” instead of “additions”, the same problem occurred.

I will appreciate any helps and advices.

Best,

Akira Takahashi

Hi,

I am not sure if I ever responded to you about this question. Was this solved or is this still an issue?

···

On Saturday, December 15, 2018 at 2:50:18 AM UTC-8, Akira Takahashi wrote:

Hi,

I am trying to use “additions” and “detours” of FWAction, but I have encountered two unexpected behaviors.

I am not sure that I’m doing correct things, so please correct or advise me if I’m wrong.

My test script is as follows;

my_fw_for_fwaction.py:

import os

from fireworks import explicit_serialize, FiretaskBase, FWAction, Firework

from fireworks.features.dupefinder import DupeFinderBase

path = [os.path.dirname(file)]

@explicit_serialize

class PrintTask(FiretaskBase):

required_params = [“name”]

def run_task(self, fw_spec):

   print("---self---\n")
   for k in self.keys():
       print("{}: {}\n".format(k, self[k]))
   print("---fw_spec---\n")
   for k in fw_spec.keys():
       print("{}: {}\n".format(k, fw_spec[k]))
   return FWAction()

@explicit_serialize

class AdditionsTask(FiretaskBase):

required_params = ["name", "parents"]

def run_task(self, fw_spec):

   new_spec = fw_spec if fw_spec else {}
   new_spec["added"] = True
   print(new_spec)
   tasks = [PrintTask(name="Additions")]
   return FWAction(additions=[Firework(tasks,
                                       spec=new_spec,
                                       parents=self["parents"])])

class MyDupeFinder(DupeFinderBase):
_fw_name = ‘MyDupeFinder’

def verify(self, spec1, spec2):

   return True

def query(self, spec):

   print("query called!")
   return {"$and": [{"launches": {"$ne": []}}]}

class FirstFW(Firework):

_fw_name = “FirstFW”

def init(self, parents=None):

   self._task = [PrintTask(name="fw1_print"),
                 AdditionsTask(name="add1", parents=self)]
   super().__init__(self._task, parents=parents, name=self._fw_name,
                    spec={"_dupefinder": MyDupeFinder()})

class SecondFW(Firework):

_fw_name = “SecondFW”

def init(self, parents=None):

   self._task = [PrintTask(name="fw2-print")]
   super().__init__(self._task, parents=parents, name=self._fw_name)

``

add_wf_to_lpad.py:

from future import division, unicode_literals, print_function

from fireworks import LaunchPad, Workflow

from my_fw_for_fwaction import FirstFW, SecondFW

lpad = LaunchPad.auto_load()

def add_wf(name):

first = FirstFW()

second = SecondFW(parents=first)

wf = [first, second]

wf = Workflow(wf, name=name)

lpad.add_wf(wf)

if name == “main”:

add_wf(“test”)

``

Command I executed on terminal:

lpad reset

python add_wf_to_lpad.py
rlaunch singleshot
rlaunch singleshot

``

1st problem:

When I wrote script and executed add_wf_to_lpad.py, error as following occurs.

Traceback (most recent call last):

File “add_wf_to_lpad.py”, line 20, in

add_wf(“test”)

File “add_wf_to_lpad.py”, line 16, in add_wf

lpad.add_wf(wf)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/launchpad.py”, line 320, in add_wf

old_new = self._upsert_fws(list(wf.id_fw.values()), reassign_all=reassign_all)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/launchpad.py”, line 1352, in _upsert_fws

self.fireworks.insert_many((fw.to_db_dict() for fw in fws))

File “/home/common/virtualenv/python3.6/lib/python3.6/site-packages/pymongo/collection.py”, line 752, in insert_many

blk.ops = [doc for doc in gen()]

File “/home/common/virtualenv/python3.6/lib/python3.6/site-packages/pymongo/collection.py”, line 752, in

blk.ops = [doc for doc in gen()]

File “/home/common/virtualenv/python3.6/lib/python3.6/site-packages/pymongo/collection.py”, line 742, in gen

for document in documents:

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/launchpad.py”, line 1352, in

self.fireworks.insert_many((fw.to_db_dict() for fw in fws))

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/firework.py”, line 301, in to_db_dict

m_dict = self.to_dict()

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 141, in _decorator

m_dict = func(self, *args, **kwargs)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/firework.py”, line 257, in to_dict

spec[’_tasks’] = [t.to_dict() for t in self.tasks]

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/core/firework.py”, line 257, in

spec[’_tasks’] = [t.to_dict() for t in self.tasks]

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 168, in _decorator

m_dict = func(self, *args, **kwargs)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 142, in _decorator

m_dict = recursive_dict(m_dict)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 83, in recursive_dict

for k, v in obj.items()}

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 83, in

for k, v in obj.items()}

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 76, in recursive_dict

return recursive_dict(obj.as_dict(), preserve_unicode)

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 212, in as_dict

return self.to_dict()

File “/usr_psi/local/lib/python3.6/dist-packages/fireworks/utilities/fw_serializers.py”, line 141, in _decorator

m_dict = func(self, *args, **kwargs)

… (very long error message, but simply repeated)

RecursionError: maximum recursion depth exceeded while calling a Python object

``

When I specified parents=None in FirstFW, instead of parents=self, this problem was solved.

Don’t I need to specify added task’s “parents”?

2nd Problem:

Even I rewirted FirstFW as above (parents=self -> parents=None), the second rlaunch fails.

The output is as follows;

rlaunch singleshot
2018-12-15 19:28:37,542 INFO Hostname/IP lookup (this will take a few seconds)

2018-12-15 19:28:37,543 INFO Launching Rocket

query called!

2018-12-15 19:28:37,596 INFO Duplicate found! fwids 3 and 2

query called!

2018-12-15 19:28:37,623 INFO Duplicate found! fwids 4 and 3

query called!

2018-12-15 19:28:37,654 INFO Duplicate found! fwids 5 and 4

query called!

(…infinite loop)

``

Even I used “detours” instead of “additions”, the same problem occurred.

I will appreciate any helps and advices.

Best,

Akira Takahashi