Hi,
Is anyone of devs supports Fireworks! It seems that project is not supported at all?
Maybe I am missing something …
It’s supported.
2 Likes
Hi, thanks for reply…
So that I am clear, I like framework, I used very successfully at nyse. I want to use it but pytask option is dead.
Can someone look at that and confirm or provide working solution.
I need to know as I have tight deadlines and if this is not working for me. I will use celery for dispatching things if fireworks is not doing pytask.
Later I can fork and fix your framework if need be, but now it would be easier if someone with more insights gives me rundown on how stable framework is.
PyTasks are still inside of the current repo and can be used. See
Nicholas,
That is all fine! I am aware of it…
What is not ok is that I posted three questions and submitted a bug report with no responses.
Look at the PyTask questions and bugs from me on github and discourse. I worked with Anubhav long back and even fixed timestamps for Fireworks at the time.
The problem is that Pytask, even examples from documentation, will fail miserably when a rocket is launched. Launchpad and serialization and registration of the task is not working correctly.
So that is the issue I have. I need an example that works, but I believe that launchpad is not capable of running PyTasks, due to these issues. I actually provided a temp fix around checking for data types to avoid complete error but this reflects later in lp object as another error.
If you look at those tickets you will see what I am talking about.
Yes I could use script tasks to execute python app that I am interested in, but I would rather invoke modules and methods more directly via PyTask then dealing with rewrites of my architecture to use script tasks…
Regards,
I’m not a dev so I don’t know the issue history on github. I just use fireworks often.
In any case, I’d probably just suggest ignoring PyTask and using a quick custom Firetask, which definitely work as I use them all the time.
fw_timer = Firework(PyTask(func='time.sleep',args=[5]))
Becomes
@explicit_serialize
class Sleep(FiretaskBase):
required_params = ["sleep_time"]
def run_task(self, fw_spec):
time.sleep(self.get("sleep_time"))
fw_timer = Firework(Sleep(sleep_time=5))
Which should work fine. Hopefully that works for you.
What is not ok is that I posted three questions and submitted a bug report with no responses.
On this point, we are open-source codes that ultimately run on good will; in general, people are not paid to work on these codes, and people using the codes are not paying to use them.
I am also not a FireWorks developer but, speaking broadly for Materials Project and its codes, it doesn’t make us happy if an issue or a question sits without answers for a while – we would much prefer prompt, timely answers for inquiries – nevertheless, we have many constraints on our time and this is not always possible, and sometimes significant delays are unavoidable.
I hope @Nicholas_Winner’s answer can help in the meantime.
1 Like
Yeah, good point…
Will do that
When we are on the subject of custom tasks, may i ask this:
I need to push callable, meaning python objects containing bunch of objects and attributes to mongo queue.
I understand you are not dev, but maybe you guys know if this is possible with custom FT.
If not I can serialize those objects on NFS and pick them up from required_params …?
Thanks for help
Hi,
yes we know good will! I am down with that, and I do understand that everyone is busy! I am in the same boat and I also have open source projects that I support…
I like idea behind fireworks, and I think it is worth developing and pushing it forward.
Thanks for chiming in, and please do not take personally my frustrations …
Regards,
1 Like
Hi Sashak,
Thanks for sharing the issues you’ve found with FireWorks and to echo Matt - we do try to provide timely responses but this is not always possible.
As far as I know there haven’t been any recent changes to PyTask. Any idea why it stopped working suddenly? Does it work with a previous version of FireWorks?
Note that we are aware that pymongo 4.0 has broken some things and we attempted to fix them, although I am not sure that is the case here.
I’ve put two people in my group, Duo Wang and Zhuoying Zhu, in charge to investigate this.
Hi Anubhav,
Thanks for the reply…
PyTask is definitely failing here:
self.fireworks.find_one_and_replace({‘fw_id’: fw.fw_id},
fw.to_db_dict(),
upsert=True)
so:
def to_dict(self):
…
spec[’_tasks’] = [t.to_dict() for t in self.tasks]
t is already a dict and spec[’_tasks’] is trying to convert dict to dict failing miserably …
{‘spec’: {’_tasks’: [:{‘func’: ‘time.sleep’, ‘args’: [5], ‘_fw_name’: ‘PyTask’}]}, ‘fw_id’: -1, ‘created_on’: datetime.datetime(2022, 1, 16, 20, 20, 16, 789436), ‘updated_on’: datetime.datetime(2022, 1, 16, 20, 20, 16, 789437), ‘name’: ‘Unnamed FW’}
so I changed this code to debug a bit:
@recursive_serialize
def to_dict(self):
put tasks in a special location of the spec
spec = self.spec
spec[’_tasks’] = list()
for t in self.tasks:
print(t)
if isinstance(t, dict):
spec[’_tasks’].append(t)
else:
td = t.to_dict()
spec[’_tasks’].append(td)
##spec['_tasks'] = [t.to_dict() for t in self.tasks]
m_dict = {'spec': spec, 'fw_id': self.fw_id, 'created_on': self.created_on,
'updated_on': self.updated_on}
# only serialize these fields if non-empty
if len(list(self.launches)) > 0:
m_dict['launches'] = self.launches
if len(list(self.archived_launches)) > 0:
m_dict['archived_launches'] = self.archived_launches
# keep export of new FWs to files clean
if self.state != 'WAITING':
m_dict['state'] = self.state
m_dict['name'] = self.name
return m_dict
Now firework is working correctly, meaning rocket can finish…
problem at this point is:
2022-01-16 15:28:50,762 INFO Rocket finished
Traceback (most recent call last):
File “/cube/api/py-3.9.7/lib/python3.9/site-packages/fireworks/core/rocket.py”, line 248, in run
l_logger.log(logging.INFO, "Task started: s. t.fw_name)
AttributeError: ‘dict’ object has no attribute ‘fw_name’
It looks to me now that i am looking into my own post that it might be that documentation needs an update or that Pytask needs some updates on the spec side.
Not sure, but I can contribute and help if anyone points to me the whole architecture and where is what.
Sorry to hear about pymongo, and that might be an issue with me too.
I have 5.x mongo, 3.10 python and latest fireworks.
On the side, my question for you is this:
I need to push large objects as args to tasks…
It looks to me that fireworks in custum task and pyTask does not allow required args to be python objects.
I do want to use callable objects to push them to worknodes and let the worknodes do the work. These objects are essentially template structures of info that worknode drivers will use to do the work.
in mongo on my side no matter how I construct the firetask i endup with string object in mongo representing python class object instance.
All what I need from you guys is yes or no that this can be done, and if not can we do something in custom metacalss for FireBase to actually allow this to be added as a param. maybe is pymongo that is normalizing all of this to strings, but i am pretty sure that mongo can be feed pickle binary …
I know how I will go around this. I will serialize py objs with Dill on NFS, put the mapping to object as string into workflow task req field and let driver sort the things when it pick up a task and workflow.
it would be nice and I can contribute to change this in your code if someone gives me good rundown on flow of fireworks so that I am not lost in see of code.
Thanks again for replying and i am happy that your project lived for so long … It means that all of you who contributed did a good job …
Hi Sashak,
For the PyTask I will leave to Duo / Zhuoying.
In terms of passing objects as arguments you need to make sure the objects can be serialized/deserialized properly. To do that the objects must subclass FWSerializable. The main requirement is that the object has a to_dict() and from_dict() method. This will make sure that the object can be converted into MongoDB and (via to_dict()) and re-instantiated as an object (via from_dict()).
1 Like
Hello @sashak ! Hope @Anubhav_Jain has answered your question in serializable object using FWSerializable (fireworks.utilities package — FireWorks 1.9.8 documentation).
Thanks for bringing up this problem in PyTask. We will try to look into the branch and resolve the fw_name() attributeError happened in fireworks/PyTask. You are also welcome to contribute to fireworks by opening PR to fix the to_dict() issue.
I will create a separate method for checking types in the class. Then integrate with to_dict method.
This way any other method that needs to convert anything to dict coudl utilize this method. If anything else aside from list of tupels need to be converted in the future we can decorate that method to deal with any type that needs to convert.
Which branch you need me to do this in … I guess I fork the project do my stuff and create merge for you guys to approve it …
Right?
Thanks for the note on serialization.
Problem that I have is that I am using a state machine to create this object.
If I understand correctly:
class A(FWSerializable):
pass
then do the custom task which will also use decorator to register
@explicit_serialize
class MPACWFTask(FiretaskBase):
“”"
Custom firetask for attestation framework
Args:
"""
_fw_name = 'AttestationTask'
required_params = ["serviceDriver"]
optional_params = ['optional']
def run_task(self, fw_spec):
print(self['serviceDriver'])
obj = ast.literal_eval(self['serviceDriver'])# print the values of required parameters
print(self.fw_spec['name']) # print the name of the Firewor
return FWAction()
something like above
my problem is that
original class A is currently using another metaclass
class A(StateMachine …
so if I now add
class A(StateMachine,FWSerializable):
…
i am getting metaclasses conflicts …
So I am a bit confused…
basically my state machine contains everything that I need pass to worker as param
so if I do:
mpacTask = Firework(MPACWFTask(serviceDriver= self))
mongo will accept this but on
def run_task(self, fw_spec):
print(self[‘serviceDriver’])
serviceDriver is string representing an object
maybe i should within StateMachine create a new FWSerializable object and add attributes to that object thus insulating class inheritance and metaclasses clashes alltogether.
Any thoughts?
never mind, I just read note from Duo Wang and I will look at the utilities package
Hello @sashak. What we usually do here in materials project is to folk the package to your own, and then create new branch and make the change of your own codes. Once you finish, you can pull a request (PR) to the main branch of fireworks. Once all the modified codes are reviewed and all the tests are passed, the persons in charge of maintenance will help you to merge the PR into the main/master branch.
Hope this information is helpful. Please let us know for any problems or concerns.
Cool thanks
Hi, it seems that fireworks can iterate (class.repr)
when I create wrapped object for customTask and try to feed to launchpd.add_wf
Example:
@explicit_serialize
class MPACWFTask(FiretaskBase):
“”"
Custom firetask for attestation framework
Args:
"""
_fw_name = 'AttestationTask'
required_params = ["serviceDriver"]
optional_params = ['optional']
def run_task(self, fw_spec):
print(self['serviceDriver'])
obj = ast.literal_eval(self['serviceDriver'])# print the values of required parameters
print(self.fw_spec['name']) # print the name of the Firewor
return FWAction()
class MyTaskObject(FWSerializable):
@serialize_fw
def to_dict(self):
return dict(self)
@classmethod
def from_dict(cls, m_dict):
return cls(m_dict)
def __repr__(self):
return dict(self)
self.mpacTask = MyTaskObject()
self.mpacTask.someAttrib = ‘sasha’
mpacTask =MPACWFTask(serviceDriver= self.mpacTask)
firework = Firework(mpacTask )
launchpad = LaunchPad()
launchpad.reset(’’, require_password=False)
try:
launchpad.add_wf(firework)
except Exception as lpErr:
logger.error(lpErr)
raise PropFailure(lpErr)
Unable to get repr for <class ‘FireTaskCustom.MPACWFTask’>
so what am I doing wrong …
i am not sure, but it seems to me that i am duplicating serialization via my own task implementation and custom task implementation.
So possibly that is causing repr and MytaskObject no being iterable.
Please provide me with simple example of what you mean by custom task req_param being object that is serialize to dict and back to object on the remote end.