Fireworks

itsshawnzhang · January 20, 2022, 7:28pm

Hope you can find this test file for testing fw_serializers useful.

materialsproject/fireworks/blob/83a907c19baf2a5c9fdcf63996f9797c3c85b785/fireworks/utilities/tests/test_fw_serializers.py

# from __future__ import unicode_literals

from fireworks.user_objects.firetasks.unittest_tasks import (
    ExportTestSerializer,
    TestSerializer,
)
from fireworks.utilities.fw_serializers import (
    FWSerializable,
    load_object,
    recursive_dict,
)
from fireworks.utilities.fw_utilities import explicit_serialize

__author__ = "Anubhav Jain"
__copyright__ = "Copyright 2013, The Materials Project"
__version__ = "0.1"
__maintainer__ = "Anubhav Jain"
__email__ = "[email protected]"
__date__ = "Jan 26, 2013"

This file has been truncated. show original

sashak · January 20, 2022, 8:01pm

Hi, thanks for the pointer.
I am playing with this for days now.
Funny pat is that I have my own dill implemetaion of moving py objects across network but I am in ocean of fireworks code trying to mimic the right solution. This might help when I get back to pycharm.

Zhuoying · January 20, 2022, 9:25pm

Hi @sashak. Hope this example of TestSerializer could help.

github.com

materialsproject/fireworks/blob/83a907c19baf2a5c9fdcf63996f9797c3c85b785/fireworks/user_objects/firetasks/unittest_tasks.py

import datetime

from fireworks.utilities.fw_serializers import FWSerializable, serialize_fw

__author__ = "Anubhav Jain"
__copyright__ = "Copyright 2014, The Materials Project"
__version__ = "0.1"
__maintainer__ = "Anubhav Jain"
__email__ = "[email protected]"
__date__ = "Jan 21, 2014"


class TestSerializer(FWSerializable):

    _fw_name = "TestSerializer Name"

    def __init__(self, a, m_date):
        if not isinstance(m_date, datetime.datetime):
            raise ValueError("m_date must be a datetime instance!")

This file has been truncated. show original

You can compare the way you define/init your MyTaskObject.

For pymongo>=4.0.0 issues, something we came across is AttributeError like

‘Cursor’ object has no attribute ‘count()’

So if you are debugging your class with those errors, please do check the pymongo 4.0 changes.

If you want to PR something later when you finish debugging, please create a new branch other than main and then submit the PR. We have a standard review and merge process with timely responses for those valued PRs.

Thanks for your efforts.

sashak · January 21, 2022, 7:07pm

Hi, pymongo was not an issue for me.
I am running pymongo4.0.1 against
mongodb-org-shell-5.0.5-1.el8.x86_64
mongodb-org-server-5.0.5-1.el8.x86_64

Serialization and deserialization via FW did not work for me.
What did work is following
I am converting nested object to nested dicts via my own method.
Then custom dict is passed to fireworks custom task,
All of thi is done with my own wrappers to FW
e.g.
workFlowObjects = list()
mpacObj = self.myDict(self.mpacTask)
mpacTask = MPACWFTask(serviceDriver= mpacObj)
workFlowObjects.append(Firework(mpacTask))
# ## store workflow
# ##change the port from str to int
self.mpacTask.clusterObj.lpad.port = int(self.mpacTask.clusterObj.lpad.port)
fire = LpadMpac(workFlowObjects, self.mpacTask.clusterObj.lpad)
# ## init mongo DB queue
try:
launhcpad = fire.setupLP(reset=True, needPasswd=True)
except Exception as clusterErr:
print(clusterErr)
# ## load workflows
fire.setup()

on the other end (remote)

passed = fire.singleShoot()
will invoke

Custom FireTask
def run_task(self, fw_spec):
mpacTaskObj = type(‘mpacTaskObj’, (object,), self[‘serviceDriver’])
print(mpacTaskObj.mObject)
print(self._fw_name) # print the name of the Fireworker

    return FWAction()

This is all good, one more thing is that I need to expand on the nested dict to nested objects and not just inital layer of dicts.

All in all it works now for me.

I will in Feb look into serialization again and possibly fix things if they need to be fixed.
Unit test do work, so most likely ether I am not composing object correctly through my app thus repr is not present or otherwise I am not using FireTask documentation correctly …

Thanks all for help…

PyTask will be forked and merged this weekend …

ardunn · January 23, 2022, 4:05am

Hi @sashak

This has likely been addressed but Fireworks is indeed supported. We are currently having some issues with Pymongo 4.0 right now (due to a merge I committed, I admit ). See this PR here Pymongo 4 Fixes (WIP) by ardunn · Pull Request #473 · materialsproject/fireworks · GitHub where I aim to fix pymongo 4.0 compatibility issues, but it is likely due to upstream problems with the mongo source code and it should only affect fws tests, not many real uses of fws (to my knowledge).