I used way back fireworks, but honestly do not remember what version.
I know that yaml templating was not in existence in the project at that time.
It served me well, and it was a huge success. I was normalizing 24TB of stock data super fast and in very robust way …
Now I need it again so I need to get some answers from devs on what is possible.
I need a launch pad to setup pretty complex setup of options and invoke specific heavy python apps on fireworker servers that will execute these tasks. Tasks are not changing and they are pretty resource hungry on the fireworker side. They do manage and report on resource consumption.
Idea is that launchpad will do autodiscovery of some service end-points in cloud environments and will create a queue (list) of these services with all the necessary templates that will be pushed with execution on fireworker.
I actually do not want to drive the process from the launchpad, I just want to setup a queue and let fireworkers pick up a job at the time and execute tasks serializing output on the NFS shared storage point.
This goes to on and on until queue is exhausted.
In some cases, workflow changes, but return objects to launchpad can signal that and workflow can be changed for those so that more parallelization can take place in those cases.
I am also not at all fan of yaml! my templates are my own INI files with pretty heavy configparser overwrite.
My firetask (heavy python app) is using those and that is not changing.
So my apologies for this rough intro, here are my questions for community:
I want to run programmatically via Python 3 no command line FW framework without yaml templates, is this possible?
I have no issues with json and I can use that no problem
next question is I know queue still exist but I am not seeing any examples of the framework (python code wise) to give me a sense of how to utilize a queue and how to bootstrap fireworkers and listen of the queue and wait for jobs to show up.
Any recommendations on the performance side would be welcome, specifically on the shared file system.
I need a NFS to be able to funnel stuff that i am producing and process final outputs with launchpad task that is a same app that fireworker is executing on work servers.
Much appreciated
to add on I know that programmatically I can setup a job without Yaml so I answered that part myself…