passing data between firework specs using yaml?

Can anyone tell me if there is a way to pass data from the spec of one firework to the spec of a child firework in yaml? I feel like I read this is possible, and it certainly seems intuitive but now I can’t find it.

The data are just simple items that I want to have available to the inputs section of my children. I have a number of common configuration parameters that have to be sent to many python scripts whose API is locked in. I’d like to know if I can do this with standard features or whether I need to subclass PyTask to accomplish config-file-awareness.

Thanks.

Hi Eli,

You can’t specify what data to pass from between Fireworks in the YAML file itself. If you know the spec parameters at the time of creating your workflow YAML file, you can of course include those spec parameters (hard-coded) into the YAML file itself.

If you need to pass data between the Fireworks that is not known in advance, i.e., can only be known after performing the first Firework computation, you need to use the dynamic workflows functionality. Thus far, everyone I know that has used this has written Python code to return a FWAction() object that specifies the data to pass. But, there is a little known (and perhaps never used) feature that allows you to instead write a file called FWAction.json that contains that information, and which doesn’t require returning a FWAction. e.g. via lines: fireworks/core/rocket.py:308 in the code.

···

Best,
Anubhav

Thanks Anubhav. I’ll check it out. A lot of this is known advance, I just don’t want to write it hard-coded multiple times in each spec which will surely lead to a mistake.

As you know, I’m just getting my feet wet. I didn’t want to overlook out-of-the-box functionality. It probably isn’t hard to write a config-aware wrapper around PyTask that contains the common data.

···

On Tuesday, February 12, 2019 at 1:42:45 PM UTC-8, ajain wrote:

Hi Eli,

You can’t specify what data to pass from between Fireworks in the YAML file itself. If you know the spec parameters at the time of creating your workflow YAML file, you can of course include those spec parameters (hard-coded) into the YAML file itself.

If you need to pass data between the Fireworks that is not known in advance, i.e., can only be known after performing the first Firework computation, you need to use the dynamic workflows functionality. Thus far, everyone I know that has used this has written Python code to return a FWAction() object that specifies the data to pass. But, there is a little known (and perhaps never used) feature that allows you to instead write a file called FWAction.json that contains that information, and which doesn’t require returning a FWAction. e.g. via lines: fireworks/core/rocket.py:308 in the code.

On Tue, Feb 12, 2019 at 1:26 PM [email protected] wrote:

Can anyone tell me if there is a way to pass data from the spec of one firework to the spec of a child firework in yaml? I feel like I read this is possible, and it certainly seems intuitive but now I can’t find it.

The data are just simple items that I want to have available to the inputs section of my children. I have a number of common configuration parameters that have to be sent to many python scripts whose API is locked in. I’d like to know if I can do this with standard features or whether I need to subclass PyTask to accomplish config-file-awareness.

Thanks.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

Anubhav,

Is the following super simple workaround dangerous or inadvisable for the known-in-advance case? It seems to work OK on my little toy examples. I just create one of these ConfigTasks in yaml. It has a single spec/task variable “config”, which in turn is a dictionary containing variables to be passed on to children. It works on my toy examples, but I am too inexperienced to understand the possible side effects.Thanks.

class ConfigTask(FiretaskBase):
“”"
Task to push yaml-specified variables onto spec
so it can be inherited by children
Required parameters:
- config: (dict) - parameters to be pushed to children
“”"
_fw_name = ‘ConfigTask’

def run_task(self, fw_spec):
    if self.get("use_global_spec"):
        self._load_params(fw_spec)
    else:
        self._load_params(self)    

    return FWAction(mod_spec=[{'_push': self.config}])

def _load_params(self, d):
    self.config = d['config']

``

···

On Tuesday, February 12, 2019 at 1:42:45 PM UTC-8, ajain wrote:

Hi Eli,

You can’t specify what data to pass from between Fireworks in the YAML file itself. If you know the spec parameters at the time of creating your workflow YAML file, you can of course include those spec parameters (hard-coded) into the YAML file itself.

If you need to pass data between the Fireworks that is not known in advance, i.e., can only be known after performing the first Firework computation, you need to use the dynamic workflows functionality. Thus far, everyone I know that has used this has written Python code to return a FWAction() object that specifies the data to pass. But, there is a little known (and perhaps never used) feature that allows you to instead write a file called FWAction.json that contains that information, and which doesn’t require returning a FWAction. e.g. via lines: fireworks/core/rocket.py:308 in the code.

On Tue, Feb 12, 2019 at 1:26 PM [email protected] wrote:

Can anyone tell me if there is a way to pass data from the spec of one firework to the spec of a child firework in yaml? I feel like I read this is possible, and it certainly seems intuitive but now I can’t find it.

The data are just simple items that I want to have available to the inputs section of my children. I have a number of common configuration parameters that have to be sent to many python scripts whose API is locked in. I’d like to know if I can do this with standard features or whether I need to subclass PyTask to accomplish config-file-awareness.

Thanks.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

Hi Eli

I don’t think there’s anything dangerous in the code you wrote. I don’t know exactly how your code and workflow works in an overall sense so I can’t provide much more specific advice.

Note that the “update_spec” FWAction might be easier to use than the “mod_spec” one; typically we just use the latter if there is some complex operation needed.

Best,

Anubhav

···

Best,
Anubhav

Agreed about update_spec. Even given I used mod_spec, “_set” would have been better than “_push”.

I’ve been playing with this it is a nice way of setting configuration variables that remain in place for the rest of the flow. Although it works, and it is super easy for any user-written task, there is uneven support for lookup in the spec across the built in tasks. For some it is an intrinsic part of how they work (PyTask). For others it is something you can turn on (use_global_spec) but you have to choose between task and spec. Still others don’t offer it at all (some of the FileIO tasks).

···

On Thursday, February 14, 2019 at 5:53:53 AM UTC-8, ajain wrote:

Hi Eli

I don’t think there’s anything dangerous in the code you wrote. I don’t know exactly how your code and workflow works in an overall sense so I can’t provide much more specific advice.

Note that the “update_spec” FWAction might be easier to use than the “mod_spec” one; typically we just use the latter if there is some complex operation needed.

Best,

Anubhav

On Tue, Feb 12, 2019 at 3:17 PM [email protected] wrote:

Anubhav,

Is the following super simple workaround dangerous or inadvisable for the known-in-advance case? It seems to work OK on my little toy examples. I just create one of these ConfigTasks in yaml. It has a single spec/task variable “config”, which in turn is a dictionary containing variables to be passed on to children. It works on my toy examples, but I am too inexperienced to understand the possible side effects.Thanks.

class ConfigTask(FiretaskBase):
“”"
Task to push yaml-specified variables onto spec
so it can be inherited by children
Required parameters:
- config: (dict) - parameters to be pushed to children
“”"
_fw_name = ‘ConfigTask’

def run_task(self, fw_spec):
    if self.get("use_global_spec"):
        self._load_params(fw_spec)
    else:
        self._load_params(self)    

    return FWAction(mod_spec=[{'_push': self.config}])

def _load_params(self, d):
    self.config = d['config']

``

On Tuesday, February 12, 2019 at 1:42:45 PM UTC-8, ajain wrote:

Hi Eli,

You can’t specify what data to pass from between Fireworks in the YAML file itself. If you know the spec parameters at the time of creating your workflow YAML file, you can of course include those spec parameters (hard-coded) into the YAML file itself.

If you need to pass data between the Fireworks that is not known in advance, i.e., can only be known after performing the first Firework computation, you need to use the dynamic workflows functionality. Thus far, everyone I know that has used this has written Python code to return a FWAction() object that specifies the data to pass. But, there is a little known (and perhaps never used) feature that allows you to instead write a file called FWAction.json that contains that information, and which doesn’t require returning a FWAction. e.g. via lines: fireworks/core/rocket.py:308 in the code.

On Tue, Feb 12, 2019 at 1:26 PM [email protected] wrote:

Can anyone tell me if there is a way to pass data from the spec of one firework to the spec of a child firework in yaml? I feel like I read this is possible, and it certainly seems intuitive but now I can’t find it.

The data are just simple items that I want to have available to the inputs section of my children. I have a number of common configuration parameters that have to be sent to many python scripts whose API is locked in. I’d like to know if I can do this with standard features or whether I need to subclass PyTask to accomplish config-file-awareness.

Thanks.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

Hi Eli

Most (i.e. almost all) of our own work is done with custom FireTasks. I coded up some of the default FireTasks a long time ago without really knowing how they would be used, what would be useful, etc. and just taking a guess. I had also opened an issue about a possible way forward here:

https://github.com/materialsproject/fireworks/issues/283

I’m flexible as to what comes next for the built-in FireTasks …

Best,

Anubhav

···

Best,
Anubhav

I’m sure I’ll be writing custom tasks myself soon, and I certainly appreciate having some examples. As far as the inheritance and replacement, the description in this issue is exactly what I had in mind.

What might enhance this proposal would be a universal and explicit marker indicating an item represents a key that should be looked up in the spec. I’d like to write a spec (syntax negotiable):

source : “{ input file }

…and be assured that “input file” will be looked up from the spec as it exists at run_task time with little or no additional coding in the FireTask. This would also reduce boilerplate code for figuring out substitutions. What do you think?

···

On Thursday, February 14, 2019 at 2:34:33 PM UTC-8, ajain wrote:

Hi Eli

Most (i.e. almost all) of our own work is done with custom FireTasks. I coded up some of the default FireTasks a long time ago without really knowing how they would be used, what would be useful, etc. and just taking a guess. I had also opened an issue about a possible way forward here:

https://github.com/materialsproject/fireworks/issues/283

I’m flexible as to what comes next for the built-in FireTasks …

Best,

Anubhav

On Thu, Feb 14, 2019 at 1:47 PM [email protected] wrote:

Agreed about update_spec. Even given I used mod_spec, “_set” would have been better than “_push”.

I’ve been playing with this it is a nice way of setting configuration variables that remain in place for the rest of the flow. Although it works, and it is super easy for any user-written task, there is uneven support for lookup in the spec across the built in tasks. For some it is an intrinsic part of how they work (PyTask). For others it is something you can turn on (use_global_spec) but you have to choose between task and spec. Still others don’t offer it at all (some of the FileIO tasks).

On Thursday, February 14, 2019 at 5:53:53 AM UTC-8, ajain wrote:

Hi Eli

I don’t think there’s anything dangerous in the code you wrote. I don’t know exactly how your code and workflow works in an overall sense so I can’t provide much more specific advice.

Note that the “update_spec” FWAction might be easier to use than the “mod_spec” one; typically we just use the latter if there is some complex operation needed.

Best,

Anubhav

On Tue, Feb 12, 2019 at 3:17 PM [email protected] wrote:

Anubhav,

Is the following super simple workaround dangerous or inadvisable for the known-in-advance case? It seems to work OK on my little toy examples. I just create one of these ConfigTasks in yaml. It has a single spec/task variable “config”, which in turn is a dictionary containing variables to be passed on to children. It works on my toy examples, but I am too inexperienced to understand the possible side effects.Thanks.

class ConfigTask(FiretaskBase):
“”"
Task to push yaml-specified variables onto spec
so it can be inherited by children
Required parameters:
- config: (dict) - parameters to be pushed to children
“”"
_fw_name = ‘ConfigTask’

def run_task(self, fw_spec):
    if self.get("use_global_spec"):
        self._load_params(fw_spec)
    else:
        self._load_params(self)    

    return FWAction(mod_spec=[{'_push': self.config}])

def _load_params(self, d):
    self.config = d['config']

``

On Tuesday, February 12, 2019 at 1:42:45 PM UTC-8, ajain wrote:

Hi Eli,

You can’t specify what data to pass from between Fireworks in the YAML file itself. If you know the spec parameters at the time of creating your workflow YAML file, you can of course include those spec parameters (hard-coded) into the YAML file itself.

If you need to pass data between the Fireworks that is not known in advance, i.e., can only be known after performing the first Firework computation, you need to use the dynamic workflows functionality. Thus far, everyone I know that has used this has written Python code to return a FWAction() object that specifies the data to pass. But, there is a little known (and perhaps never used) feature that allows you to instead write a file called FWAction.json that contains that information, and which doesn’t require returning a FWAction. e.g. via lines: fireworks/core/rocket.py:308 in the code.

On Tue, Feb 12, 2019 at 1:26 PM [email protected] wrote:

Can anyone tell me if there is a way to pass data from the spec of one firework to the spec of a child firework in yaml? I feel like I read this is possible, and it certainly seems intuitive but now I can’t find it.

The data are just simple items that I want to have available to the inputs section of my children. I have a number of common configuration parameters that have to be sent to many python scripts whose API is locked in. I’d like to know if I can do this with standard features or whether I need to subclass PyTask to accomplish config-file-awareness.

Thanks.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav