Using TemplateWriterTask to write yaml for another firework

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``

Hi

If I understand correctly, at the end of running TemplateWriterTask, you will have an output file on the file system called “filled.yaml” representing a Firework that you’d like to be added to the workflow (automatically).

My suggestion is that you add a second Firetask (within the same Firework) that runs immediately after the TemplateWriterTask. The job of this FireTask will be to read “filled.yaml” (convert the YAML to a Firework, using Firework.from_file(“filled.yaml”)) and then return a FWAction that adds this Firework to the workflow. You can either write up a Python function to do this and use PyTask for this purpose, or you could create a brand new FireTask whose job is to read a file and add it to the workflow. I’d probably suggest the latter.

Hope this helps …

···

On Friday, February 8, 2019 at 9:48:54 AM UTC-8, [email protected] wrote:

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``

Just to confirm, this works very well for the most part. I tried it on a Workflow with Workflow.from_file as below.

There is one quirk. It seems that the fw_id for the added yaml have to be negative. The links still work, so presumably they are adjusted the same way. My one concern is this … won’t this make it hard to use capabilities that are based on id numbers (things that have -i options)?

I assume this is happening to prevent collisions. I am using a capability that would normally be used for dynamic additions, and in that case (like the Fibonacci,adder) the user would typically not be able to anticipate or know the fw_id and use them for things like fizzled workflows. But for a parsed file like I’ve got, presumably it does make sense and the user is going to be counting on it. Would it. make sense to let the user supply a positive id if she wants? If she supplies a positive one it would be used unless it is not unique, then the workflow fails? .If it is negative, behavior would be the same as now?

Or is there more to it?

class AddYAMLToFlowTask(FiretaskBase):
_fw_name = “AddYAMLToFlowTask”

def run_task(self, fw_spec):
fname = fw_spec[‘filename’]
print(“Filename is: {}”.format(fname))
new_fw = Workflow.from_file(fname)
return FWAction(stored_data={‘launchyaml’: fname}, additions=new_fw)

``

···

On Friday, February 8, 2019 at 10:56:15 AM UTC-8, Anubhav Jain wrote:

Hi

If I understand correctly, at the end of running TemplateWriterTask, you will have an output file on the file system called “filled.yaml” representing a Firework that you’d like to be added to the workflow (automatically).

My suggestion is that you add a second Firetask (within the same Firework) that runs immediately after the TemplateWriterTask. The job of this FireTask will be to read “filled.yaml” (convert the YAML to a Firework, using Firework.from_file(“filled.yaml”)) and then return a FWAction that adds this Firework to the workflow. You can either write up a Python function to do this and use PyTask for this purpose, or you could create a brand new FireTask whose job is to read a file and add it to the workflow. I’d probably suggest the latter.

Hope this helps …

On Friday, February 8, 2019 at 9:48:54 AM UTC-8, [email protected] wrote:

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``

I would suggest that you use the “name” or some key you add in the spec (e.g. spec.your_id) to help keep track of your FireWorks. For example, you could do a search like:

lpad get_fws -n my_custom_name

lpad get_fws -q ‘{“spec.my_id”:7}’

I prefer not to have the user be setting fw_ids as this could cause issues. For example, FWS would have to keep track of every id that the user has already allocated in order to make sure that the FWS system doesn’t assign a new id that conflicts with a previous user id (instead of just incrementing a counter of FW ids). Generally, anything you want to do with ids could be done with other things like FW name or a spec field instead.

···

On Friday, February 8, 2019 at 4:34:44 PM UTC-8, [email protected] wrote:

Just to confirm, this works very well for the most part. I tried it on a Workflow with Workflow.from_file as below.

There is one quirk. It seems that the fw_id for the added yaml have to be negative. The links still work, so presumably they are adjusted the same way. My one concern is this … won’t this make it hard to use capabilities that are based on id numbers (things that have -i options)?

I assume this is happening to prevent collisions. I am using a capability that would normally be used for dynamic additions, and in that case (like the Fibonacci,adder) the user would typically not be able to anticipate or know the fw_id and use them for things like fizzled workflows. But for a parsed file like I’ve got, presumably it does make sense and the user is going to be counting on it. Would it. make sense to let the user supply a positive id if she wants? If she supplies a positive one it would be used unless it is not unique, then the workflow fails? .If it is negative, behavior would be the same as now?

Or is there more to it?

class AddYAMLToFlowTask(FiretaskBase):
_fw_name = “AddYAMLToFlowTask”

def run_task(self, fw_spec):
fname = fw_spec[‘filename’]
print(“Filename is: {}”.format(fname))
new_fw = Workflow.from_file(fname)
return FWAction(stored_data={‘launchyaml’: fname}, additions=new_fw)

``

On Friday, February 8, 2019 at 10:56:15 AM UTC-8, Anubhav Jain wrote:

Hi

If I understand correctly, at the end of running TemplateWriterTask, you will have an output file on the file system called “filled.yaml” representing a Firework that you’d like to be added to the workflow (automatically).

My suggestion is that you add a second Firetask (within the same Firework) that runs immediately after the TemplateWriterTask. The job of this FireTask will be to read “filled.yaml” (convert the YAML to a Firework, using Firework.from_file(“filled.yaml”)) and then return a FWAction that adds this Firework to the workflow. You can either write up a Python function to do this and use PyTask for this purpose, or you could create a brand new FireTask whose job is to read a file and add it to the workflow. I’d probably suggest the latter.

Hope this helps …

On Friday, February 8, 2019 at 9:48:54 AM UTC-8, [email protected] wrote:

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``

OK, I appreciate the idea of reducing need to deal with fw_id. In that spirit though, two suggestions that would take that a step further.

  1. It would be nice to not have to specify fw_id. As far as I can tell, in my Workflow.from_file context I need to put in a negative number or I get a complaint that the id must be negative…

  2. Given we omit fw_id, it would be nice if we had one of the following two designs for parsing links from the yaml file:

a. links could be be based on name or other spec field:

links:

{name : step1}

  • {name : step2}

b. [parents and children could be parsed from the file (or maybe only one or the other to prevent conflicts):

fws:

  • name: step1

children: {name : step2}

spec:

_tasks:

  • _fw_name: PyTask

func: simple.echo

args:

  • hello

What do you think?

···

On Friday, February 8, 2019 at 8:28:24 PM UTC-8, Anubhav Jain wrote:

I would suggest that you use the “name” or some key you add in the spec (e.g. spec.your_id) to help keep track of your FireWorks. For example, you could do a search like:

lpad get_fws -n my_custom_name

lpad get_fws -q ‘{“spec.my_id”:7}’

I prefer not to have the user be setting fw_ids as this could cause issues. For example, FWS would have to keep track of every id that the user has already allocated in order to make sure that the FWS system doesn’t assign a new id that conflicts with a previous user id (instead of just incrementing a counter of FW ids). Generally, anything you want to do with ids could be done with other things like FW name or a spec field instead.

On Friday, February 8, 2019 at 4:34:44 PM UTC-8, [email protected] wrote:

Just to confirm, this works very well for the most part. I tried it on a Workflow with Workflow.from_file as below.

There is one quirk. It seems that the fw_id for the added yaml have to be negative. The links still work, so presumably they are adjusted the same way. My one concern is this … won’t this make it hard to use capabilities that are based on id numbers (things that have -i options)?

I assume this is happening to prevent collisions. I am using a capability that would normally be used for dynamic additions, and in that case (like the Fibonacci,adder) the user would typically not be able to anticipate or know the fw_id and use them for things like fizzled workflows. But for a parsed file like I’ve got, presumably it does make sense and the user is going to be counting on it. Would it. make sense to let the user supply a positive id if she wants? If she supplies a positive one it would be used unless it is not unique, then the workflow fails? .If it is negative, behavior would be the same as now?

Or is there more to it?

class AddYAMLToFlowTask(FiretaskBase):
_fw_name = “AddYAMLToFlowTask”

def run_task(self, fw_spec):
fname = fw_spec[‘filename’]
print(“Filename is: {}”.format(fname))
new_fw = Workflow.from_file(fname)
return FWAction(stored_data={‘launchyaml’: fname}, additions=new_fw)

``

On Friday, February 8, 2019 at 10:56:15 AM UTC-8, Anubhav Jain wrote:

Hi

If I understand correctly, at the end of running TemplateWriterTask, you will have an output file on the file system called “filled.yaml” representing a Firework that you’d like to be added to the workflow (automatically).

My suggestion is that you add a second Firetask (within the same Firework) that runs immediately after the TemplateWriterTask. The job of this FireTask will be to read “filled.yaml” (convert the YAML to a Firework, using Firework.from_file(“filled.yaml”)) and then return a FWAction that adds this Firework to the workflow. You can either write up a Python function to do this and use PyTask for this purpose, or you could create a brand new FireTask whose job is to read a file and add it to the workflow. I’d probably suggest the latter.

Hope this helps …

On Friday, February 8, 2019 at 9:48:54 AM UTC-8, [email protected] wrote:

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``

Hi

If you work within Python, you don’t need to specify any fw ids. When making connections, you can use the “parents” kwarg to directly reference a Firework object higher up in the code (this is covered in the docs).

As for removing fw_ids from YAML file specification, or allowing names to replace them, honestly I don’t have it anywhere on the roadmap for FWS. If you’d like to take a stab at implementing it’s OK by me. The “parents” version of the connections would map back more naturally to the Python code than specifying “children”, since parents is already a kwarg of the Firework object.

Note that while there is a restriction on fw_ids being unique, there is no restriction on names being unique. So anyone using names as an identifier should be careful not to create duplicates as FWS will not check it for you.

···

On Sunday, February 10, 2019 at 5:22:17 PM UTC-8, [email protected] wrote:

OK, I appreciate the idea of reducing need to deal with fw_id. In that spirit though, two suggestions that would take that a step further.

  1. It would be nice to not have to specify fw_id. As far as I can tell, in my Workflow.from_file context I need to put in a negative number or I get a complaint that the id must be negative…
  1. Given we omit fw_id, it would be nice if we had one of the following two designs for parsing links from the yaml file:

a. links could be be based on name or other spec field:

links:

{name : step1}

  • {name : step2}

b. [parents and children could be parsed from the file (or maybe only one or the other to prevent conflicts):

fws:

  • name: step1

children: {name : step2}

spec:

_tasks:

  • _fw_name: PyTask

func: simple.echo

args:

  • hello

What do you think?

On Friday, February 8, 2019 at 8:28:24 PM UTC-8, Anubhav Jain wrote:

I would suggest that you use the “name” or some key you add in the spec (e.g. spec.your_id) to help keep track of your FireWorks. For example, you could do a search like:

lpad get_fws -n my_custom_name

lpad get_fws -q ‘{“spec.my_id”:7}’

I prefer not to have the user be setting fw_ids as this could cause issues. For example, FWS would have to keep track of every id that the user has already allocated in order to make sure that the FWS system doesn’t assign a new id that conflicts with a previous user id (instead of just incrementing a counter of FW ids). Generally, anything you want to do with ids could be done with other things like FW name or a spec field instead.

On Friday, February 8, 2019 at 4:34:44 PM UTC-8, [email protected] wrote:

Just to confirm, this works very well for the most part. I tried it on a Workflow with Workflow.from_file as below.

There is one quirk. It seems that the fw_id for the added yaml have to be negative. The links still work, so presumably they are adjusted the same way. My one concern is this … won’t this make it hard to use capabilities that are based on id numbers (things that have -i options)?

I assume this is happening to prevent collisions. I am using a capability that would normally be used for dynamic additions, and in that case (like the Fibonacci,adder) the user would typically not be able to anticipate or know the fw_id and use them for things like fizzled workflows. But for a parsed file like I’ve got, presumably it does make sense and the user is going to be counting on it. Would it. make sense to let the user supply a positive id if she wants? If she supplies a positive one it would be used unless it is not unique, then the workflow fails? .If it is negative, behavior would be the same as now?

Or is there more to it?

class AddYAMLToFlowTask(FiretaskBase):
_fw_name = “AddYAMLToFlowTask”

def run_task(self, fw_spec):
fname = fw_spec[‘filename’]
print(“Filename is: {}”.format(fname))
new_fw = Workflow.from_file(fname)
return FWAction(stored_data={‘launchyaml’: fname}, additions=new_fw)

``

On Friday, February 8, 2019 at 10:56:15 AM UTC-8, Anubhav Jain wrote:

Hi

If I understand correctly, at the end of running TemplateWriterTask, you will have an output file on the file system called “filled.yaml” representing a Firework that you’d like to be added to the workflow (automatically).

My suggestion is that you add a second Firetask (within the same Firework) that runs immediately after the TemplateWriterTask. The job of this FireTask will be to read “filled.yaml” (convert the YAML to a Firework, using Firework.from_file(“filled.yaml”)) and then return a FWAction that adds this Firework to the workflow. You can either write up a Python function to do this and use PyTask for this purpose, or you could create a brand new FireTask whose job is to read a file and add it to the workflow. I’d probably suggest the latter.

Hope this helps …

On Friday, February 8, 2019 at 9:48:54 AM UTC-8, [email protected] wrote:

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``

I’ll likely give it a go unless my collaborators prefer the “global configuration” approach I posted on another thread. That isn’t as powerful an approach, so I think we need to get templatized yaml working. Parents is fine. As far as uniqueness of names, I could either make it break or make it have multiple parents in case of a name clash and/or also prohibit clashes within the yaml file.

Eli

···

On Tuesday, February 12, 2019 at 1:50:20 PM UTC-8, Anubhav Jain wrote:

Hi

If you work within Python, you don’t need to specify any fw ids. When making connections, you can use the “parents” kwarg to directly reference a Firework object higher up in the code (this is covered in the docs).

As for removing fw_ids from YAML file specification, or allowing names to replace them, honestly I don’t have it anywhere on the roadmap for FWS. If you’d like to take a stab at implementing it’s OK by me. The “parents” version of the connections would map back more naturally to the Python code than specifying “children”, since parents is already a kwarg of the Firework object.

Note that while there is a restriction on fw_ids being unique, there is no restriction on names being unique. So anyone using names as an identifier should be careful not to create duplicates as FWS will not check it for you.

On Sunday, February 10, 2019 at 5:22:17 PM UTC-8, [email protected] wrote:

OK, I appreciate the idea of reducing need to deal with fw_id. In that spirit though, two suggestions that would take that a step further.

  1. It would be nice to not have to specify fw_id. As far as I can tell, in my Workflow.from_file context I need to put in a negative number or I get a complaint that the id must be negative…
  1. Given we omit fw_id, it would be nice if we had one of the following two designs for parsing links from the yaml file:

a. links could be be based on name or other spec field:

links:

{name : step1}

  • {name : step2}

b. [parents and children could be parsed from the file (or maybe only one or the other to prevent conflicts):

fws:

  • name: step1

children: {name : step2}

spec:

_tasks:

  • _fw_name: PyTask

func: simple.echo

args:

  • hello

What do you think?

On Friday, February 8, 2019 at 8:28:24 PM UTC-8, Anubhav Jain wrote:

I would suggest that you use the “name” or some key you add in the spec (e.g. spec.your_id) to help keep track of your FireWorks. For example, you could do a search like:

lpad get_fws -n my_custom_name

lpad get_fws -q ‘{“spec.my_id”:7}’

I prefer not to have the user be setting fw_ids as this could cause issues. For example, FWS would have to keep track of every id that the user has already allocated in order to make sure that the FWS system doesn’t assign a new id that conflicts with a previous user id (instead of just incrementing a counter of FW ids). Generally, anything you want to do with ids could be done with other things like FW name or a spec field instead.

On Friday, February 8, 2019 at 4:34:44 PM UTC-8, [email protected] wrote:

Just to confirm, this works very well for the most part. I tried it on a Workflow with Workflow.from_file as below.

There is one quirk. It seems that the fw_id for the added yaml have to be negative. The links still work, so presumably they are adjusted the same way. My one concern is this … won’t this make it hard to use capabilities that are based on id numbers (things that have -i options)?

I assume this is happening to prevent collisions. I am using a capability that would normally be used for dynamic additions, and in that case (like the Fibonacci,adder) the user would typically not be able to anticipate or know the fw_id and use them for things like fizzled workflows. But for a parsed file like I’ve got, presumably it does make sense and the user is going to be counting on it. Would it. make sense to let the user supply a positive id if she wants? If she supplies a positive one it would be used unless it is not unique, then the workflow fails? .If it is negative, behavior would be the same as now?

Or is there more to it?

class AddYAMLToFlowTask(FiretaskBase):
_fw_name = “AddYAMLToFlowTask”

def run_task(self, fw_spec):
fname = fw_spec[‘filename’]
print(“Filename is: {}”.format(fname))
new_fw = Workflow.from_file(fname)
return FWAction(stored_data={‘launchyaml’: fname}, additions=new_fw)

``

On Friday, February 8, 2019 at 10:56:15 AM UTC-8, Anubhav Jain wrote:

Hi

If I understand correctly, at the end of running TemplateWriterTask, you will have an output file on the file system called “filled.yaml” representing a Firework that you’d like to be added to the workflow (automatically).

My suggestion is that you add a second Firetask (within the same Firework) that runs immediately after the TemplateWriterTask. The job of this FireTask will be to read “filled.yaml” (convert the YAML to a Firework, using Firework.from_file(“filled.yaml”)) and then return a FWAction that adds this Firework to the workflow. You can either write up a Python function to do this and use PyTask for this purpose, or you could create a brand new FireTask whose job is to read a file and add it to the workflow. I’d probably suggest the latter.

Hope this helps …

On Friday, February 8, 2019 at 9:48:54 AM UTC-8, [email protected] wrote:

I am trying to create a workflow with firework specs that depends on user parameters and executes fireworks-unaware python functions. What is different about my use case compared to the examples is that instead of filling input files, I want to fill in yaml files that are then used to construct the rest of the workflow.

The sample below is a toy illustration: template_task.yaml fills the template and the filled.yaml represents the final workflow. If I do it in two steps (add-launch-add-launch) the example is trivial.

What I would like is to make this a single dynamic workflow, so that after filling in the context, the yaml workflow is parsed and added to the workflow. I imagine that this would require that I subclass TemplateWriterTask, intercept the FWAction and add the new firework or workflow using the "additions’ field.

Can someone tell me the correct way to construct the new fireworks or workflow from my filled yaml file? Is it OK if I just intercept the FWAction of the parent class and set additions field? Can I parse multiple fireworks from one filled yaml file and express links between them? Could the template be a template for a workflow?

If it isn’t feasible, I’ll guess I’ll write python wrapper around fireworks that does the add-launch-add-launch sequence, but I thought this might be an interesting capability. - Thanks

===== file template.yaml

spec:
_tasks:

  • _fw_name: PyTask
    func: simple.echo
    args:
    • {{arg1}}

==== file template_task.yaml
spec:
_tasks:

  • _fw_name: TemplateWriterTask
    context:
    arg1: hello
    output_file: filled.yaml
    template_file: template.yaml

===== file simple.py
def echo(arg1):
print(“Arg is {}”.format(arg1))

``