exporting and sharing environment variables across multiple fireworks

Hi,

What is the correct way to export and share environment variables within a workflow?

My trial involved the following procedure:

  1. I export an environment variable FOO in a script export_env.sh in a FireTask that is part of FireWork 1 as follows:

export FOO=BAR

  1. I source that script in the corresponding yaml file

  2. In a consecutive FireWork I execute a script call_env.sh in a FireTask in which that same environment variable is called:

echo "value of FOO: "$FOO

The yaml file would look like this:

fws:

  • fw_id: 1

    spec:

    _tasks:

    • _fw_name: ScriptTask

      script: . ./export_env.sh

      use_shell: true

  • fw_id: 2

    spec:

    _tasks:

    • _fw_name: ScriptTask

      script: ./call_env.sh

      use_shell: true

As far as I experienced, environment variables set in 1 FireWork, are not retained in a consecutive FireWork, even when the native shell is used.

How do I go about setting environment variables that are accessible in multiple FireWorks and that can only be set dynamically, i.e. they cannot be exported beforehand?

Thanks!

This is a linux question as opposed to Fireworks. is there a reason you can’t write your export to a file, and then call “source” on that file in your “call_env.sh” ?
(Or, just put your variable in the file and read it directly?)

more generally: http://superuser.com/questions/636219/where-should-i-export-an-environment-variable-so-that-all-combinations-of-bash-d

Since you probably don’t want to modify these global files to get these dynamic exports to appear, I suggest you drop a file out or use the dynamic workflow functionality of fireworks to avoid this environment variable problem altogether:

https://pythonhosted.org/FireWorks/dynamic_wf_tutorial.html

···

On Saturday, April 2, 2016 at 10:09:38 AM UTC-4, Nick Vandewiele wrote:

Hi,

What is the correct way to export and share environment variables within a workflow?

My trial involved the following procedure:

  1. I export an environment variable FOO in a script export_env.sh in a FireTask that is part of FireWork 1 as follows:

export FOO=BAR

  1. I source that script in the corresponding yaml file
  1. In a consecutive FireWork I execute a script call_env.sh in a FireTask in which that same environment variable is called:

echo "value of FOO: "$FOO

The yaml file would look like this:

fws:

  • fw_id: 1

spec:

_tasks:
- _fw_name: ScriptTask
  script: . ./export_env.sh 
  use_shell: true
  • fw_id: 2

spec:

_tasks:
- _fw_name: ScriptTask
  script: ./call_env.sh
  use_shell: true

As far as I experienced, environment variables set in 1 FireWork, are not retained in a consecutive FireWork, even when the native shell is used.

How do I go about setting environment variables that are accessible in multiple FireWorks and that can only be set dynamically, i.e. they cannot be exported beforehand?

Thanks!

Hi all

Chris is correct that you should refresh your knowledge on sourcing vs executing bash scripts to set environment variables. Also, I would suggest going through the basic FWS tutorials. The one on how to pass data is:

https://pythonhosted.org/FireWorks/dynamic_wf_tutorial.html

Best,

Anubhav

···

On Monday, April 4, 2016 at 8:38:26 AM UTC-7, Chris H wrote:

This is a linux question as opposed to Fireworks. is there a reason you can’t write your export to a file, and then call “source” on that file in your “call_env.sh” ?
(Or, just put your variable in the file and read it directly?)

more generally: http://superuser.com/questions/636219/where-should-i-export-an-environment-variable-so-that-all-combinations-of-bash-d

Since you probably don’t want to modify these global files to get these dynamic exports to appear, I suggest you drop a file out or use the dynamic workflow functionality of fireworks to avoid this environment variable problem altogether:

https://pythonhosted.org/FireWorks/dynamic_wf_tutorial.html

https://github.com/materialsproject/fireworks/tree/master/fw_tutorials/dynamic_wf

On Saturday, April 2, 2016 at 10:09:38 AM UTC-4, Nick Vandewiele wrote:

Hi,

What is the correct way to export and share environment variables within a workflow?

My trial involved the following procedure:

  1. I export an environment variable FOO in a script export_env.sh in a FireTask that is part of FireWork 1 as follows:

export FOO=BAR

  1. I source that script in the corresponding yaml file
  1. In a consecutive FireWork I execute a script call_env.sh in a FireTask in which that same environment variable is called:

echo "value of FOO: "$FOO

The yaml file would look like this:

fws:

  • fw_id: 1

spec:

_tasks:
- _fw_name: ScriptTask
  script: . ./export_env.sh 
  use_shell: true
  • fw_id: 2

spec:

_tasks:
- _fw_name: ScriptTask
  script: ./call_env.sh
  use_shell: true

As far as I experienced, environment variables set in 1 FireWork, are not retained in a consecutive FireWork, even when the native shell is used.

How do I go about setting environment variables that are accessible in multiple FireWorks and that can only be set dynamically, i.e. they cannot be exported beforehand?

Thanks!