Using queue system in python script

Leip · January 30, 2019, 4:22pm

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

Anubhav_Jain · February 2, 2019, 1:02am

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

···

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Leip · February 4, 2019, 3:55pm

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

···

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Anubhav_Jain · February 4, 2019, 4:45pm

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

···

Best,
Anubhav

Leip · February 4, 2019, 4:54pm

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

···

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Leip · February 5, 2019, 2:44pm

I could (almost) make it run, but I have a connection problem with the mongo DB:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

I have this very simple code:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘password’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

The connection is working well when using it locally (with the default rapidfire). When I use the queue rapidfire, I get the above arror. Why?

···

Le lundi 4 février 2019 17:54:45 UTC+1, Leip a écrit :

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Anubhav_Jain · February 5, 2019, 2:51pm

Hi Leip

Where are you running the code when running queue rapidfire?

Your database credentials specify “localhost” with port “27017”. That means the database must be running on the same machine as where you run the script and open on port 27017. If you want to run the database on a different machine then where you run the script (very common), you need to specify your db host properly.

···

Best,
Anubhav

Leip · February 5, 2019, 3:13pm

The database is on a different machine. I give all the informations in the Launchpad instance, and I give this instance to rapidfire :

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,

                  port=27017,
                  name='fireworks',
                  username='fireworks',

password=‘password’)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

So it should work, no?

It’s working when using on my computer with fireworks.core.rapidfire. It’s not working with fireworks.queue.queue_launcher.rapidfire.

···

Le mardi 5 février 2019 15:51:58 UTC+1, ajain a écrit :

Hi Leip

Where are you running the code when running queue rapidfire?

Your database credentials specify “localhost” with port “27017”. That means the database must be running on the same machine as where you run the script and open on port 27017. If you want to run the database on a different machine then where you run the script (very common), you need to specify your db host properly.

On Tue, Feb 5, 2019 at 6:44 AM Leip [email protected] wrote:

I could (almost) make it run, but I have a connection problem with the mongo DB:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

I have this very simple code:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘password’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

The connection is working well when using it locally (with the default rapidfire). When I use the queue rapidfire, I get the above arror. Why?

Le lundi 4 février 2019 17:54:45 UTC+1, Leip a écrit :

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Anubhav_Jain · February 5, 2019, 3:26pm

You probably need to set your firewall settings which can be tricky in MongoDB

I’d say to first try connecting to your remote machine using the default MongoDB tools (eg mongo shell) and not FWS. This will make sure your database setup is correct. Once you have that working go back to FWS.

···

Best,
Anubhav

Leip · February 5, 2019, 3:35pm

The error message shows that FWS is trying to connect to localhost, not to foo.com (which is the correct URL for mongo, as stated in Launchpad parameters). So I’m pretty confident the problem comes from FWS, not from the connection to the remote mongo machine.

···

Le mardi 5 février 2019 16:26:19 UTC+1, ajain a écrit :

You probably need to set your firewall settings which can be tricky in MongoDB

I’d say to first try connecting to your remote machine using the default MongoDB tools (eg mongo shell) and not FWS. This will make sure your database setup is correct. Once you have that working go back to FWS.

On Tue, Feb 5, 2019 at 7:13 AM Leip [email protected] wrote:

The database is on a different machine. I give all the informations in the Launchpad instance, and I give this instance to rapidfire :

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,

                  port=27017,
                  name='fireworks',
                  username='fireworks',

password=‘password’)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

So it should work, no?

It’s working when using on my computer with fireworks.core.rapidfire. It’s not working with fireworks.queue.queue_launcher.rapidfire.

Le mardi 5 février 2019 15:51:58 UTC+1, ajain a écrit :

Hi Leip

Where are you running the code when running queue rapidfire?

Your database credentials specify “localhost” with port “27017”. That means the database must be running on the same machine as where you run the script and open on port 27017. If you want to run the database on a different machine then where you run the script (very common), you need to specify your db host properly.

On Tue, Feb 5, 2019 at 6:44 AM Leip [email protected] wrote:

I could (almost) make it run, but I have a connection problem with the mongo DB:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

I have this very simple code:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘password’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

The connection is working well when using it locally (with the default rapidfire). When I use the queue rapidfire, I get the above arror. Why?

Le lundi 4 février 2019 17:54:45 UTC+1, Leip a écrit :

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Anubhav_Jain · February 5, 2019, 5:24pm

Can you provide more details on the error?

Full stack trace (not a snippet) as well as full set of commands performed
Is the problem that it won’t submit jobs to the queue at all, or that when the jobs start running (after being submitted to the queue) they can’t connect to the database. At what point does the error occur?
Any preliminary debugging you did - e.g., inspecting the state of the launchpad object within the rapidfire function to check the host that is being queried

···

Best,
Anubhav

Leip · February 6, 2019, 8:46am

To summarize:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(’

echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

This previous script is working. But it is launched locally (not queued), so I modified it to change the rapidfire() function to use the queue system:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

This script is not working. The only differences are the import (I import the rapidfire from the queue package) and the rapidfire arguments. Rapidfire is using the same launchpad, with the same parameters to access the mongo DB. Those parameters are working with the previous version of the script, so I know there are good.

The full stack trace is on this paste (this is the paste for one job).

The jobs start running (after being submitted to the queue) but they can’t connect to the database.I can see them on the queue. The problem is that the jobs are trying to connect to the mongo DB using the wrong parameters: they try to connect to localhost, so I think they are not using the parameters given in the Launchpad.

I don’t know how to inspect the launchpad object within the rapidfire function, but I hope you’ll have enough information to help me. Thanks!

···

Le mardi 5 février 2019 18:25:17 UTC+1, ajain a écrit :

Can you provide more details on the error?

Full stack trace (not a snippet) as well as full set of commands performed

Is the problem that it won’t submit jobs to the queue at all, or that when the jobs start running (after being submitted to the queue) they can’t connect to the database. At what point does the error occur?

Any preliminary debugging you did - e.g., inspecting the state of the launchpad object within the rapidfire function to check the host that is being queried

On Tue, Feb 5, 2019 at 7:35 AM Leip [email protected] wrote:

The error message shows that FWS is trying to connect to localhost, not to foo.com (which is the correct URL for mongo, as stated in Launchpad parameters). So I’m pretty confident the problem comes from FWS, not from the connection to the remote mongo machine.

Le mardi 5 février 2019 16:26:19 UTC+1, ajain a écrit :

You probably need to set your firewall settings which can be tricky in MongoDB

I’d say to first try connecting to your remote machine using the default MongoDB tools (eg mongo shell) and not FWS. This will make sure your database setup is correct. Once you have that working go back to FWS.

On Tue, Feb 5, 2019 at 7:13 AM Leip [email protected] wrote:

The database is on a different machine. I give all the informations in the Launchpad instance, and I give this instance to rapidfire :

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,

                  port=27017,
                  name='fireworks',
                  username='fireworks',

password=‘password’)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

So it should work, no?

It’s working when using on my computer with fireworks.core.rapidfire. It’s not working with fireworks.queue.queue_launcher.rapidfire.

Le mardi 5 février 2019 15:51:58 UTC+1, ajain a écrit :

Hi Leip

Where are you running the code when running queue rapidfire?

Your database credentials specify “localhost” with port “27017”. That means the database must be running on the same machine as where you run the script and open on port 27017. If you want to run the database on a different machine then where you run the script (very common), you need to specify your db host properly.

On Tue, Feb 5, 2019 at 6:44 AM Leip [email protected] wrote:

I could (almost) make it run, but I have a connection problem with the mongo DB:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

I have this very simple code:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘password’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

The connection is working well when using it locally (with the default rapidfire). When I use the queue rapidfire, I get the above arror. Why?

Le lundi 4 février 2019 17:54:45 UTC+1, Leip a écrit :

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Leip · February 7, 2019, 8:07am

Any idea?

···

Le mercredi 6 février 2019 09:46:55 UTC+1, Leip a écrit :

To summarize:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(’

echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

This previous script is working. But it is launched locally (not queued), so I modified it to change the rapidfire() function to use the queue system:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

This script is not working. The only differences are the import (I import the rapidfire from the queue package) and the rapidfire arguments. Rapidfire is using the same launchpad, with the same parameters to access the mongo DB. Those parameters are working with the previous version of the script, so I know there are good.

The full stack trace is on this paste (this is the paste for one job).

The jobs start running (after being submitted to the queue) but they can’t connect to the database.I can see them on the queue. The problem is that the jobs are trying to connect to the mongo DB using the wrong parameters: they try to connect to localhost, so I think they are not using the parameters given in the Launchpad.

I don’t know how to inspect the launchpad object within the rapidfire function, but I hope you’ll have enough information to help me. Thanks!

Le mardi 5 février 2019 18:25:17 UTC+1, ajain a écrit :

Can you provide more details on the error?

Full stack trace (not a snippet) as well as full set of commands performed

Is the problem that it won’t submit jobs to the queue at all, or that when the jobs start running (after being submitted to the queue) they can’t connect to the database. At what point does the error occur?

Any preliminary debugging you did - e.g., inspecting the state of the launchpad object within the rapidfire function to check the host that is being queried

On Tue, Feb 5, 2019 at 7:35 AM Leip [email protected] wrote:

The error message shows that FWS is trying to connect to localhost, not to foo.com (which is the correct URL for mongo, as stated in Launchpad parameters). So I’m pretty confident the problem comes from FWS, not from the connection to the remote mongo machine.

Le mardi 5 février 2019 16:26:19 UTC+1, ajain a écrit :

You probably need to set your firewall settings which can be tricky in MongoDB

I’d say to first try connecting to your remote machine using the default MongoDB tools (eg mongo shell) and not FWS. This will make sure your database setup is correct. Once you have that working go back to FWS.

On Tue, Feb 5, 2019 at 7:13 AM Leip [email protected] wrote:

The database is on a different machine. I give all the informations in the Launchpad instance, and I give this instance to rapidfire :

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,

                  port=27017,
                  name='fireworks',
                  username='fireworks',

password=‘password’)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

So it should work, no?

It’s working when using on my computer with fireworks.core.rapidfire. It’s not working with fireworks.queue.queue_launcher.rapidfire.

Le mardi 5 février 2019 15:51:58 UTC+1, ajain a écrit :

Hi Leip

Where are you running the code when running queue rapidfire?

Your database credentials specify “localhost” with port “27017”. That means the database must be running on the same machine as where you run the script and open on port 27017. If you want to run the database on a different machine then where you run the script (very common), you need to specify your db host properly.

On Tue, Feb 5, 2019 at 6:44 AM Leip [email protected] wrote:

I could (almost) make it run, but I have a connection problem with the mongo DB:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

I have this very simple code:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘password’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

The connection is working well when using it locally (with the default rapidfire). When I use the queue rapidfire, I get the above arror. Why?

Le lundi 4 février 2019 17:54:45 UTC+1, Leip a écrit :

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

Anubhav_Jain · February 8, 2019, 6:46pm

Hi Leip

If you can submit OK, that means the queue launcher rapidfire is doing fine.

The problem stems from whatever you put as the “rocket_launch” command. Internally, what happens is that when your job starts running, it will just execute whatever is in the rocket_launch command. You can check for example your FW_submit.script which is the file created by the queue launcher - this will show you exactly what is being submitted and what the job is doing when it starts running.

Typically, the rocket_launch command looks something like “rlaunch -l path/to/my_launchpad.yaml singleshot” (there are many variations). However, if you do not specify the launchpad correctly, it will assume you are trying to connect to localhost 27017

There are two things you could do to fix

Update your rocket_launch to give a path to my_launchpad
Set up your FWConfig so that the default Launchpad being connected to is not localhost:27017 but rather some other database. See: https://materialsproject.github.io/fireworks/config_tutorial.html

Best

Anubhav

···

On Thursday, February 7, 2019 at 12:07:01 AM UTC-8, Leip wrote:

Any idea?

Le mercredi 6 février 2019 09:46:55 UTC+1, Leip a écrit :

To summarize:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(’

echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

This previous script is working. But it is launched locally (not queued), so I modified it to change the rapidfire() function to use the queue system:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

This script is not working. The only differences are the import (I import the rapidfire from the queue package) and the rapidfire arguments. Rapidfire is using the same launchpad, with the same parameters to access the mongo DB. Those parameters are working with the previous version of the script, so I know there are good.

The full stack trace is on this paste (this is the paste for one job).

The jobs start running (after being submitted to the queue) but they can’t connect to the database.I can see them on the queue. The problem is that the jobs are trying to connect to the mongo DB using the wrong parameters: they try to connect to localhost, so I think they are not using the parameters given in the Launchpad.

I don’t know how to inspect the launchpad object within the rapidfire function, but I hope you’ll have enough information to help me. Thanks!

Le mardi 5 février 2019 18:25:17 UTC+1, ajain a écrit :

Can you provide more details on the error?

Full stack trace (not a snippet) as well as full set of commands performed

Is the problem that it won’t submit jobs to the queue at all, or that when the jobs start running (after being submitted to the queue) they can’t connect to the database. At what point does the error occur?

Any preliminary debugging you did - e.g., inspecting the state of the launchpad object within the rapidfire function to check the host that is being queried

On Tue, Feb 5, 2019 at 7:35 AM Leip [email protected] wrote:

The error message shows that FWS is trying to connect to localhost, not to foo.com (which is the correct URL for mongo, as stated in Launchpad parameters). So I’m pretty confident the problem comes from FWS, not from the connection to the remote mongo machine.

Le mardi 5 février 2019 16:26:19 UTC+1, ajain a écrit :

You probably need to set your firewall settings which can be tricky in MongoDB

I’d say to first try connecting to your remote machine using the default MongoDB tools (eg mongo shell) and not FWS. This will make sure your database setup is correct. Once you have that working go back to FWS.

On Tue, Feb 5, 2019 at 7:13 AM Leip [email protected] wrote:

The database is on a different machine. I give all the informations in the Launchpad instance, and I give this instance to rapidfire :

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,

                  port=27017,
                  name='fireworks',
                  username='fireworks',

password=‘password’)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

So it should work, no?

It’s working when using on my computer with fireworks.core.rapidfire. It’s not working with fireworks.queue.queue_launcher.rapidfire.

Le mardi 5 février 2019 15:51:58 UTC+1, ajain a écrit :

Hi Leip

Where are you running the code when running queue rapidfire?

Your database credentials specify “localhost” with port “27017”. That means the database must be running on the same machine as where you run the script and open on port 27017. If you want to run the database on a different machine then where you run the script (very common), you need to specify your db host properly.

On Tue, Feb 5, 2019 at 6:44 AM Leip [email protected] wrote:

I could (almost) make it run, but I have a connection problem with the mongo DB:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

I have this very simple code:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

set up the LaunchPad and reset it

launchpad = LaunchPad(host=‘rs0/foo.com:27070’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘password’)
launchpad.reset(’’, require_password=False)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch="rlaunch singleshot "), reserve=False)

The connection is working well when using it locally (with the default rapidfire). When I use the queue rapidfire, I get the above arror. Why?

Le lundi 4 février 2019 17:54:45 UTC+1, Leip a écrit :

I could use this queue adapter: https://github.com/materialsproject/fireworks/blob/master/fw_tutorials/queue/qadapter_SGE.yaml

But then I would need path/to/my_fworker.yaml and path/to/my_launchpad.yaml. As I use a single Python script, I don’t have any of those paths.

Do I have to completely change the way I use Fireworks and use yaml files instead of Python? (That would be sad)

Le lundi 4 février 2019 17:46:20 UTC+1, ajain a écrit :

Hi Leip

I’d say you have two options:

(probably the easiest) If you already have a queueadapter file that works on your filesystem (i.e., my_qadapter.yaml), you can use the following as your qadapter object (instead of trying to create a CommonAdapter object yourself):

from fireworks.utilities.fw_serializers import load_object_from_file

qadapter = load_object_from_file(“path/to/my_qadapter.yaml”)

If you don’t want to load the queue parameters from an external file, you need to fill in more parameters for the CommonAdapter. Namely, for all the keys that currently in your “my_qadapter.yaml” file (like the “rocket_launch” key) and that don’t start with an underscore, you need to feed them as kwargs. e.g., CommonAdapter(“SGE”, “fireworks_queue”, rocket_launch=“adsfasf”, etc. etc.)

On Mon, Feb 4, 2019 at 7:55 AM Leip [email protected] wrote:

Hi,

So I did that:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.queue.queue_launcher import rapidfire
from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.core.fworker import FWorker

[…]

rapidfire(launchpad, FWorker(), CommonAdapter(“SGE”, “fireworks_queue”))

But then:

python3 01_workflow_pipeline.py
2019-02-04 16:49:33,920 INFO Performing db tune-up
2019-02-04 16:49:33,932 INFO LaunchPad was RESET.
2019-02-04 16:49:33,936 INFO Added a workflow. id_map: {-3: 1, -2: 2, -1: 3}
2019-02-04 16:49:33,936 INFO getting queue adapter
2019-02-04 16:49:33,940 INFO Found previous block, using fireworks/workflows/06_conf_workflow/block_2019-02-04-14-31-37-057502
2019-02-04 16:49:34,126 INFO 0 jobs in queue. Maximum allowed by user: 0
2019-02-04 16:49:34,379 INFO Launching a rocket!
2019-02-04 16:49:34,395 ERROR ----|vvv|----
2019-02-04 16:49:34,396 ERROR Error with queue launcher rapid fire!
2019-02-04 16:49:34,398 ERROR Traceback (most recent call last):
File “fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

2019-02-04 16:49:34,398 ERROR ----|^^^|----

Do you have any idea why?

Thanks a lot!

Le samedi 2 février 2019 02:02:47 UTC+1, ajain a écrit :

Hi Leip

To use it with a queue system, change the “rapidfire” to be the qadaptor version of rapidfire

from fireworks.core.queue_launcher import rapidfire

That queue_launcher rapidfire function requires an argument for the qadaptor, which can be a Python object for your SGE queue adapter (e.g., CommonAdapter())

Best,

Anubhav

On Wed, Jan 30, 2019 at 8:22 AM Leip [email protected] wrote:

Hi,

I’ve got a very simple workflow written in Python only:

from fireworks import Firework, Workflow, LaunchPad, ScriptTask, PyTask
from fireworks.core.rocket_launcher import rapidfire

launchpad = LaunchPad(host=‘foo.com’,
port=27017,
name=‘fireworks’,
username=‘fireworks’,
password=‘foobar’)

create the individual FireWorks and Workflow

fw1 = Firework(ScriptTask.from_str(‘echo hello’), name=“hello”)
fw2 = Firework(ScriptTask.from_str(‘echo stuff’), name=“stuff”)
fw3 = Firework(ScriptTask.from_str(‘echo goodbye’), name=“goodbye”)
wf = Workflow([fw1, fw2, fw3], {fw1: fw2, fw2: fw3}, name=“Basic pipeline workflow”)

store workflow and launch it locally

launchpad.add_wf(wf)
rapidfire(launchpad)

It is working well (but again, it’s very simple). Now I’d like to use it with a queue system. We are using SGE (UGE actually, but it should be the same). The documentation is only using yaml files. Is there any way to include the SGE worker description in the python file (or, if needed, in a yaml file read by the python script)? Ultimately, I’d like to keep using this workflow by simply running “python3 my_workflow.py”.

Thanks!

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

melkhatibr · February 8, 2019, 11:27pm

Hi,

I am getting a similar issue. I am able to run my script locally with rapidfire imported fromfireworks.core.rocket_launcher. When I try the same from fireworks.queue.queue_launcher` it fails. Below I describe what I do:

I created a FW_config.yaml file pointing out to the different

LAUNCHPAD_LOC: launchpad.yaml
FWORKER_LOC: fowrker.yaml
QUEUEADAPTER_LOC: slurm.yaml

``

In my script I have:

from fireworks.user_objects.queue_adapters.common_adapter import CommonAdapter
from fireworks.queue.queue_launcher import rapidfire

def create_queue_adapater(q_type, template):
slurm_adapter = CommonAdapter(q_type=q_type,
template_file=template,
reserve=True)
return slurm_adapter

lpad = LaunchPad().auto_load()
fworker = FWorker().auto_load()
queue = create_queue_adapater(‘SLURM’, template) # There is no auto_load() classmethod for CommonAdapter()

rapidfire(lpad, fworker, queue)

``

If I print lpad and fworker I can clearly see that everything was loaded according to my FW_config but queue is returning:

CommonAdapter(None, {‘reserve’: True})

``

Calling rapidfire fails with a KeyError:

2019-02-08 15:24:44,282 ERROR Traceback (most recent call last):
File “/global/homes/m/melkhati/.conda/envs/metatlas/lib/python3.7/site-packages/fireworks/queue/queue_launcher.py”, line 231, in rapidfire
strm_lvl, True, fill_mode)
File “/global/homes/m/melkhati/.conda/envs/metatlas/lib/python3.7/site-packages/fireworks/queue/queue_launcher.py”, line 63, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: ‘rocket_launch’

``

Any idea why is this happening?

Best,

···

–

Muammar

Anubhav_Jain · February 9, 2019, 6:00pm

Hi Muammar,

This might be best to follow up on a separate thread, but the easiest way to load any FWSerializable object from a file is simply something like:

from fireworks.utilities.fw_serializers import load_object_from_file

queueadapter = load_object_from_file(queueadapter_file)

If you wanted to make your existing code work, note that your “template_file” should point to “fireworks/user_objects/queue_adapters/SLURM_template.txt” and your **kwargs should include all the variable substitutions into that template - things like the rocket_launch key that would normally be filled by your my_qadapter.yaml. Note that the “reserve” option is not one of the template keywords, and is instead a keywoard arg into the rapidfire() function.

···

Best,
Anubhav