Rapidfire not working correctly with MOAB queue adapter

FireWorks
1

Dear Fireworks Community,

Im having a tough time trying to get qlaunch rapidfire to run correctly with the MOAB queue_adapter, fireworks seems to exit before submitting all the FWs in the launchpad. For example,

Let’s say I have 4 FWs to run, and say I want to run 2 jobs at a time in the queue, also I would like qlaunch to stop once all the FWs in the launchpad are submitted to the queue. If there’s an option to make fireworks do this in the background that would be great as well.

I tried with,

$ qlaunch rapidfire -m 2

The outputs of the lpad get_fws -d less before and after submission are copied below, looks like only the first 2 FWs were submitted.

Is that an error or am I missing something here…?

Thanks and regards.

Arun

[agp971@quser13 queue_testing]$ lpad get_fws -d less

successfully loaded your custom FW_config.yaml!

[

  • {*

  •    "fw_id": 1,*

  •    "created_on": "2018-02-07T04:20:22.775372",*

  •    "updated_on": "2018-02-07T04:20:22.775534",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*

  • },*

  • {*

  •    "fw_id": 2,*

  •    "created_on": "2018-02-07T04:20:26.960880",*

  •    "updated_on": "2018-02-07T04:20:26.961029",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*

  • },*

  • {*

  •    "fw_id": 3,*

  •    "created_on": "2018-02-07T04:20:29.331432",*

  •    "updated_on": "2018-02-07T04:20:29.331600",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*

  • },*

  • {*

  •    "fw_id": 4,*

  •    "created_on": "2018-02-07T04:20:31.677318",*

  •    "updated_on": "2018-02-07T04:20:31.677489",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*

  • }*

]

[agp971@quser13 queue_testing]$ qlaunch rapidfire -m 2

successfully loaded your custom FW_config.yaml!

2018-02-06 22:21:34,193 INFO getting queue adapter

2018-02-06 22:21:34,194 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904

2018-02-06 22:21:34,558 INFO The number of jobs currently in the queue is: 0

2018-02-06 22:21:34,560 INFO 0 jobs in queue. Maximum allowed by user: 2

2018-02-06 22:21:34,650 INFO Launching a rocket!

2018-02-06 22:21:34,734 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-34-733733

2018-02-06 22:21:34,734 INFO moving to launch_dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-34-733733

/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/queue/queue_adapter.py:142: UserWarning: Key logdir has been specified in qadapter but it is not present in template, please check template (/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/user_objects/queue_adapters/MOAB_template.txt) for supported keys.

  • .format(subs_key, self.template_file))*

2018-02-06 22:21:34,753 INFO submitting queue script

2018-02-06 22:21:35,111 INFO Job submission was successful and job_id is 22490305

2018-02-06 22:21:35,111 INFO Sleeping for 5 seconds…zzz…

2018-02-06 22:21:40,198 INFO Launching a rocket!

2018-02-06 22:21:40,281 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-40-279789

2018-02-06 22:21:40,281 INFO moving to launch_dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-40-279789

/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/queue/queue_adapter.py:142: UserWarning: Key logdir has been specified in qadapter but it is not present in template, please check template (/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/user_objects/queue_adapters/MOAB_template.txt) for supported keys.

  • .format(subs_key, self.template_file))*

2018-02-06 22:21:40,283 INFO submitting queue script

2018-02-06 22:21:40,698 INFO Job submission was successful and job_id is 22490306

2018-02-06 22:21:40,699 INFO Sleeping for 5 seconds…zzz…

[agp971@quser13 queue_testing]$ lpad get_fws -d less

successfully loaded your custom FW_config.yaml!

[

  • {*

  •    "fw_id": 1,*

  •    "created_on": "2018-02-07T04:20:22.775372",*

  •    "updated_on": "2018-02-07T04:22:47.032971",*

  •    "state": "COMPLETED",*

  •    "name": "Unnamed FW"*

  • },*

  • {*

  •    "fw_id": 2,*

  •    "created_on": "2018-02-07T04:20:26.960880",*

  •    "updated_on": "2018-02-07T04:22:47.155621",*

  •    "state": "COMPLETED",*

  •    "name": "Unnamed FW"*

  • },*

  • {*

  •    "fw_id": 3,*

  •    "created_on": "2018-02-07T04:20:29.331432",*

  •    "updated_on": "2018-02-07T04:20:29.331600",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*

  • },*

  • {*

  •    "fw_id": 4,*

  •    "created_on": "2018-02-07T04:20:31.677318",*

  •    "updated_on": "2018-02-07T04:20:31.677489",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*

  • }*

2

Hi Arun,

Can you clarify if these 4 Fireworks are independent (i.e., you have 4 different workflows) or if they are part of 2 workflows (i.e., 2 Fireworks per workflow)?

Anubhav

···

On Wed, Feb 7, 2018 at 9:30 AM, Arun Gopalan [email protected] wrote:

Dear Fireworks Community,

Im having a tough time trying to get qlaunch rapidfire to run correctly with the MOAB queue_adapter, fireworks seems to exit before submitting all the FWs in the launchpad. For example,

Let’s say I have 4 FWs to run, and say I want to run 2 jobs at a time in the queue, also I would like qlaunch to stop once all the FWs in the launchpad are submitted to the queue. If there’s an option to make fireworks do this in the background that would be great as well.

I tried with,

$ qlaunch rapidfire -m 2

The outputs of the lpad get_fws -d less before and after submission are copied below, looks like only the first 2 FWs were submitted.

Is that an error or am I missing something here…?

Thanks and regards.

Arun

[agp971@quser13 queue_testing]$ lpad get_fws -d less

successfully loaded your custom FW_config.yaml!

[

  • {*
  •    "fw_id": 1,*

  •    "created_on": "2018-02-07T04:20:22.775372",*

  •    "updated_on": "2018-02-07T04:20:22.775534",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 2,*

  •    "created_on": "2018-02-07T04:20:26.960880",*

  •    "updated_on": "2018-02-07T04:20:26.961029",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 3,*

  •    "created_on": "2018-02-07T04:20:29.331432",*

  •    "updated_on": "2018-02-07T04:20:29.331600",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 4,*

  •    "created_on": "2018-02-07T04:20:31.677318",*

  •    "updated_on": "2018-02-07T04:20:31.677489",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • }*

]

[agp971@quser13 queue_testing]$ qlaunch rapidfire -m 2

successfully loaded your custom FW_config.yaml!

2018-02-06 22:21:34,193 INFO getting queue adapter

2018-02-06 22:21:34,194 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904

2018-02-06 22:21:34,558 INFO The number of jobs currently in the queue is: 0

2018-02-06 22:21:34,560 INFO 0 jobs in queue. Maximum allowed by user: 2

2018-02-06 22:21:34,650 INFO Launching a rocket!

2018-02-06 22:21:34,734 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-34-733733

2018-02-06 22:21:34,734 INFO moving to launch_dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-34-733733

/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/queue/queue_adapter.py:142: UserWarning: Key logdir has been specified in qadapter but it is not present in template, please check template (/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/user_objects/queue_adapters/MOAB_template.txt) for supported keys.

  • .format(subs_key, self.template_file))*

2018-02-06 22:21:34,753 INFO submitting queue script

2018-02-06 22:21:35,111 INFO Job submission was successful and job_id is 22490305

2018-02-06 22:21:35,111 INFO Sleeping for 5 seconds…zzz…

2018-02-06 22:21:40,198 INFO Launching a rocket!

2018-02-06 22:21:40,281 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-40-279789

2018-02-06 22:21:40,281 INFO moving to launch_dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-40-279789

/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/queue/queue_adapter.py:142: UserWarning: Key logdir has been specified in qadapter but it is not present in template, please check template (/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/user_objects/queue_adapters/MOAB_template.txt) for supported keys.

  • .format(subs_key, self.template_file))*

2018-02-06 22:21:40,283 INFO submitting queue script

2018-02-06 22:21:40,698 INFO Job submission was successful and job_id is 22490306

2018-02-06 22:21:40,699 INFO Sleeping for 5 seconds…zzz…

[agp971@quser13 queue_testing]$ lpad get_fws -d less

successfully loaded your custom FW_config.yaml!

[

  • {*
  •    "fw_id": 1,*

  •    "created_on": "2018-02-07T04:20:22.775372",*

  •    "updated_on": "2018-02-07T04:22:47.032971",*

  •    "state": "COMPLETED",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 2,*

  •    "created_on": "2018-02-07T04:20:26.960880",*

  •    "updated_on": "2018-02-07T04:22:47.155621",*

  •    "state": "COMPLETED",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 3,*

  •    "created_on": "2018-02-07T04:20:29.331432",*

  •    "updated_on": "2018-02-07T04:20:29.331600",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 4,*

  •    "created_on": "2018-02-07T04:20:31.677318",*

  •    "updated_on": "2018-02-07T04:20:31.677489",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • }*

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/1e43fb84-e7dd-4a73-9a31-9e7328398b33%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

3

Looks like the full discussion was not posted to the list, but I’ve tried to improve the qlaunch rapidfire logic in FWS v1.6.6.

Arun can you try pulling the latest version of FWS and see if it helps? I didn’t have much of a chance to test it so please let us know if there are still problems.

···

On Wednesday, February 7, 2018 at 9:58:51 AM UTC-8, ajain wrote:

Hi Arun,

Can you clarify if these 4 Fireworks are independent (i.e., you have 4 different workflows) or if they are part of 2 workflows (i.e., 2 Fireworks per workflow)?

Anubhav

On Wed, Feb 7, 2018 at 9:30 AM, Arun Gopalan [email protected] wrote:

Dear Fireworks Community,

Im having a tough time trying to get qlaunch rapidfire to run correctly with the MOAB queue_adapter, fireworks seems to exit before submitting all the FWs in the launchpad. For example,

Let’s say I have 4 FWs to run, and say I want to run 2 jobs at a time in the queue, also I would like qlaunch to stop once all the FWs in the launchpad are submitted to the queue. If there’s an option to make fireworks do this in the background that would be great as well.

I tried with,

$ qlaunch rapidfire -m 2

The outputs of the lpad get_fws -d less before and after submission are copied below, looks like only the first 2 FWs were submitted.

Is that an error or am I missing something here…?

Thanks and regards.

Arun

[agp971@quser13 queue_testing]$ lpad get_fws -d less

successfully loaded your custom FW_config.yaml!

[

  • {*
  •    "fw_id": 1,*

  •    "created_on": "2018-02-07T04:20:22.775372",*

  •    "updated_on": "2018-02-07T04:20:22.775534",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 2,*

  •    "created_on": "2018-02-07T04:20:26.960880",*

  •    "updated_on": "2018-02-07T04:20:26.961029",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 3,*

  •    "created_on": "2018-02-07T04:20:29.331432",*

  •    "updated_on": "2018-02-07T04:20:29.331600",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 4,*

  •    "created_on": "2018-02-07T04:20:31.677318",*

  •    "updated_on": "2018-02-07T04:20:31.677489",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • }*

]

[agp971@quser13 queue_testing]$ qlaunch rapidfire -m 2

successfully loaded your custom FW_config.yaml!

2018-02-06 22:21:34,193 INFO getting queue adapter

2018-02-06 22:21:34,194 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904

2018-02-06 22:21:34,558 INFO The number of jobs currently in the queue is: 0

2018-02-06 22:21:34,560 INFO 0 jobs in queue. Maximum allowed by user: 2

2018-02-06 22:21:34,650 INFO Launching a rocket!

2018-02-06 22:21:34,734 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-34-733733

2018-02-06 22:21:34,734 INFO moving to launch_dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-34-733733

/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/queue/queue_adapter.py:142: UserWarning: Key logdir has been specified in qadapter but it is not present in template, please check template (/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/user_objects/queue_adapters/MOAB_template.txt) for supported keys.

  • .format(subs_key, self.template_file))*

2018-02-06 22:21:34,753 INFO submitting queue script

2018-02-06 22:21:35,111 INFO Job submission was successful and job_id is 22490305

2018-02-06 22:21:35,111 INFO Sleeping for 5 seconds…zzz…

2018-02-06 22:21:40,198 INFO Launching a rocket!

2018-02-06 22:21:40,281 INFO Created new dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-40-279789

2018-02-06 22:21:40,281 INFO moving to launch_dir /home/agp971/SCOTTY/fireworks/queue_testing/block_2018-02-07-04-21-34-193904/launcher_2018-02-07-04-21-40-279789

/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/queue/queue_adapter.py:142: UserWarning: Key logdir has been specified in qadapter but it is not present in template, please check template (/home/agp971/anaconda3/lib/python3.6/site-packages/fireworks/user_objects/queue_adapters/MOAB_template.txt) for supported keys.

  • .format(subs_key, self.template_file))*

2018-02-06 22:21:40,283 INFO submitting queue script

2018-02-06 22:21:40,698 INFO Job submission was successful and job_id is 22490306

2018-02-06 22:21:40,699 INFO Sleeping for 5 seconds…zzz…

[agp971@quser13 queue_testing]$ lpad get_fws -d less

successfully loaded your custom FW_config.yaml!

[

  • {*
  •    "fw_id": 1,*

  •    "created_on": "2018-02-07T04:20:22.775372",*

  •    "updated_on": "2018-02-07T04:22:47.032971",*

  •    "state": "COMPLETED",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 2,*

  •    "created_on": "2018-02-07T04:20:26.960880",*

  •    "updated_on": "2018-02-07T04:22:47.155621",*

  •    "state": "COMPLETED",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 3,*

  •    "created_on": "2018-02-07T04:20:29.331432",*

  •    "updated_on": "2018-02-07T04:20:29.331600",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • },*
  • {*
  •    "fw_id": 4,*

  •    "created_on": "2018-02-07T04:20:31.677318",*

  •    "updated_on": "2018-02-07T04:20:31.677489",*

  •    "state": "READY",*

  •    "name": "Unnamed FW"*
  • }*

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/1e43fb84-e7dd-4a73-9a31-9e7328398b33%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav