Queue, reservation, offline mode from Python

Anubhav_Jain · November 18, 2015, 7:18pm

Matthew, I am having trouble with the fireworkflows Google Group and my response got deleted. Here I will try to reproduce quickly.

The qadapter is a combination of q_type, template file, and substitutions into the template file represented by **kwargs. The template file is not the same as the qadapter.

If you want to see what the default template file (only) looks for SLURM, go here:

https://github.com/materialsproject/fireworks/blob/master/fireworks/user_objects/queue_adapters/SLURM_template.txt

Now to the solutions:

Create a complete “offline_adapter.yaml” file (including all keys, like _fw_q_type) and use:

qadapter = CommonAdapter.from_file(“offline_adapter.yaml”)

or

qadapter = load_obj_from_file(“offline_adapter.yaml”)

If you want the solution to look like your original, you can try a hack like this:

qadapter = CommonAdapter(q_type=‘SLURM’, **{“rocket_launch”: “rlaunch --offline”})

That will override the **kwargs to your desired rocket launch

Best

Anubhav

···

On Tue, Nov 17, 2015 at 11:56 AM, Matthew Cahn [email protected] wrote:

I’ve succeeded in running a job using a slurm queue in offline/reservation mode, from yaml files and the command line. Now I’d like to do the same thing from Python (and an additional template file). My code is below. It gives me the error:

./testqueue1.py
2015-11-17 14:24:13,635 INFO Added a workflow. id_map: {-1: 13}
Traceback (most recent call last):
File “./testqueue1.py”, line 31, in
queue_launcher.launch_rocket_to_queue(launchpad, fwReal, qadapter, launcher_dir=qdir, reserve=True, create_launcher_dir=True)
File “/path/to/fireworks/queue/queue_launcher.py”, line 59, in launch_rocket_to_queue
if ‘–offline’ in qadapter[‘rocket_launch’] and not reserve:
KeyError: u’rocket_launch’

What am I doing wrong? (Some of this is admittedly guesswork).

Thanks,
Matthew

offline_adapter.yaml:

rocket_launch: rlaunch --offline

testqueue1.py:

import os

from fireworks import Firework, LaunchPad, ScriptTask
from fireworks.queue import queue_launcher
from fireworks.core import fworker
from fireworks.user_objects.queue_adapters import common_adapter

qdir = ‘/path/to/queue_tests2’

launchpad = LaunchPad(host=u’’, port=27017)
firetask1 = ScriptTask.from_str(‘hostname > hostname.txt’)
firework = Firework([firetask1])
launchpad.add_wf(firework)
fwReal = fworker.FWorker()

qadapter = common_adapter.CommonAdapter(q_type=‘SLURM’, template_file=os.path.join(qdir, ‘offline_adapter.yaml’))

fwReserved, launchId = launchpad.reserve_fw(fworker.FWorker(), qdir)
queue_launcher.setup_offline_job(launchpad, fwReserved, launchId)

queue_launcher.launch_rocket_to_queue(launchpad, fwReal, qadapter, launcher_dir=qdir, reserve=True, create_launcher_dir=True)

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/764d8643-f40d-47c9-b248-41a53553272d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Matthew_Cahn · November 18, 2015, 8:17pm

Thanks very much. I also made a posting to the google group that seemed to disappear, which said that I’d figured out how to make the code work, by supplying the appropriate parameters. Also, calling launchpad.reserve_fw() and setup_offline_job() seem unnecessary. Below is the code that works. Thanks for your reply – it’s clearer now.

– Matthew

from fireworks import Firework, LaunchPad, ScriptTask

from fireworks.queue import queue_launcher

from fireworks.core import fworker

from fireworks.user_objects.queue_adapters import common_adapter

qdir = ‘/path/to/queue_tests2’

launchpad = LaunchPad(host=u’’, port=27017)

firetask1 = ScriptTask.from_str(‘hostname > hostname.txt’)

firework = Firework([firetask1])

launchpad.add_wf(firework)

worker = fworker.FWorker()

qadapter = common_adapter.CommonAdapter(q_type=‘SLURM’,

                                    **{'nodes': 1, 'ntasks': 1, 'ntasks_per_node': 1, 'cpus_per_task' : 2, 'walltime': '00:01:30',

                                       'rocket_launch': 'rlaunch -l /path/to/queue_tests2/my_launchpad.yaml singleshot --offline'})

queue_launcher.launch_rocket_to_queue(launchpad, worker, qadapter, reserve=True, create_launcher_dir=True)

···

On Wed, Nov 18, 2015 at 2:18 PM, Anubhav Jain [email protected] wrote: