error in "Launch a single job through a queue" tutorial

FireWorks
1

I get an error in following command:


$ qlaunch singleshot

2016-09-14 16:03:28,156 INFO moving to launch_dir /home/python/queue_tests

2016-09-14 16:03:28,162 INFO submitting queue script

2016-09-14 16:03:28,246 ERROR ----|vvv|----

2016-09-14 16:03:28,246 ERROR Could not parse job id following sbatch due to error a bytes-like object is required, not 'str'...

2016-09-14 16:03:28,249 ERROR Traceback (most recent call last):

File "/home/python/sf_box/fireworks/fireworks/user_objects/queue_adapters/common_adapter.py", line 200, in submit_to_queue

job_id = self._parse_jobid(p.stdout.read())

File "/home/python/sf_box/fireworks/fireworks/user_objects/queue_adapters/common_adapter.py", line 71, in _parse_jobid

for l in output_str.split("\n"):

TypeError: a bytes-like object is required, not 'str'

2016-09-14 16:03:28,250 ERROR ----|^^^|----

2016-09-14 16:03:28,250 ERROR ----|vvv|----

2016-09-14 16:03:28,250 ERROR Error writing/submitting queue script!

2016-09-14 16:03:28,256 ERROR Traceback (most recent call last):

File "/home/python/sf_box/fireworks/fireworks/queue/queue_launcher.py", line 133, in launch_rocket_to_queue

launchpad.set_reservation_id(launch_id, reservation_id)

File "/home/python/sf_install/python/3.5.2/lib/python3.5/contextlib.py", line 77, in __exit__

self.gen.throw(type, value, traceback)

File "/home/python/sf_install/python/3.5.2/lib/python3.5/site-packages/monty-0.9.1-py3.5.egg/monty/os/__init__.py", line 32, in cd

yield

File "/home/python/sf_box/fireworks/fireworks/queue/queue_launcher.py", line 130, in launch_rocket_to_queue

raise RuntimeError('queue script could not be submitted, check queue '

RuntimeError: queue script could not be submitted, check queue script/queue adapter/queue server status!

2016-09-14 16:03:28,256 ERROR ----|^^^|----

But i get the correct result:


$ls

FW_job-2315017.error  FW.json           fw_test.yaml  logging          my_launchpad.yaml

FW_job-2315017.out    FW_submit.script  howdy.txt     my_fworker.yaml  my_qadapter.yaml

$ cat howdy.txt

howdy, your job launched successfully!

$ cat FW_job-2315017.out

2016-09-14 16:03:23,198 INFO Hostname/IP lookup (this will take a few seconds)

2016-09-14 16:03:23,201 INFO Launching Rocket

2016-09-14 16:03:24,258 INFO RUNNING fw_id: 1 in directory: /home/queue_tests

2016-09-14 16:03:24,265 INFO Task started: ScriptTask.

2016-09-14 16:03:24,283 INFO Task completed: ScriptTask

2016-09-14 16:03:24,590 INFO Rocket finished

$ cat my_qadapter.yaml

_fw_name: CommonAdapter

_fw_q_type: SLURM

rocket_launch: rlaunch -w /home/python/queue_tests/my_fworker.yaml -l /home/python/queue_tests/my_launchpad.yaml singleshot

ntasks: 1

cpus_per_task: 1

ntasks_per_node: 1

walltime: '00:02:00'

queue: null

account: null

job_name: null

logdir: /home/python/queue_tests/logging

pre_rocket: null

post_rocket: null

#You can override commands by uncommenting and changing the following lines:

#_q_commands_override:

#submit_cmd: my_qsubmit

#status_cmd: my_qstatus

#You can also supply your own template by uncommenting and changing the following line:

#template_file: /full/path/to/template

Am i make mistakes?

2

Hi Pin,

First - can you make sure you have the latest FireWorks? If you are using Py3, we just patched some issues related to that recently.

Second, I don’t think you are doing anything wrong. After submitting a job to SLURM, FireWorks expects that SLURM will return back a response in a particular format and try to parse the job id from that format. In this case, the relevant code is in:

fireworks.user_objects.queue_adapters.common_adapter.CommonAdapter#_parse_jobid

i.e.

    def _parse_jobid(self, output_str):
if self.q_type == "SLURM":
for l in output_str.split("\n"):
if l.startswith("Submitted batch job"):
return int(l.split()[-1])

​

i.e. in this case, FireWorks expects that SLURM will return a response that looks like

“Submitted batch job 1”

where 1 is the job id, and then parse that the job_id is 1. It will then store that information in the database when in reservation mode.

The job id parsing happens after the job is already submitted to the queue. Your jobs should still run OK as you are seeing.

But, the error is annoying so we should fix it. Let me know some more details of your problem so we can try to look into it. e.g., what version of FWS are you running, what version of Python are you running, and what text does SLURM return back when you submit a job (normally, without FWS).

When you submit a job to SLURM, what is the output that is returned by your SLURM

···

On Wednesday, September 14, 2016 at 7:31:51 PM UTC-7, pin chen wrote:

I get an error in following command:

$ qlaunch singleshot

2016-09-14 16:03:28,156 INFO moving to launch_dir /home/python/queue_tests

2016-09-14 16:03:28,162 INFO submitting queue script

2016-09-14 16:03:28,246 ERROR ----|vvv|----

2016-09-14 16:03:28,246 ERROR Could not parse job id following sbatch due to error a bytes-like object is required, not ‘str’…

2016-09-14 16:03:28,249 ERROR Traceback (most recent call last):

File “/home/python/sf_box/fireworks/fireworks/user_objects/queue_adapters/common_adapter.py”, line 200, in submit_to_queue

job_id = self._parse_jobid(p.stdout.read())

File “/home/python/sf_box/fireworks/fireworks/user_objects/queue_adapters/common_adapter.py”, line 71, in _parse_jobid

for l in output_str.split("\n"):

TypeError: a bytes-like object is required, not ‘str’

2016-09-14 16:03:28,250 ERROR ----|^^^|----

2016-09-14 16:03:28,250 ERROR ----|vvv|----

2016-09-14 16:03:28,250 ERROR Error writing/submitting queue script!

2016-09-14 16:03:28,256 ERROR Traceback (most recent call last):

File “/home/python/sf_box/fireworks/fireworks/queue/queue_launcher.py”, line 133, in launch_rocket_to_queue

launchpad.set_reservation_id(launch_id, reservation_id)

File “/home/python/sf_install/python/3.5.2/lib/python3.5/contextlib.py”, line 77, in exit

self.gen.throw(type, value, traceback)

File “/home/python/sf_install/python/3.5.2/lib/python3.5/site-packages/monty-0.9.1-py3.5.egg/monty/os/init.py”, line 32, in cd

yield

File “/home/python/sf_box/fireworks/fireworks/queue/queue_launcher.py”, line 130, in launch_rocket_to_queue

raise RuntimeError('queue script could not be submitted, check queue ’

RuntimeError: queue script could not be submitted, check queue script/queue adapter/queue server status!

2016-09-14 16:03:28,256 ERROR ----|^^^|----

But i get the correct result:

$ls

FW_job-2315017.error FW.json fw_test.yaml logging my_launchpad.yaml

FW_job-2315017.out FW_submit.script howdy.txt my_fworker.yaml my_qadapter.yaml

$ cat howdy.txt

howdy, your job launched successfully!

$ cat FW_job-2315017.out

2016-09-14 16:03:23,198 INFO Hostname/IP lookup (this will take a few seconds)

2016-09-14 16:03:23,201 INFO Launching Rocket

2016-09-14 16:03:24,258 INFO RUNNING fw_id: 1 in directory: /home/queue_tests

2016-09-14 16:03:24,265 INFO Task started: ScriptTask.

2016-09-14 16:03:24,283 INFO Task completed: ScriptTask

2016-09-14 16:03:24,590 INFO Rocket finished

$ cat my_qadapter.yaml

_fw_name: CommonAdapter

_fw_q_type: SLURM

rocket_launch: rlaunch -w /home/python/queue_tests/my_fworker.yaml -l /home/python/queue_tests/my_launchpad.yaml singleshot

ntasks: 1

cpus_per_task: 1

ntasks_per_node: 1

walltime: ‘00:02:00’

queue: null

account: null

job_name: null

logdir: /home/python/queue_tests/logging

pre_rocket: null

post_rocket: null

#You can override commands by uncommenting and changing the following lines:

#_q_commands_override:

#submit_cmd: my_qsubmit

#status_cmd: my_qstatus

#You can also supply your own template by uncommenting and changing the following line:

#template_file: /full/path/to/template

Am i make mistakes?