If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
https://pythonhosted.org/FireWorks/priority_tutorial.html
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sorry, I didn’t metion that I know the job priority can do that.
But our situation is different. We can’t set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one.
Do you have any suggessiont to deal with this? Thanks,
Ling
On Thursday, July 7, 2016 at 4:32:50 PM UTC-5, ajain wrote:
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi Ling
Sorry, but I do not understand your question or use case.
There are only a couple of ways that FireWorks can set job priority - either by setting the priority within each Firework (which can be done either at the time the workflow is created, or can be added at anytime using the lpad set_priority command), or by setting the order of workflow pulling as FIFO vs FILO in the FWConfig.
If users are submitting the workflows themselves, you can write a function that finds jobs with no priority set and assign priorities to each job such that “root” Fireworks have lower priority and “children” Fireworks have higher priority. Or you can write a function that takes in any workflow and adds priorities to each Firework, and ask users to run their workflows through that function before submitting them. e.g.
def add_priority(original_wf, root_priority, child_priority=None):
"""
Adds priority to a workflow
Args:
original_wf (Workflow): original WF
root_priority (int): priority of first (root) job(s)
child_priority(int): priority of all child jobs. Defaults to
root_priority
Returns:
(Workflow) priority-decorated workflow
"""
child_priority = child_priority or root_priority
root_fw_ids = original_wf.root_fw_ids
for fw in original_wf.fws:
if fw.fw_id in root_fw_ids:
fw.spec["_priority"] = root_priority
else:
fw.spec["_priority"] = child_priority
return original_wf
If you run a workflow through that function, and set the child priority higher than the root priority, it will finish that workflow before starting another workflow with the same root priority.
Best,
Anubhav
On Fri, Jul 8, 2016 at 9:16 AM, [email protected] wrote:
Sorry, I didn’t metion that I know the job priority can do that.
But our situation is different. We can’t set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one.
Do you have any suggessiont to deal with this? Thanks,
Ling
On Thursday, July 7, 2016 at 4:32:50 PM UTC-5, ajain wrote:
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6d79cc86-60a6-4b9e-9d66-cfdec60197e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Thanks a lot, Aunbhav.
I tried to set up FIFO in the FWConfig, and set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
Really appreciate your help.
Ling
On Friday, July 8, 2016 at 3:29:21 PM UTC-5, ajain wrote:
Hi Ling
Sorry, but I do not understand your question or use case.
There are only a couple of ways that FireWorks can set job priority - either by setting the priority within each Firework (which can be done either at the time the workflow is created, or can be added at anytime using the lpad set_priority command), or by setting the order of workflow pulling as FIFO vs FILO in the FWConfig.
If users are submitting the workflows themselves, you can write a function that finds jobs with no priority set and assign priorities to each job such that “root” Fireworks have lower priority and “children” Fireworks have higher priority. Or you can write a function that takes in any workflow and adds priorities to each Firework, and ask users to run their workflows through that function before submitting them. e.g.
def add_priority(original_wf, root_priority, child_priority=None): """ Adds priority to a workflow Args: original_wf (Workflow): original WF root_priority (int): priority of first (root) job(s) child_priority(int): priority of all child jobs. Defaults to root_priority Returns: (Workflow) priority-decorated workflow """ child_priority = child_priority or root_priority root_fw_ids = original_wf.root_fw_ids for fw in original_wf.fws: if fw.fw_id in root_fw_ids: fw.spec["_priority"] = root_priority else: fw.spec["_priority"] = child_priority return original_wf
If you run a workflow through that function, and set the child priority higher than the root priority, it will finish that workflow before starting another workflow with the same root priority.
Best,
Anubhav
On Fri, Jul 8, 2016 at 9:16 AM, [email protected] wrote:
Sorry, I didn’t metion that I know the job priority can do that.
But our situation is different. We can’t set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one.
Do you have any suggessiont to deal with this? Thanks,
Ling
On Thursday, July 7, 2016 at 4:32:50 PM UTC-5, ajain wrote:
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6d79cc86-60a6-4b9e-9d66-cfdec60197e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi Ling
set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
If the priority is set correctly (e.g., as the “_priority” key in the spec - don’t forget the underscore) - then this should not happen. Can you attach your code that you are using to test this?
Best,
Anubhav
On Fri, Jul 8, 2016 at 2:51 PM, [email protected] wrote:
Thanks a lot, Aunbhav.
I tried to set up FIFO in the FWConfig, and set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
Really appreciate your help.
Ling
On Friday, July 8, 2016 at 3:29:21 PM UTC-5, ajain wrote:
Hi Ling
Sorry, but I do not understand your question or use case.
There are only a couple of ways that FireWorks can set job priority - either by setting the priority within each Firework (which can be done either at the time the workflow is created, or can be added at anytime using the lpad set_priority command), or by setting the order of workflow pulling as FIFO vs FILO in the FWConfig.
If users are submitting the workflows themselves, you can write a function that finds jobs with no priority set and assign priorities to each job such that “root” Fireworks have lower priority and “children” Fireworks have higher priority. Or you can write a function that takes in any workflow and adds priorities to each Firework, and ask users to run their workflows through that function before submitting them. e.g.
def add_priority(original_wf, root_priority, child_priority=None): """ Adds priority to a workflow Args: original_wf (Workflow): original WF root_priority (int): priority of first (root) job(s) child_priority(int): priority of all child jobs. Defaults to root_priority Returns: (Workflow) priority-decorated workflow """ child_priority = child_priority or root_priority root_fw_ids = original_wf.root_fw_ids for fw in original_wf.fws: if fw.fw_id in root_fw_ids: fw.spec["_priority"] = root_priority else: fw.spec["_priority"] = child_priority return original_wf
If you run a workflow through that function, and set the child priority higher than the root priority, it will finish that workflow before starting another workflow with the same root priority.
Best,
Anubhav
On Fri, Jul 8, 2016 at 9:16 AM, [email protected] wrote:
Sorry, I didn’t metion that I know the job priority can do that.
But our situation is different. We can’t set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one.
Do you have any suggessiont to deal with this? Thanks,
Ling
On Thursday, July 7, 2016 at 4:32:50 PM UTC-5, ajain wrote:
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6d79cc86-60a6-4b9e-9d66-cfdec60197e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/d83f8daf-15e0-47b4-b1d0-c769c290dd64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
from fireworks import LaunchPad, PyTask, ScriptTask, Workflow
from fireworks.core.firework import FWAction, Firework, FireTaskBase, Workflow
from fireworks.core.rocket_launcher import rapidfire
from fireworks.utilities.fw_utilities import explicit_serialize
import time
@explicit_serialize
class exceptionTask(FireTaskBase):
def run_task(self, fw_spec):
raise NameError(“just to throw an exception”)
@explicit_serialize
class testTask(FireTaskBase):
def run_task(self, fw_spec):
fireworks = []
name1 = “sleep 1”
fw1 = Firework(ScriptTask.from_str(“sleep 15”), spec={"_priority": 10},
name=name1)
fireworks.append(fw1)
name1 = “sleep 2”
fw2 = Firework(ScriptTask.from_str(“sleep 15”), spec={"_priority": 10},
name=name1,
parents=[fw1])
fireworks.append(fw2)
name1 = “sleep 3”
fw3 = Firework(ScriptTask.from_str(“sleep 15”), spec={"_priority": 10},
name=name1,
parents=[fw2])
fireworks.append(fw3)
links = {}
for i in range(0, len(fireworks) - 1):
links[fireworks[i]] = fireworks[i + 1]
workflow = Workflow(fireworks, links_dict=links)
return FWAction(detours=workflow)
@explicit_serialize
class endTask(FireTaskBase):
def run_task(self, fw_spec):
time.sleep(30)
launchpad = LaunchPad()
fireworks = []
name = “start”
sfw = Firework(ScriptTask.from_str(“sleep 15”), name=name, spec={"_priority": 1})
fireworks.append(sfw)
name = “detours”
dfw = Firework(testTask(), name=name, parents=[sfw])
fireworks.append(dfw)
name = “end”
efw = Firework(ScriptTask.from_str(“sleep 15”), name=name, spec={"_priority": 10},
parents=[dfw]
)
fireworks.append(efw)
wf = Workflow(fireworks, name=“test detours”)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
On Friday, July 8, 2016 at 4:57:38 PM UTC-5, ajain wrote:
Hi Ling
set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
If the priority is set correctly (e.g., as the “_priority” key in the spec - don’t forget the underscore) - then this should not happen. Can you attach your code that you are using to test this?
Best,
Anubhav
On Fri, Jul 8, 2016 at 2:51 PM, [email protected] wrote:
Thanks a lot, Aunbhav.
I tried to set up FIFO in the FWConfig, and set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
Really appreciate your help.
Ling
On Friday, July 8, 2016 at 3:29:21 PM UTC-5, ajain wrote:
Hi Ling
Sorry, but I do not understand your question or use case.
There are only a couple of ways that FireWorks can set job priority - either by setting the priority within each Firework (which can be done either at the time the workflow is created, or can be added at anytime using the lpad set_priority command), or by setting the order of workflow pulling as FIFO vs FILO in the FWConfig.
If users are submitting the workflows themselves, you can write a function that finds jobs with no priority set and assign priorities to each job such that “root” Fireworks have lower priority and “children” Fireworks have higher priority. Or you can write a function that takes in any workflow and adds priorities to each Firework, and ask users to run their workflows through that function before submitting them. e.g.
def add_priority(original_wf, root_priority, child_priority=None): """ Adds priority to a workflow Args: original_wf (Workflow): original WF root_priority (int): priority of first (root) job(s) child_priority(int): priority of all child jobs. Defaults to root_priority Returns: (Workflow) priority-decorated workflow """ child_priority = child_priority or root_priority root_fw_ids = original_wf.root_fw_ids for fw in original_wf.fws: if fw.fw_id in root_fw_ids: fw.spec["_priority"] = root_priority else: fw.spec["_priority"] = child_priority return original_wf
If you run a workflow through that function, and set the child priority higher than the root priority, it will finish that workflow before starting another workflow with the same root priority.
Best,
Anubhav
On Fri, Jul 8, 2016 at 9:16 AM, [email protected] wrote:
Sorry, I didn’t metion that I know the job priority can do that.
But our situation is different. We can’t set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one.
Do you have any suggessiont to deal with this? Thanks,
Ling
On Thursday, July 7, 2016 at 4:32:50 PM UTC-5, ajain wrote:
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6d79cc86-60a6-4b9e-9d66-cfdec60197e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/d83f8daf-15e0-47b4-b1d0-c769c290dd64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi Ling
You did not set a priority for the “detours” Firework. As stated in the docs, Fireworks that have no priority set will be run last. So your “detours” Firework is waiting for all the other Fireworks in all the other workflows to finish first since there is no priority at all set for this Firework.
You can fix your code just by changing this line to add a priority to dfw:
dfw = Firework(testTask(), name=name, parents=[sfw], spec={"_priority": 10})
On Fri, Jul 8, 2016 at 3:07 PM, [email protected] wrote:
from fireworks import LaunchPad, PyTask, ScriptTask, Workflow
from fireworks.core.firework import FWAction, Firework, FireTaskBase, Workflow
from fireworks.core.rocket_launcher import rapidfire
from fireworks.utilities.fw_utilities import explicit_serialize
import time
@explicit_serialize
class exceptionTask(FireTaskBase):
def run_task(self, fw_spec):
raise NameError(“just to throw an exception”)
@explicit_serialize
class testTask(FireTaskBase):
def run_task(self, fw_spec):
fireworks = []
name1 = “sleep 1”
fw1 = Firework(ScriptTask.from_str(“sleep 15”), spec={"_priority": 10},
name=name1)
fireworks.append(fw1)
name1 = “sleep 2”
fw2 = Firework(ScriptTask.from_str(“sleep 15”), spec={"_priority": 10},
name=name1,
parents=[fw1])
fireworks.append(fw2)
name1 = “sleep 3”
fw3 = Firework(ScriptTask.from_str(“sleep 15”), spec={"_priority": 10},
name=name1,
parents=[fw2])
fireworks.append(fw3)
links = {}
for i in range(0, len(fireworks) - 1):
links[fireworks[i]] = fireworks[i + 1]
workflow = Workflow(fireworks, links_dict=links)
return FWAction(detours=workflow)
@explicit_serialize
class endTask(FireTaskBase):
def run_task(self, fw_spec):
time.sleep(30)
launchpad = LaunchPad()
fireworks = []
name = “start”
sfw = Firework(ScriptTask.from_str(“sleep 15”), name=name, spec={"_priority": 1})
fireworks.append(sfw)
name = “detours”
dfw = Firework(testTask(), name=name, parents=[sfw])
fireworks.append(dfw)
name = “end”
efw = Firework(ScriptTask.from_str(“sleep 15”), name=name, spec={"_priority": 10},
parents=[dfw]
)
fireworks.append(efw)
wf = Workflow(fireworks, name=“test detours”)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
On Friday, July 8, 2016 at 4:57:38 PM UTC-5, ajain wrote:
Hi Ling
set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
If the priority is set correctly (e.g., as the “_priority” key in the spec - don’t forget the underscore) - then this should not happen. Can you attach your code that you are using to test this?
Best,
Anubhav
On Fri, Jul 8, 2016 at 2:51 PM, [email protected] wrote:
Thanks a lot, Aunbhav.
I tried to set up FIFO in the FWConfig, and set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.
Any more thoughts to fix this.
Really appreciate your help.
Ling
On Friday, July 8, 2016 at 3:29:21 PM UTC-5, ajain wrote:
Hi Ling
Sorry, but I do not understand your question or use case.
There are only a couple of ways that FireWorks can set job priority - either by setting the priority within each Firework (which can be done either at the time the workflow is created, or can be added at anytime using the lpad set_priority command), or by setting the order of workflow pulling as FIFO vs FILO in the FWConfig.
If users are submitting the workflows themselves, you can write a function that finds jobs with no priority set and assign priorities to each job such that “root” Fireworks have lower priority and “children” Fireworks have higher priority. Or you can write a function that takes in any workflow and adds priorities to each Firework, and ask users to run their workflows through that function before submitting them. e.g.
def add_priority(original_wf, root_priority, child_priority=None): """ Adds priority to a workflow Args: original_wf (Workflow): original WF root_priority (int): priority of first (root) job(s) child_priority(int): priority of all child jobs. Defaults to root_priority Returns: (Workflow) priority-decorated workflow """ child_priority = child_priority or root_priority root_fw_ids = original_wf.root_fw_ids for fw in original_wf.fws: if fw.fw_id in root_fw_ids: fw.spec["_priority"] = root_priority else: fw.spec["_priority"] = child_priority return original_wf
If you run a workflow through that function, and set the child priority higher than the root priority, it will finish that workflow before starting another workflow with the same root priority.
Best,
Anubhav
On Fri, Jul 8, 2016 at 9:16 AM, [email protected] wrote:
Sorry, I didn’t metion that I know the job priority can do that.
But our situation is different. We can’t set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one.
Do you have any suggessiont to deal with this? Thanks,
Ling
On Thursday, July 7, 2016 at 4:32:50 PM UTC-5, ajain wrote:
On Thu, Jul 7, 2016 at 2:28 PM, [email protected] wrote:
If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times.
some thing like this:
launchpad.add_wf(wf)
launchpad.add_wf(wf)
launchpad.add_wf(wf)
rapidfire(launchpad)
But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.
What should I do to make it finish WF one by one, instead of starting all the WFs at one time?
Thanks,
Ling
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6d79cc86-60a6-4b9e-9d66-cfdec60197e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/d83f8daf-15e0-47b4-b1d0-c769c290dd64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ff70b048-68df-44de-923b-7fd8921605fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.