qadapter

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

···

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Anubhav,
Thanks on the answer,

I am familiar with MPI framework, I used it in the past.

Maybe I am not enough familiar with fireworks, but if I am understanding you correctly, fireworks clients or computer setup node could be setup to listen for jobs.

My only concern is following:

I want to programmatically setup a node (computer resource) to listen for jobs on each node.

Let’s say I have four servers, they all share fireworks and python via NFS. fifth node is running mongoDB and has also programmatically setup firework framework to submit same job as four tasks or workflows.

You are right, jobs are same python app with instructions on how to slice big data

All I want to is to submit these tasks or work flows and as schedule them for parallel execution on each node.

If you are telling me that fireworks has nothing that synchronize this but relies on the external protocol framework like MPI to do that via hooks, then I will need to go that way.

I understood that fireworks has also a built in queue, possibly via mongodb or otherwise. I apologies I did not look at the source yet…

Let me know if I am correct, and I will use your mlaunch tutorial!

Thanks again, --sasha

···

On Monday, March 2, 2015 at 1:21:11 PM UTC-5, Anubhav Jain wrote:

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Sasha,

If I understand correctly, you want to use MongoDB as a queue. That’s what FWS is for; just submit all your individual jobs to the “LaunchPad”, which will store them in MongoDB as part of a “queue”. Then, on all your other servers (what FWS calls “FireWorkers”), you want to run a script that pulls jobs from the queue and runs it. This can be either the rlaunch script (as covered in the tutorials) or mlaunch (for parallelization, as mentioned before).

For example, let’s say you have 1000 workflows. You can insert them all into the LaunchPad such that 1000 workflows are all stored in the DB. Then, your 4 workers can call rlaunch or mlaunch. The workers will pull jobs from the queue and execute them. Depending on the options you use for rlaunch/mlaunch, this can be done continuously, just once, or a preset number of times.

All of this is pretty much covered in the introductory tutorials. My suggestion is that you try following some of these introductory tutorials as I think this will make the basic operation clearer, and then come back with specific questions if anything is unclear. However, from what you’re saying I don’t think you need the qadapter. FWS acts as a job queue without needing a separate qadapter, which is used to connect to externally installed job queues.

Best,

Anubhav

···

On Tue, Mar 3, 2015 at 6:58 AM, [email protected] wrote:

Hi Anubhav,
Thanks on the answer,

I am familiar with MPI framework, I used it in the past.

Maybe I am not enough familiar with fireworks, but if I am understanding you correctly, fireworks clients or computer setup node could be setup to listen for jobs.

My only concern is following:

I want to programmatically setup a node (computer resource) to listen for jobs on each node.

Let’s say I have four servers, they all share fireworks and python via NFS. fifth node is running mongoDB and has also programmatically setup firework framework to submit same job as four tasks or workflows.

You are right, jobs are same python app with instructions on how to slice big data

All I want to is to submit these tasks or work flows and as schedule them for parallel execution on each node.

If you are telling me that fireworks has nothing that synchronize this but relies on the external protocol framework like MPI to do that via hooks, then I will need to go that way.

I understood that fireworks has also a built in queue, possibly via mongodb or otherwise. I apologies I did not look at the source yet…

Let me know if I am correct, and I will use your mlaunch tutorial!

Thanks again, --sasha

On Monday, March 2, 2015 at 1:21:11 PM UTC-5, Anubhav Jain wrote:

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/157b6a4d-4206-4c3a-b313-a45e7718e147%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ok, thanks for explanation.
I will go with mongo queue for now and then we might extend to MPI if needed.
Thanks gain for prompt responses.
btw, fireworks works well with py3.4
Also I fixed one issue with helper.py in app.py for webgui bootstrap.
lpad webgui will not run as is. After I added full path to helper.py, eg,
from fireworks.flask_suite.helpers instead of from helpers
everything worked smoothly…

Regards,

···

On Tuesday, March 3, 2015 at 11:32:11 AM UTC-5, Anubhav Jain wrote:

Hi Sasha,

If I understand correctly, you want to use MongoDB as a queue. That’s what FWS is for; just submit all your individual jobs to the “LaunchPad”, which will store them in MongoDB as part of a “queue”. Then, on all your other servers (what FWS calls “FireWorkers”), you want to run a script that pulls jobs from the queue and runs it. This can be either the rlaunch script (as covered in the tutorials) or mlaunch (for parallelization, as mentioned before).

For example, let’s say you have 1000 workflows. You can insert them all into the LaunchPad such that 1000 workflows are all stored in the DB. Then, your 4 workers can call rlaunch or mlaunch. The workers will pull jobs from the queue and execute them. Depending on the options you use for rlaunch/mlaunch, this can be done continuously, just once, or a preset number of times.

All of this is pretty much covered in the introductory tutorials. My suggestion is that you try following some of these introductory tutorials as I think this will make the basic operation clearer, and then come back with specific questions if anything is unclear. However, from what you’re saying I don’t think you need the qadapter. FWS acts as a job queue without needing a separate qadapter, which is used to connect to externally installed job queues.

Best,

Anubhav

On Tue, Mar 3, 2015 at 6:58 AM, [email protected] wrote:

Hi Anubhav,
Thanks on the answer,

I am familiar with MPI framework, I used it in the past.

Maybe I am not enough familiar with fireworks, but if I am understanding you correctly, fireworks clients or computer setup node could be setup to listen for jobs.

My only concern is following:

I want to programmatically setup a node (computer resource) to listen for jobs on each node.

Let’s say I have four servers, they all share fireworks and python via NFS. fifth node is running mongoDB and has also programmatically setup firework framework to submit same job as four tasks or workflows.

You are right, jobs are same python app with instructions on how to slice big data

All I want to is to submit these tasks or work flows and as schedule them for parallel execution on each node.

If you are telling me that fireworks has nothing that synchronize this but relies on the external protocol framework like MPI to do that via hooks, then I will need to go that way.

I understood that fireworks has also a built in queue, possibly via mongodb or otherwise. I apologies I did not look at the source yet…

Let me know if I am correct, and I will use your mlaunch tutorial!

Thanks again, --sasha

On Monday, March 2, 2015 at 1:21:11 PM UTC-5, Anubhav Jain wrote:

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/157b6a4d-4206-4c3a-b313-a45e7718e147%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Thanks for following up and mentioning the py3.4 fix. Would you be willing to contribute the patch via Github? Then the patch will be tagged with your username:

http://pythonhosted.org/FireWorks/contributing.html

(skip ahead to “How to Contribute”, the criteria for inclusion doesn’t apply here)

If this is too much trouble I will go ahead and add the patch, just let me know your preference.

Best,

Anubhav

···

On Tue, Mar 3, 2015 at 10:55 AM, [email protected] wrote:

Ok, thanks for explanation.
I will go with mongo queue for now and then we might extend to MPI if needed.
Thanks gain for prompt responses.
btw, fireworks works well with py3.4
Also I fixed one issue with helper.py in app.py for webgui bootstrap.
lpad webgui will not run as is. After I added full path to helper.py, eg,
from fireworks.flask_suite.helpers instead of from helpers
everything worked smoothly…

Regards,

On Tuesday, March 3, 2015 at 11:32:11 AM UTC-5, Anubhav Jain wrote:

Hi Sasha,

If I understand correctly, you want to use MongoDB as a queue. That’s what FWS is for; just submit all your individual jobs to the “LaunchPad”, which will store them in MongoDB as part of a “queue”. Then, on all your other servers (what FWS calls “FireWorkers”), you want to run a script that pulls jobs from the queue and runs it. This can be either the rlaunch script (as covered in the tutorials) or mlaunch (for parallelization, as mentioned before).

For example, let’s say you have 1000 workflows. You can insert them all into the LaunchPad such that 1000 workflows are all stored in the DB. Then, your 4 workers can call rlaunch or mlaunch. The workers will pull jobs from the queue and execute them. Depending on the options you use for rlaunch/mlaunch, this can be done continuously, just once, or a preset number of times.

All of this is pretty much covered in the introductory tutorials. My suggestion is that you try following some of these introductory tutorials as I think this will make the basic operation clearer, and then come back with specific questions if anything is unclear. However, from what you’re saying I don’t think you need the qadapter. FWS acts as a job queue without needing a separate qadapter, which is used to connect to externally installed job queues.

Best,

Anubhav

On Tue, Mar 3, 2015 at 6:58 AM, [email protected] wrote:

Hi Anubhav,
Thanks on the answer,

I am familiar with MPI framework, I used it in the past.

Maybe I am not enough familiar with fireworks, but if I am understanding you correctly, fireworks clients or computer setup node could be setup to listen for jobs.

My only concern is following:

I want to programmatically setup a node (computer resource) to listen for jobs on each node.

Let’s say I have four servers, they all share fireworks and python via NFS. fifth node is running mongoDB and has also programmatically setup firework framework to submit same job as four tasks or workflows.

You are right, jobs are same python app with instructions on how to slice big data

All I want to is to submit these tasks or work flows and as schedule them for parallel execution on each node.

If you are telling me that fireworks has nothing that synchronize this but relies on the external protocol framework like MPI to do that via hooks, then I will need to go that way.

I understood that fireworks has also a built in queue, possibly via mongodb or otherwise. I apologies I did not look at the source yet…

Let me know if I am correct, and I will use your mlaunch tutorial!

Thanks again, --sasha

On Monday, March 2, 2015 at 1:21:11 PM UTC-5, Anubhav Jain wrote:

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/157b6a4d-4206-4c3a-b313-a45e7718e147%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/7983374d-b5bc-472b-8571-814d547e4d56%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

No problem Anubhav,
will do that this weekend, thanks–sasha

···

On Wednesday, March 4, 2015 at 1:04:27 PM UTC-5, Anubhav Jain wrote:

Thanks for following up and mentioning the py3.4 fix. Would you be willing to contribute the patch via Github? Then the patch will be tagged with your username:

http://pythonhosted.org/FireWorks/contributing.html

(skip ahead to “How to Contribute”, the criteria for inclusion doesn’t apply here)

If this is too much trouble I will go ahead and add the patch, just let me know your preference.

Best,

Anubhav

On Tue, Mar 3, 2015 at 10:55 AM, [email protected] wrote:

Ok, thanks for explanation.
I will go with mongo queue for now and then we might extend to MPI if needed.
Thanks gain for prompt responses.
btw, fireworks works well with py3.4
Also I fixed one issue with helper.py in app.py for webgui bootstrap.
lpad webgui will not run as is. After I added full path to helper.py, eg,
from fireworks.flask_suite.helpers instead of from helpers
everything worked smoothly…

Regards,

On Tuesday, March 3, 2015 at 11:32:11 AM UTC-5, Anubhav Jain wrote:

Hi Sasha,

If I understand correctly, you want to use MongoDB as a queue. That’s what FWS is for; just submit all your individual jobs to the “LaunchPad”, which will store them in MongoDB as part of a “queue”. Then, on all your other servers (what FWS calls “FireWorkers”), you want to run a script that pulls jobs from the queue and runs it. This can be either the rlaunch script (as covered in the tutorials) or mlaunch (for parallelization, as mentioned before).

For example, let’s say you have 1000 workflows. You can insert them all into the LaunchPad such that 1000 workflows are all stored in the DB. Then, your 4 workers can call rlaunch or mlaunch. The workers will pull jobs from the queue and execute them. Depending on the options you use for rlaunch/mlaunch, this can be done continuously, just once, or a preset number of times.

All of this is pretty much covered in the introductory tutorials. My suggestion is that you try following some of these introductory tutorials as I think this will make the basic operation clearer, and then come back with specific questions if anything is unclear. However, from what you’re saying I don’t think you need the qadapter. FWS acts as a job queue without needing a separate qadapter, which is used to connect to externally installed job queues.

Best,

Anubhav

On Tue, Mar 3, 2015 at 6:58 AM, [email protected] wrote:

Hi Anubhav,
Thanks on the answer,

I am familiar with MPI framework, I used it in the past.

Maybe I am not enough familiar with fireworks, but if I am understanding you correctly, fireworks clients or computer setup node could be setup to listen for jobs.

My only concern is following:

I want to programmatically setup a node (computer resource) to listen for jobs on each node.

Let’s say I have four servers, they all share fireworks and python via NFS. fifth node is running mongoDB and has also programmatically setup firework framework to submit same job as four tasks or workflows.

You are right, jobs are same python app with instructions on how to slice big data

All I want to is to submit these tasks or work flows and as schedule them for parallel execution on each node.

If you are telling me that fireworks has nothing that synchronize this but relies on the external protocol framework like MPI to do that via hooks, then I will need to go that way.

I understood that fireworks has also a built in queue, possibly via mongodb or otherwise. I apologies I did not look at the source yet…

Let me know if I am correct, and I will use your mlaunch tutorial!

Thanks again, --sasha

On Monday, March 2, 2015 at 1:21:11 PM UTC-5, Anubhav Jain wrote:

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/157b6a4d-4206-4c3a-b313-a45e7718e147%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/7983374d-b5bc-472b-8571-814d547e4d56%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Sasha,

Let me know if you still plan to add this patch, otherwise I will go ahead and include it on your behalf. Thanks again!

Best,

Anubhav

···

On Fri, Mar 6, 2015 at 6:54 AM, [email protected] wrote:

No problem Anubhav,
will do that this weekend, thanks–sasha

On Wednesday, March 4, 2015 at 1:04:27 PM UTC-5, Anubhav Jain wrote:

Thanks for following up and mentioning the py3.4 fix. Would you be willing to contribute the patch via Github? Then the patch will be tagged with your username:

http://pythonhosted.org/FireWorks/contributing.html

(skip ahead to “How to Contribute”, the criteria for inclusion doesn’t apply here)

If this is too much trouble I will go ahead and add the patch, just let me know your preference.

Best,

Anubhav

On Tue, Mar 3, 2015 at 10:55 AM, [email protected] wrote:

Ok, thanks for explanation.
I will go with mongo queue for now and then we might extend to MPI if needed.
Thanks gain for prompt responses.
btw, fireworks works well with py3.4
Also I fixed one issue with helper.py in app.py for webgui bootstrap.
lpad webgui will not run as is. After I added full path to helper.py, eg,
from fireworks.flask_suite.helpers instead of from helpers
everything worked smoothly…

Regards,

On Tuesday, March 3, 2015 at 11:32:11 AM UTC-5, Anubhav Jain wrote:

Hi Sasha,

If I understand correctly, you want to use MongoDB as a queue. That’s what FWS is for; just submit all your individual jobs to the “LaunchPad”, which will store them in MongoDB as part of a “queue”. Then, on all your other servers (what FWS calls “FireWorkers”), you want to run a script that pulls jobs from the queue and runs it. This can be either the rlaunch script (as covered in the tutorials) or mlaunch (for parallelization, as mentioned before).

For example, let’s say you have 1000 workflows. You can insert them all into the LaunchPad such that 1000 workflows are all stored in the DB. Then, your 4 workers can call rlaunch or mlaunch. The workers will pull jobs from the queue and execute them. Depending on the options you use for rlaunch/mlaunch, this can be done continuously, just once, or a preset number of times.

All of this is pretty much covered in the introductory tutorials. My suggestion is that you try following some of these introductory tutorials as I think this will make the basic operation clearer, and then come back with specific questions if anything is unclear. However, from what you’re saying I don’t think you need the qadapter. FWS acts as a job queue without needing a separate qadapter, which is used to connect to externally installed job queues.

Best,

Anubhav

On Tue, Mar 3, 2015 at 6:58 AM, [email protected] wrote:

Hi Anubhav,
Thanks on the answer,

I am familiar with MPI framework, I used it in the past.

Maybe I am not enough familiar with fireworks, but if I am understanding you correctly, fireworks clients or computer setup node could be setup to listen for jobs.

My only concern is following:

I want to programmatically setup a node (computer resource) to listen for jobs on each node.

Let’s say I have four servers, they all share fireworks and python via NFS. fifth node is running mongoDB and has also programmatically setup firework framework to submit same job as four tasks or workflows.

You are right, jobs are same python app with instructions on how to slice big data

All I want to is to submit these tasks or work flows and as schedule them for parallel execution on each node.

If you are telling me that fireworks has nothing that synchronize this but relies on the external protocol framework like MPI to do that via hooks, then I will need to go that way.

I understood that fireworks has also a built in queue, possibly via mongodb or otherwise. I apologies I did not look at the source yet…

Let me know if I am correct, and I will use your mlaunch tutorial!

Thanks again, --sasha

On Monday, March 2, 2015 at 1:21:11 PM UTC-5, Anubhav Jain wrote:

Hi Sasha,

I am a bit confused as the queue adapters are for working with queueing systems such as PBS, SLURM, SGE, etc. i.e. a queue adapter is used when you are submitting jobs to a shared computing cluster in which you cannot run jobs directly, but the cluster administrator has set up one of these systems for submitting your jobs. For this case, the queue adapter you use is just the one that matches the queueing system set up by the cluster administrator (generally, the CommonAdapter covers most of these systems).

It looks like in your case, you are looking to multiprocess many jobs on a node with different parameters. In this case, have you looked at the following documentation on “mlaunch”?

http://pythonhosted.org//FireWorks/multi_job.html

Basically, if you enter different workflows in your database, you can use the mlaunch command to parallelize their execution over multiple cores. So you would explicitly enter all your different parameter sets as different workflows (or different Fireworks within the same workflow) and then use mlaunch to parallelize.

I hope that helps, let me know if not.

Best,

Anubhav

On Fri, Feb 27, 2015 at 1:54 PM, [email protected] wrote:

Hi, sorry to intrude into your time,

Nice app…
I am testing fireworks as a master worker pattern for replacement of commercial version of software.

This is my setup:

  1. NFS share where py, libs and FW reside.
    bunch of grid computing nodes all attached to NF share

  2. I am able to run single and rapid fire clients, no issues there.

  3. I am exploring, queue adapters, and I am interested in achieving following:

mongoDB is on the node1
I would like to setup a workflow that submits ScriptTask with each bash script that is running python multiprocess applications (full node /core utilization) and is passing different input parameters

all this works in singleshot mode.

Basically, main business logic application does a chunk of job …
Could you please suggest queue adapter that i should use and modify to fit my purposes …

Regard, --sasha

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ea011431-8cb4-4d4f-8601-ede9fe34f7c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/157b6a4d-4206-4c3a-b313-a45e7718e147%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/7983374d-b5bc-472b-8571-814d547e4d56%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/06bb492c-656c-4c5c-845e-0ceb6921cac2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.