monitoring via python

Hi,
Could you point me to api py classes and packages that I can extend to monitor workflow and task execution and to stop or archive workflows.

Also, it seems that no matter how I setup a rapidfire to run or store launch directories, I can’t never execute binary wrapper from the relative path.

I mean this:

cd /cube/apps/
./myapp.sh

I always need to execute absolute path to wrapper.
I guess I could use to tasks, in a workflow but I am trying to do this via rapidfire or lpad setup…
Regards,

···

Hi Sasha,

Most of the code to manage the database is in launchpad.py. You can search for keywords such as “archive” or “defuse” (pause/stop) and “reignite” (resume).

I am pretty confused about your question regarding rapidfire. The only point of rapidfire is to create a new directory and pull a job from the database, and then repeat this multiple times. It should not be calling any other scripts; as you mentioned, that should go inside the task. Similarly, the only point of lpad setup is to create database credential files. Neither of these commands should be used to launch external scripts.

Note that if you want to run tasks in a particular directory, you can set the _launch_dir keyword as specified here (although I’m not sure this is your question):

https://pythonhosted.org/FireWorks/controlworker.html

···

On Tue, Mar 31, 2015 at 5:30 AM, [email protected] wrote:

Hi,
Could you point me to api py classes and packages that I can extend to monitor workflow and task execution and to stop or archive workflows.

Also, it seems that no matter how I setup a rapidfire to run or store launch directories, I can’t never execute binary wrapper from the relative path.

I mean this:

cd /cube/apps/
./myapp.sh

I always need to execute absolute path to wrapper.
I guess I could use to tasks, in a workflow but I am trying to do this via rapidfire or lpad setup…
Regards,

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8e9ca530-7cd8-43dd-a960-1bd1ee42617b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Anubhav,
Thanks for pointers.
I Created pretty extensive wrapper around fireworks.
Everything works.
As model I use script based template to submit tons of tasks as oen or many workflows. after that workflow is submitted to mongoDB.
That all works. I actually create task from my own manifest passing arguments line at the time.
Fw client just register itself and waits for jobs to process.

I read the page you are suggesting.

The question is this:

launch_dir key is setup via launchpad module, correct?

I am not using yaml, everything is done via python templates …

I will check launchpad.py for bellow methods …

I mentioned rapidfire because of the m_dir key that I am using to specify where the log files are going (loop is running)

My apologies for not clarifying exactly what I am trying to achieve…

···

On Tue, Mar 31, 2015 at 5:30 AM, [email protected] wrote:

Hi,
Could you point me to api py classes and packages that I can extend to monitor workflow and task execution and to stop or archive workflows.

Also, it seems that no matter how I setup a rapidfire to run or store launch directories, I can’t never execute binary wrapper from the relative path.

I mean this:

cd /cube/apps/
./myapp.sh

I always need to execute absolute path to wrapper.
I guess I could use to tasks, in a workflow but I am trying to do this via rapidfire or lpad setup…
Regards,

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8e9ca530-7cd8-43dd-a960-1bd1ee42617b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Sasha,

Thanks for the update. I am not sure how much I am able to help, however in regard to your specific question about “launch_dir”, that is set in the FW spec itself as per the docs:

“set the _launch_dir key in your Firework spec to the full path of the directory you want to execute the Firework in”

i.e., my_fw = Firework([tasks], spec={"_launch_dir":"/where/I/want/to/run/the/job"})

Best,

Anubhav

···

On Wed, Apr 1, 2015 at 5:16 PM, Sasha Kacanski [email protected] wrote:

Hi Anubhav,
Thanks for pointers.
I Created pretty extensive wrapper around fireworks.
Everything works.
As model I use script based template to submit tons of tasks as oen or many workflows. after that workflow is submitted to mongoDB.
That all works. I actually create task from my own manifest passing arguments line at the time.
Fw client just register itself and waits for jobs to process.

I read the page you are suggesting.

The question is this:

launch_dir key is setup via launchpad module, correct?

I am not using yaml, everything is done via python templates …

I will check launchpad.py for bellow methods …

I mentioned rapidfire because of the m_dir key that I am using to specify where the log files are going (loop is running)

My apologies for not clarifying exactly what I am trying to achieve…

On Apr 1, 2015 12:20 PM, “Anubhav Jain” [email protected] wrote:

Hi Sasha,

Most of the code to manage the database is in launchpad.py. You can search for keywords such as “archive” or “defuse” (pause/stop) and “reignite” (resume).

I am pretty confused about your question regarding rapidfire. The only point of rapidfire is to create a new directory and pull a job from the database, and then repeat this multiple times. It should not be calling any other scripts; as you mentioned, that should go inside the task. Similarly, the only point of lpad setup is to create database credential files. Neither of these commands should be used to launch external scripts.

Note that if you want to run tasks in a particular directory, you can set the _launch_dir keyword as specified here (although I’m not sure this is your question):

https://pythonhosted.org/FireWorks/controlworker.html

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/CAFWiiko9zwhZkej7SM40xmjpRFG7VbnJ7-HrjWOouP3ze1JK9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

On Tue, Mar 31, 2015 at 5:30 AM, [email protected] wrote:

Hi,
Could you point me to api py classes and packages that I can extend to monitor workflow and task execution and to stop or archive workflows.

Also, it seems that no matter how I setup a rapidfire to run or store launch directories, I can’t never execute binary wrapper from the relative path.

I mean this:

cd /cube/apps/
./myapp.sh

I always need to execute absolute path to wrapper.
I guess I could use to tasks, in a workflow but I am trying to do this via rapidfire or lpad setup…
Regards,

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8e9ca530-7cd8-43dd-a960-1bd1ee42617b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Cool thanks, now it is working exactly as I want…

···

On Friday, April 3, 2015 at 4:12:23 PM UTC-4, Anubhav Jain wrote:

Hi Sasha,

Thanks for the update. I am not sure how much I am able to help, however in regard to your specific question about “launch_dir”, that is set in the FW spec itself as per the docs:

“set the _launch_dir key in your Firework spec to the full path of the directory you want to execute the Firework in”

i.e., my_fw = Firework([tasks], spec={"_launch_dir":"/where/I/want/to/run/the/job"})

Best,

Anubhav

On Wed, Apr 1, 2015 at 5:16 PM, Sasha Kacanski [email protected] wrote:

Hi Anubhav,
Thanks for pointers.
I Created pretty extensive wrapper around fireworks.
Everything works.
As model I use script based template to submit tons of tasks as oen or many workflows. after that workflow is submitted to mongoDB.
That all works. I actually create task from my own manifest passing arguments line at the time.
Fw client just register itself and waits for jobs to process.

I read the page you are suggesting.

The question is this:

launch_dir key is setup via launchpad module, correct?

I am not using yaml, everything is done via python templates …

I will check launchpad.py for bellow methods …

I mentioned rapidfire because of the m_dir key that I am using to specify where the log files are going (loop is running)

My apologies for not clarifying exactly what I am trying to achieve…

On Apr 1, 2015 12:20 PM, “Anubhav Jain” [email protected] wrote:

Hi Sasha,

Most of the code to manage the database is in launchpad.py. You can search for keywords such as “archive” or “defuse” (pause/stop) and “reignite” (resume).

I am pretty confused about your question regarding rapidfire. The only point of rapidfire is to create a new directory and pull a job from the database, and then repeat this multiple times. It should not be calling any other scripts; as you mentioned, that should go inside the task. Similarly, the only point of lpad setup is to create database credential files. Neither of these commands should be used to launch external scripts.

Note that if you want to run tasks in a particular directory, you can set the _launch_dir keyword as specified here (although I’m not sure this is your question):

https://pythonhosted.org/FireWorks/controlworker.html

On Tue, Mar 31, 2015 at 5:30 AM, [email protected] wrote:

Hi,
Could you point me to api py classes and packages that I can extend to monitor workflow and task execution and to stop or archive workflows.

Also, it seems that no matter how I setup a rapidfire to run or store launch directories, I can’t never execute binary wrapper from the relative path.

I mean this:

cd /cube/apps/
./myapp.sh

I always need to execute absolute path to wrapper.
I guess I could use to tasks, in a workflow but I am trying to do this via rapidfire or lpad setup…
Regards,

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8e9ca530-7cd8-43dd-a960-1bd1ee42617b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/CAFWiiko9zwhZkej7SM40xmjpRFG7VbnJ7-HrjWOouP3ze1JK9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.