Getting nodes to pull jobs from fireworks

FireWorks
#1

Hello,

I’m trying to run jobs on remote computing clusters that pull from the fireworks database on our local server. I’ve set this up using ssh tunnels to secure the connection and have gotten rlaunch to work fine on the remote cluster. However, qlaunch is causing errors because the ssh tunnel only connects our local server to the head node, not any of the other nodes that the job may be sent to through the queue. My question is how can you get all nodes on a remote cluster to be able to read from a local fireworks database through a secure connection? My initial idea is to include a tunnel-activating script in the pre-rocket (and a tunnel-closing script in the post-rocket) of the qadapter but I want to know if there’s something I’m overlooking that would make this simpler.

Thanks,

Michael

#2

Hi Michael

Three options that I can think of (none of them ideal) are:

  1. Use offline mode, which is not ideal but should work:

https://pythonhosted.org/FireWorks/offline_tutorial.html

  1. Put the tunnel scripts in the pre and post rocket like you suggest

  2. Relocate your fireworks database to somewhere that is accessible by your workers

Sorry there aren’t any other better options at the moment -

Best,

Anubhav

···

On Fri, Aug 12, 2016 at 7:50 AM, Michael B [email protected] wrote:

Hello,

I’m trying to run jobs on remote computing clusters that pull from the fireworks database on our local server. I’ve set this up using ssh tunnels to secure the connection and have gotten rlaunch to work fine on the remote cluster. However, qlaunch is causing errors because the ssh tunnel only connects our local server to the head node, not any of the other nodes that the job may be sent to through the queue. My question is how can you get all nodes on a remote cluster to be able to read from a local fireworks database through a secure connection? My initial idea is to include a tunnel-activating script in the pre-rocket (and a tunnel-closing script in the post-rocket) of the qadapter but I want to know if there’s something I’m overlooking that would make this simpler.

Thanks,

Michael

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflows+unsubsc[email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/14325733-d02f-4e92-b127-c8ee98d1fcf2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

Best,
Anubhav

#3

Hi Anubhav,

Thank you for the suggestions, I will take a look at offline mode.

Michael