Delay between modifying settings and qlaunch detecting them?

FireWorks
#1

Hello,

I’m trying to figure out if this is a bug, expected behavior or user error on my part… Unfortunately I don’t quite have a minimal example to share.

I’m currently changing my_fworker.yaml frequently as I try out different settings (e.g. changing NPAR/KPAR, vasp_cmd variables), and then after modification I submit a job using qlaunch. However, I often find the old variables are then picked up.

I observed this when changing KPAR: 8 to NPAR: 4. If I submitted the rlaunch command, explicitly providing the path to my config file, it would correctly pick up the new NPAR: 4 setting. However, using qlaunch, which would generate a FW_submit.script containing the exact same rlaunch command, it would incorrectly pick up the old KPAR: 8 setting. I also observed similar behavior when changing the vasp_cmd variable.

I saw the Fireworks website mentions a qlaunch daemon, so I wasn’t sure if my old settings were persisting somewhere I was unaware of? If so, is there a way to make qlaunch detect the new settings? Or is this user error on my part, and I’m not understanding how Fireworks works correctly?

Many thanks for your help!

Best regards,

Matthew

#2

Actually, thinking about this more, perhaps this is not a Fireworks issue at all, but maybe a filesystem synchronization issue on the HPC system I’m using (Cori)? Will report back if I manage to figure this out…

Best,

Matthew

···

On Thursday, April 6, 2017 at 6:04:10 PM UTC-7, [email protected] wrote:

Hello,

I’m trying to figure out if this is a bug, expected behavior or user error on my part… Unfortunately I don’t quite have a minimal example to share.

I’m currently changing my_fworker.yaml frequently as I try out different settings (e.g. changing NPAR/KPAR, vasp_cmd variables), and then after modification I submit a job using qlaunch. However, I often find the old variables are then picked up.

I observed this when changing KPAR: 8 to NPAR: 4. If I submitted the rlaunch command, explicitly providing the path to my config file, it would correctly pick up the new NPAR: 4 setting. However, using qlaunch, which would generate a FW_submit.script containing the exact same rlaunch command, it would incorrectly pick up the old KPAR: 8 setting. I also observed similar behavior when changing the vasp_cmd variable.

I saw the Fireworks website mentions a qlaunch daemon, so I wasn’t sure if my old settings were persisting somewhere I was unaware of? If so, is there a way to make qlaunch detect the new settings? Or is this user error on my part, and I’m not understanding how Fireworks works correctly?

Many thanks for your help!

Best regards,

Matthew

#3

Hi Matt

What should happen is that every time you execute qlaunch (or rlaunch), the configuration files should be read again. Nothing is persisted between qlaunch or rlaunch sessions. However, if you are in rapidfire mode, then the whatever configuration was read at the beginning will apply to all the runs - even if you switch the configuration files mid-execution of qlaunch. i.e., qlaunch will not keep re-checking the configuration files for updates in the middle of rapidfire.

In terms of problems, we did have issues with “rogue” old qlaunch processes on various NERSC nodes lasting longer than expected in rapidfire mode. e.g., things would appear in the queue even if we didn’t think we had qlaunch running, and the settings would be whatever old settings that qlaunch was started with. At one point I wrote a script to detect this but can’t find it at the moment.

Best,

Anubhav

···

On Fri, Apr 7, 2017 at 8:40 AM, [email protected] wrote:

Actually, thinking about this more, perhaps this is not a Fireworks issue at all, but maybe a filesystem synchronization issue on the HPC system I’m using (Cori)? Will report back if I manage to figure this out…

Best,

Matthew

On Thursday, April 6, 2017 at 6:04:10 PM UTC-7, [email protected] wrote:

Hello,

I’m trying to figure out if this is a bug, expected behavior or user error on my part… Unfortunately I don’t quite have a minimal example to share.

I’m currently changing my_fworker.yaml frequently as I try out different settings (e.g. changing NPAR/KPAR, vasp_cmd variables), and then after modification I submit a job using qlaunch. However, I often find the old variables are then picked up.

I observed this when changing KPAR: 8 to NPAR: 4. If I submitted the rlaunch command, explicitly providing the path to my config file, it would correctly pick up the new NPAR: 4 setting. However, using qlaunch, which would generate a FW_submit.script containing the exact same rlaunch command, it would incorrectly pick up the old KPAR: 8 setting. I also observed similar behavior when changing the vasp_cmd variable.

I saw the Fireworks website mentions a qlaunch daemon, so I wasn’t sure if my old settings were persisting somewhere I was unaware of? If so, is there a way to make qlaunch detect the new settings? Or is this user error on my part, and I’m not understanding how Fireworks works correctly?

Many thanks for your help!

Best regards,

Matthew

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8440f0fd-6fbe-4fbe-ada3-2f177b4e273f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

Best,
Anubhav

#4

Hi Anubhav,

Thanks for this.

What should happen is that every time you execute qlaunch (or rlaunch), the configuration files should be read again. Nothing is persisted between qlaunch or rlaunch sessions.

This is good to know, and was what I had expected should happen.

However, if you are in rapidfire mode, then the whatever configuration was read at the beginning will apply to all the runs …

Perhaps I had an old rapidfire process that I hadn’t terminated correctly that had picked up the old settings then, which would make sense. As for any other rogue NERSC issues, I will keep an eye on this and report back if I find anything reproducible.

Best,

Matt

···

On Friday, April 7, 2017 at 9:18:32 AM UTC-7, ajain wrote:

Hi Matt

What should happen is that every time you execute qlaunch (or rlaunch), the configuration files should be read again. Nothing is persisted between qlaunch or rlaunch sessions. However, if you are in rapidfire mode, then the whatever configuration was read at the beginning will apply to all the runs - even if you switch the configuration files mid-execution of qlaunch. i.e., qlaunch will not keep re-checking the configuration files for updates in the middle of rapidfire.

In terms of problems, we did have issues with “rogue” old qlaunch processes on various NERSC nodes lasting longer than expected in rapidfire mode. e.g., things would appear in the queue even if we didn’t think we had qlaunch running, and the settings would be whatever old settings that qlaunch was started with. At one point I wrote a script to detect this but can’t find it at the moment.

Best,

Anubhav

On Fri, Apr 7, 2017 at 8:40 AM, [email protected] wrote:

Actually, thinking about this more, perhaps this is not a Fireworks issue at all, but maybe a filesystem synchronization issue on the HPC system I’m using (Cori)? Will report back if I manage to figure this out…

Best,

Matthew

On Thursday, April 6, 2017 at 6:04:10 PM UTC-7, [email protected] wrote:

Hello,

I’m trying to figure out if this is a bug, expected behavior or user error on my part… Unfortunately I don’t quite have a minimal example to share.

I’m currently changing my_fworker.yaml frequently as I try out different settings (e.g. changing NPAR/KPAR, vasp_cmd variables), and then after modification I submit a job using qlaunch. However, I often find the old variables are then picked up.

I observed this when changing KPAR: 8 to NPAR: 4. If I submitted the rlaunch command, explicitly providing the path to my config file, it would correctly pick up the new NPAR: 4 setting. However, using qlaunch, which would generate a FW_submit.script containing the exact same rlaunch command, it would incorrectly pick up the old KPAR: 8 setting. I also observed similar behavior when changing the vasp_cmd variable.

I saw the Fireworks website mentions a qlaunch daemon, so I wasn’t sure if my old settings were persisting somewhere I was unaware of? If so, is there a way to make qlaunch detect the new settings? Or is this user error on my part, and I’m not understanding how Fireworks works correctly?

Many thanks for your help!

Best regards,

Matthew

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8440f0fd-6fbe-4fbe-ada3-2f177b4e273f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav