Is it possible for a ScriptTask to pass variables from stored_data as command line parameters to the script?
Hi Jeff,
No, the ScriptTask can only take parameters from the “spec” (if use_global_spec is set True) or from the initialization parameters from the task (default). If you have a previous task/Firework that needs to pass some parameter to the following ScriptTask (e.g. the “script” command), you can have the previous Firework pass the an “update_spec” command to the downstream ScriptTask to use the new “script”. i.e. this will modify the spec of the downstream ScriptTask
If what you’re doing is more complicated than that, I would suggest upgrading to PyTask or writing your own FireTask
Best,
Anubhav
···
On Mon, Aug 8, 2016 at 12:00 PM, Jeff Graves [email protected] wrote:
Is it possible for a ScriptTask to pass variables from stored_data as command line parameters to the script?
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/880e948c-96fe-4413-ae96-b55ff7a3a6db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
Best,
Anubhav
Thank you for your feed back.
I have one more question pertaining to ScriptTask. I have a FireServer and several FireWorkers running across 20 machines. I have a custom binary (compiled C++ program) I am trying to execute with ScriptTask. Is there any way to get the executable to the FireWorker that will need to execute it with out putting it on an NFS or pre-staging it to every possible worker?
I thought I could accomplish this by writing a Firework that contained the FileTransferTask followed by the ScriptTask to first pull the executable before running it. However, as far as I can tell, I can only use the FileTransferTask to send data, not pull data. Is this correct?
···
On Monday, August 8, 2016 at 3:07:33 PM UTC-4, ajain wrote:
Hi Jeff,
No, the ScriptTask can only take parameters from the “spec” (if use_global_spec is set True) or from the initialization parameters from the task (default). If you have a previous task/Firework that needs to pass some parameter to the following ScriptTask (e.g. the “script” command), you can have the previous Firework pass the an “update_spec” command to the downstream ScriptTask to use the new “script”. i.e. this will modify the spec of the downstream ScriptTask
If what you’re doing is more complicated than that, I would suggest upgrading to PyTask or writing your own FireTask
Best,
Anubhav
On Mon, Aug 8, 2016 at 12:00 PM, Jeff Graves [email protected] wrote:
Is it possible for a ScriptTask to pass variables from stored_data as command line parameters to the script?
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/880e948c-96fe-4413-ae96-b55ff7a3a6db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
Best,
Anubhav
Hi Jeff
You are correct that the FileTransferTask implementation only allows for transfer to remote servers, and not from remote servers. This was not done intentionally; if you wanted to update the code to allow two-way transfers, I would be happy to include it in the repo. Unfortunately, I don’t have much time at the moment to implement this myself.
If you have passwordless SSH set up, will it work to just include a ScriptTask that will scp the needed file from wherever it is located? i.e., use a ScriptTask instead of FileTransferTask to do the file transfer with scp.
Best,
Anubhav
···
On Monday, August 8, 2016 at 12:31:46 PM UTC-7, Jeff Graves wrote:
Thank you for your feed back.
I have one more question pertaining to ScriptTask. I have a FireServer and several FireWorkers running across 20 machines. I have a custom binary (compiled C++ program) I am trying to execute with ScriptTask. Is there any way to get the executable to the FireWorker that will need to execute it with out putting it on an NFS or pre-staging it to every possible worker?
I thought I could accomplish this by writing a Firework that contained the FileTransferTask followed by the ScriptTask to first pull the executable before running it. However, as far as I can tell, I can only use the FileTransferTask to send data, not pull data. Is this correct?
On Monday, August 8, 2016 at 3:07:33 PM UTC-4, ajain wrote:
Hi Jeff,
No, the ScriptTask can only take parameters from the “spec” (if use_global_spec is set True) or from the initialization parameters from the task (default). If you have a previous task/Firework that needs to pass some parameter to the following ScriptTask (e.g. the “script” command), you can have the previous Firework pass the an “update_spec” command to the downstream ScriptTask to use the new “script”. i.e. this will modify the spec of the downstream ScriptTask
If what you’re doing is more complicated than that, I would suggest upgrading to PyTask or writing your own FireTask
Best,
Anubhav
On Mon, Aug 8, 2016 at 12:00 PM, Jeff Graves [email protected] wrote:
Is it possible for a ScriptTask to pass variables from stored_data as command line parameters to the script?
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/880e948c-96fe-4413-ae96-b55ff7a3a6db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
Best,
Anubhav
I was wrapped up with trying to fine a native FireWorks solution that I didn’t think of using scp. That should work just fine for now.
···
On Monday, August 8, 2016 at 6:33:59 PM UTC-4, Anubhav Jain wrote:
Hi Jeff
You are correct that the FileTransferTask implementation only allows for transfer to remote servers, and not from remote servers. This was not done intentionally; if you wanted to update the code to allow two-way transfers, I would be happy to include it in the repo. Unfortunately, I don’t have much time at the moment to implement this myself.
If you have passwordless SSH set up, will it work to just include a ScriptTask that will scp the needed file from wherever it is located? i.e., use a ScriptTask instead of FileTransferTask to do the file transfer with scp.
Best,
Anubhav
On Monday, August 8, 2016 at 12:31:46 PM UTC-7, Jeff Graves wrote:
Thank you for your feed back.
I have one more question pertaining to ScriptTask. I have a FireServer and several FireWorkers running across 20 machines. I have a custom binary (compiled C++ program) I am trying to execute with ScriptTask. Is there any way to get the executable to the FireWorker that will need to execute it with out putting it on an NFS or pre-staging it to every possible worker?
I thought I could accomplish this by writing a Firework that contained the FileTransferTask followed by the ScriptTask to first pull the executable before running it. However, as far as I can tell, I can only use the FileTransferTask to send data, not pull data. Is this correct?
On Monday, August 8, 2016 at 3:07:33 PM UTC-4, ajain wrote:
Hi Jeff,
No, the ScriptTask can only take parameters from the “spec” (if use_global_spec is set True) or from the initialization parameters from the task (default). If you have a previous task/Firework that needs to pass some parameter to the following ScriptTask (e.g. the “script” command), you can have the previous Firework pass the an “update_spec” command to the downstream ScriptTask to use the new “script”. i.e. this will modify the spec of the downstream ScriptTask
If what you’re doing is more complicated than that, I would suggest upgrading to PyTask or writing your own FireTask
Best,
Anubhav
On Mon, Aug 8, 2016 at 12:00 PM, Jeff Graves [email protected] wrote:
Is it possible for a ScriptTask to pass variables from stored_data as command line parameters to the script?
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/880e948c-96fe-4413-ae96-b55ff7a3a6db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
Best,
Anubhav