I have been using Atomate for a few months now and I have found it very convenient. There is one particular thing I’m trying to do with my workflows that I’m running into issues on, and I was wondering if anybody here knew of a workaround or potential solution.
I have access to two workers A & B, where worker A is on our local University cluster (with a much lower wait-time before running a job), and worker B is on a much more powerful national cluster (with correspondingly longer wait-time).
I would like to split up my workflow across these two workers. The workflow I am running is a simple relaxation + static calculation, made up of the OptimizeFW and the StaticFW. Since the relaxation calculations are more resource-intensive/time-consuming, I would like to run the OptimizeFWs exclusively on Worker B, and run the comparatively fast StaticFWs on Worker A. (I have already set up my worker categories so that Worker B is only pulling the OptimizeFWs)
My preliminary attempts to do this have resulted in the StaticFWs fizzling on Worker A. The errors are all “no such directoy exists” types of errors; in looking at the code for the StaticFW, it looks like it is trying to pull the structure from the directory of the previous relaxation calculation, but because that relaxation was done on Worker B, the firework is unable to find the corresponding directory on Worker A.
Is it possible to have the StaticFW pull the structure from the results database instead of the previous directory, and if not, is such a feature under consideration for a future update?