Slow `lpad get_fws` command

FireWorks
#1

My lpad get_fws -d less command seems to be quite slow and scales poorly with the number of fws to be returned.

For example, I have the following timing:

  • -m 1: 2.4s
  • -m 10: 4.5s
  • -m 100: 27.6s

I understand that the code needs to query the launches collection, but with -d less option no information about launches are printed.

Am I having the expected behaviour?
I am with fireworks=1.9.5 on WSL Ubuntu 18.04

#2

After some profiling I found this is due to the lpad script trying to deserialize the retrived Fireworks one by one. Most of the time seems to spent on searching and importing the modules.

#3

Hi,

These times are much longer than expected.

Do you have your own custom Firetasks that are being deserialized? If so, perhaps switching to the explicit serialization would help as this avoids the need to search through modules.

If your FW spec are very large (Eg 10MB per Firework) or your database installation is slow, then querying can also be slow. Also if you use the -q flag in your get_fws query and don’t set up proper indexes.

Otherwise I would expect things to finish within seconds. We’ve never had a problem with deserialization itself being slow.

2 Likes
#4

Hi ajain,

Thanks for the reply. Yes, I have them deserialized, they were not explicit serialization previously. After switching to explicit serialization I can see that the time is no longer dominated by importing. But it is still quite slow, unfortunately.

The FW spec is not very large either - should be less than 10K at most.
I tried to use some profiling tools (py-spy) to see where the time is mostly spent on.

Here is the profiling result for getting 1000 FWs.
profile

You can see that most of the time is spent on get_fw_dict_by_id. My understanding is that this is because this function has to be called on each FW to be retrieved, and within each call, it queries the fireworks collection to get the fireworks, and then two queries to launch collection to get the launch. These queries only return a small amount of data and doing them repetitively is bound to be slow (scales linearly with the number of FWs to be retrieved).

Maybe this part of the code is worth refactoring? I feel this is getting too technical, perhaps I should open an issue on the Github page…

When you say “seconds”, do you mean that getting things like 500 FWs should finish within seconds? It would take one around 2 minutes on my laptop

Thanks