Dear LAMMPS users,
are there any ways to run multiple LAMMPS calculations (e.g. running in NVT ensemble for different structures) with reading a potential once?
My ML potential becomes large, it requires much more memories and takes long reading time. Every new calculations I need to newly read the potential and it wastes a lot of computational resources.
Many thanks in advanced for your help.
If reading your machine learning potential takes more time than the simulation, you have a serious problem with how your potential is implemented and how you read its settings, or you are running a system that is far too small to make sense for classical MD.
also, avoiding to read it multiple times does not lower the memory consumption unless your code has memory leaks, which are bugs that need to be fixed.
do you have some reliable numbers as to how much resources are actually spent on the reading the potential versus the calculation itself?
what type of machine learning model are you using? is this any of the packages in LAMMPS or your own implementation?
Thanks a lot for the reply.
I’m using Gaussian Approximation Potentials in QUIP package, and my trained potential is as heavy over 2GB. I’ve trained it for a quaternary system and double SOAPs for each element, it ends up with such a large potential. I have experiences with different binary systems, and I had no such problem as I had way lighter potential ~150MB. I also didn’t expect that avoiding to read it multiple times lower the memory consumption, but it will reduce the time.
I have tested minimization runs so the calculation itself doesn’t spend much resources at this point. Reading the potential takes more than an hour with 288 cores and 240GB memory. By the way, I’m not sure whether LAMMPS can read the potential in parallel or not.
I don’t want to start an argument about the benefits and shortcomings of different machine learning potentials (and I am lacking the expertise and experience for that anyway), but this sounds a lot like you are pushing the limits (if you are not over the limit) of what is meaningful within a given framework. Perhaps it could be beneficial to look for a methodology where you can get a similar quality of approximation with far fewer basis functions.
With the QUIP package, most of the “work” is not done by LAMMPS. LAMMPS only provides a very light wrapper on top of the library functions from the QUIP library. Of course the potential parameters need to be present on every processor and thus the parameters are read by every MPI rank. That is not very efficient, but this is how the QUIP library is implemented. Reading on a single MPI rank and then broadcasting the data to all other MPI ranks is usually faster (depending on your specific machine, number of nodes and file system), but since the QUIP library is written in Fortran, this includes lots of compilations: both LAMMPS and the QUIP library need to be compiled with compilers using the same MPI library, and the interface must pass an MPI communicator from the C++ code to the Fortran code (doable, but not everybody likes or wants to do that). So for discussing more details on this you have to contact the QUIP library developers.
As far as LAMMPS is concerned, I can see that the quip_lammps_potential_initialise() function is called twice by every MPI ranks when reading the potential. What the code in this function does, is not part of LAMMPS so I cannot say whether it is parsing the entire file(s) completely multiple times.
File parsing can be done more or less efficient. From what I see in the quip example folder in LAMMPS, the GAP potentials come with multiple files, either plain text or text file encoded in XML.
Reading floating point numbers from a text file is much less efficient than reading an unformatted binary file but also less portable. Perhaps there is an option to replace the text file with a binary file somehow.
Also reading and parsing XML files is notoriously slow (if the XML implementation is not taking any implementation specific shortcuts and then can handle only XML files conforming to specific conventions) because of the nature of the XML syntax. Other structured text formats are simpler and faster to read and process and structured binary file formats like hdf5 even faster. Again, these are all choices that need to be implemented in the QUIP library by the QUIP developers (if they care) and not something that can be addressed on the LAMMPS side of things (unless you would be switching to a “native” LAMMPS ML potential, of course 
).
Finally, coming back to your original question. You will have to re-initialize the entire system and thus the potential for any calculation where you have a different number of atom types (this is assuming that you don’t have any explicit bonds or similar in your model where things are much more complicated). The simplest option you have to continue would be to write out different sets of coordinates in dump files and then use the read_dump command read_dump command — LAMMPS documentation to change the system. You can add or delete atoms in the process, but you cannot add or use or change atom types that were not defined the first time. A similar but less flexible procedure is also possible with deleting all atoms and then using read_data with the “add” option to read in a new set of atoms.
This is amazing. I truly appreciate your help and insights, it already pushed me further a lot. Many thanks!