Hey folks,
I’m using PyLammps to evaluate energies of several structures with a whole bunch of different parameter sets.
I start by creating one PyLammps object for each object and read in the structure as follows:
L = PyLammps()
L.atom_style(‘charge’)
L.units(‘real’)
L.boundary(‘f f f’)
L.read_data(fname)
L.pair_style(‘reax/c NULL’)
L.neighbor(‘2.0’, ‘bin’)
L.neigh_modify(‘every 1 delay 0 check yes’)
L.timestep(‘0.02’)
L.pair_coeff(f’* * ffield.reax.start Cr H O Si’)
L.fix(‘1 all qeq/reax 1 0.0 10.0 1.0e-4 reax/c’)
L.run(0)
Then, I loop over a bunch of different parameter files and evaluate the energy with each parameter set
def calcPE(L, ffieldname):
L.pair_coeff(f’* * {ffieldname} Cr H O Si’)
L.run(0, “pre no post no”)
return L.eval(‘pe’)
This is working fine except I keep running out of memory. I have narrowed the problem down to the repeated evaluation of this calcPE() function. I haven’t been able to figure out from the documentation what objects are being created within this loop that are consuming progressively more memory. Any suggestions on the origin of this problem what I need to change to avoid it?
Seth Martin (he/him/his)
PhD candidate
Dept. of Chemistry
The University of Kansas
Seth,
it would be extremely helpful if you could provide some information where the memory is leaking through running under valgrind (watch out it is going to be very slow).
If you want one of the LAMMPS developers to look into this, you should convert your python script to a regular LAMMPS input deck (this should be quite easily possible using the loop construct of LAMMPS and an index variable with all the force field names. This would simplify debugging and would also help determining whether the leakage is due to LAMMPS itself or the Python interfacing.
The reax/c pair style is known to have some minor memory leaks when being called in a loop, but I don’t recall anything that was accumulating memory in a prohibitively significant way for normal cases.
Axel.
write_restart and read_restart should be much faster than write_data and read_data (binary data is processed much faster), also restarts store more information.
rather than creating a new instance, you can save time there, too, and use the clear command. that however may not fully avoid the memory leakage.
what might be even faster would be to do this with the C++ library interface: https://docs.lammps.org/Cplusplus.html
you create an instance of the LAMMPS class, feed it the input up to the point of the first run 0 statement.
and then do a loop where you do fork() and then do the next pair_coeff and run 0 command in the child process, write/append the properties you are after to a file and then exit. in the parent you wait for the child to exit and do the next loop iteration. this way if there is leakage, it will be in the forked child, which will inherit that state of the parent process with copy-on-write handling of all memory. of course this would probably only work with a serial executable.
You might be able to do something similar with Python using the subprocess module. Please note that the PyLammps class has much more overhead than the lammps class.
HTH,
axel.
Hi Axel,
I’m waiting to hear back from my cluster admins to see if we can set up valgrind so I can find out where the memory leak is.
Below is a lammps input deck that more or less mimics the behavior of the python loop, and I do indeed see the out-of-memory kill command from slurm when I run this input deck directly. This is maybe not a “normal” case, as I am looping over potentially hundreds of thousands of force field files (this is a force field optimization script).
Originally, I was creating new PyLammps objects with each energy evaluation, which avoided this memory leak by starting new lammps objects each time, but this has the downside that I have to read in the structures every time I want to evaluate the energy of the structure, leading to large IO overhead. Any suggested work-arounds on the PyLammps side, e.g. some way of passing the result of the read_data command from lammps object to lammps object as I create new ones?
I’ll try and get back to you with valgrind results as soon as I can.
Seth
atom_style charge
units real
boundary f f f
read_data data.CrSi0
pair_style reax/c NULL
timestep 0.02
pair_coeff * * ffield.reax.test Cr H O Si
fix 1 all qeq/reax 1 0.0 10.0 1.0e-4 reax/c
run 0
variable a loop 10000
label loop
pair_coeff * * ffield.reax.test Cr H O Si
run 0 pre no post no
next a
jump in.mem_test loop
Thanks Axel, I’ll use the read_restart command for now while I try and identify the memory leak. I may also try the clear command and see if it gets around the memory leak so I don’t have the PyLammps initialization overhead every time.
I’ve avoided C++ so far because I have less experience writing C++ code and the development time would take much too long.
please note that the “time to solution” depends on both, the time spent on development and time spent on execution. so it may be worth upgrading your C++ knowledge.
if you worry about this, you should avoid PyLammps in any case and instead use either the “lammps” class in python or C/C++.
Another auxillary question I had is about the output from PyLammps. Every time I create a new PyLammps object, I get several lines that look like:
LAMMPS (29 Oct 2020)
OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:94)
using 1 OpenMP thread(s) per MPI task
Please see the log.cite file for references relevant to this simulation
Total wall time: 0:00:00
written to stdout. For my use case, this generates a LOT of unnecessary stuff written to stdout. Is there a way to suppress this output, or redirect it to a different buffer so it’s not in the stdout buffer?
when creating a LAMMPS instance you can pass standard command line arguments to the LAMMPS class that are also available for the LAMMPS executable: https://docs.lammps.org/Run_options.html
so it should be possible to suppress all output or divert it to files.
Please note that some of the PyLammps functionality depends on capturing and parsing the output to the screen (which is part of the reason why it has more overhead).
Thanks Axel, I’ll use the read_restart command for now while I try and identify the memory leak. I may also try the clear command and see if it gets around the memory leak so I don’t have the PyLammps initialization overhead every time.
I’ve avoided C++ so far because I have less experience writing C++ code and the development time would take much too long.
Another auxillary question I had is about the output from PyLammps. Every time I create a new PyLammps object, I get several lines that look like:
LAMMPS (29 Oct 2020)
OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:94)
using 1 OpenMP thread(s) per MPI task
Please see the log.cite file for references relevant to this simulation
Total wall time: 0:00:00
written to stdout. For my use case, this generates a LOT of unnecessary stuff written to stdout. Is there a way to suppress this output, or redirect it to a different buffer so it’s not in the stdout buffer?
Thanks again for your help,
Seth