Hello,
I have a problem identifying what's the peak of memory use of a
process in lammps. Let me explain.
I run lammps with the benchmark config file in.eam on 1024 MPI
processes (no OMP). I use the code from the tarball release from
"lammps-1Feb14". I removed all the packages except for manybody. The
app runs in a linux cluster.
I added "replicate 8 8 16" to the input file to scale the problem
to 1024 processes.
Sometime during the MPI_Finalize our customized MPI library gets the
value of the peak virtual memory(VmPeak) from /proc/self/status and
prints it out to a file. And for some reason I get this peak virtual
memory extremely huge: around 1.6GB. This is what I get if I just cat
/proc/someid/status:
Name: lmp_linux
State: R (running)
Tgid: 14884
Pid: 14884
PPid: 14697
TracerPid: 0
Uid: 10700 10700 10700 10700
Gid: 8000 8000 8000 8000
FDSize: 64
Groups: 8000 9004 9024 10000
VmPeak: 1686004 kB
VmSize: 1686004 kB
VmLck: 0 kB
VmHWM: 21856 kB
VmRSS: 21856 kB
VmData: 1592104 kB
VmStk: 136 kB
VmExe: 11732 kB
VmLib: 15928 kB
VmPTE: 196 kB
VmSwap: 0 kB
Threads: 1
However in the output of lammps it printed: Memory usage per processor
= 17.8377 Mbytes
I tried running smaller tests with in.eam on my laptop with ubuntu and
I get nowhere near that number: in a test with 8 MPI processes and 32
atoms the process' average memory peak is 49MB, if I replicate so that
I have 8 times more atoms and run on 8 processes the average memory is
still just around 62MB. I don't understand where does those GBs come
from?
Anyone has any idea?
best regards,
Hello,
I have a problem identifying what's the peak of memory use of a
process in lammps. Let me explain.
I run lammps with the benchmark config file in.eam on 1024 MPI
processes (no OMP). I use the code from the tarball release from
"lammps-1Feb14". I removed all the packages except for manybody. The
app runs in a linux cluster.
I added "replicate 8 8 16" to the input file to scale the problem
to 1024 processes.
Sometime during the MPI_Finalize our customized MPI library gets the
value of the peak virtual memory(VmPeak) from /proc/self/status and
prints it out to a file. And for some reason I get this peak virtual
memory extremely huge: around 1.6GB. This is what I get if I just cat
/proc/someid/status:
the VmPeak refers to reserved address space, not really used memory.
are you by chance using infiniband on your cluster? then this large
address space need would result from infiniband and not LAMMPS.
if my hunch is correct, you should consult the machine documentation
to enable using a "shared request queue" instead of the default, which
will massively reduce the amount of address space (and pinned memory)
required for your job.
mind you, the size of reserved address space, has little to do with
the amount of memory that LAMMPS is using for real and on top of that,
the output that LAMMPS sends to the screen is an estimate and often
incomplete, since only large allocations are being tracked and in some
cases they are not tracked at all.
axel.
Hello,
thanks for your reply. Indeed we use infiniband. But I don't
understand then, why I don't see similar behavior with other
applications? I use 4 different applications and only for lammps I
get such big VmPeak.
Tatiana
Hello,
thanks for your reply. Indeed we use infiniband. But I don’t
understand then, why I don’t see similar behavior with other
applications? I use 4 different applications and only for lammps I
get such big VmPeak.
I don’t know either, but please try out the shared request queue.