lammps-19Jul11 : large virtual memory consumption

dear theo,

please always keep the list in cc:

Dear Axel,

Thanks for your very quick reply.
Indeed, our cluster uses Infiniband connections. I'll back up with our system administrator to see what to do with respect to the MPI use.
Do you have some (general) advice?

this is a typical behavior of infiniband. most MPI implementations
have an O(N**2) requirement with the number of MPI tasks of this
backing storage address space, which can exhaust your virtual
(and physical) memory quickly when you run across larger numbers
of nodes. however, for most implementations, you can switch to
using a "shared request queue" (= SRQ) which will significantly
lower that requirement (and often even speed up the communication),
but it will still always be large. but most of it will be mapped to the
same physical memory, so there is not so much to worry about.

we use OpenMPI instead of Intel MPI where you can use the
--mca btl_openib_use_srq 1 flag to switch to SRQ, which has
been very beneficial on any of the machines that i've been running on.

Actually, due to core-instabilities the system administrator wanted to switch off swap memory on the all nodes of our cluster. However, due to fact that LAMMPS didn't run anymore without swap memory, I came beware of the huge swap memory LAMMPS uses.

that is a very odd way to deal with this kind of problem.

axel.

Dear Axel,

dear theo,

please always keep the list in cc:

Oupss, I'm sorry about that!

> Dear Axel,
>
> Thanks for your very quick reply.
> Indeed, our cluster uses Infiniband connections. I'll back
up with our system administrator to see what to do with
respect to the MPI use.
> Do you have some (general) advice?

this is a typical behavior of infiniband. most MPI implementations
have an O(N**2) requirement with the number of MPI tasks of this
backing storage address space, which can exhaust your virtual
(and physical) memory quickly when you run across larger numbers
of nodes. however, for most implementations, you can switch to
using a "shared request queue" (= SRQ) which will significantly
lower that requirement (and often even speed up the communication),
but it will still always be large. but most of it will be
mapped to the
same physical memory, so there is not so much to worry about.

we use OpenMPI instead of Intel MPI where you can use the
--mca btl_openib_use_srq 1 flag to switch to SRQ, which has
been very beneficial on any of the machines that i've been running on.

> Actually, due to core-instabilities the system
administrator wanted to switch off swap memory on the all
nodes of our cluster. However, due to fact that LAMMPS didn't
run anymore without swap memory, I came beware of the huge
swap memory LAMMPS uses.

that is a very odd way to deal with this kind of problem.

Thanks for your quick and very instructive reply!
I'll send this information to our system administators and hope that they can repair this problem.
Kindest regards
Theo

LAMMPS does not use any swap memory, unless you run a problem
larger than you have physical memory.

MPI may use swap memory internally, but that is an MPI issue,
not a LAMMPS issue.

Steve

Dear All,

It seemed that the huge swap memory consumption was due to the increase of some variables in reax_defs.h (e.g. NATDEF, NBOALLMAXDEF...)
I increased those number since I'd encountered errors complaining that there too many bonds during the simulations.
However, this problem could be solved (or at lease circumvented) by increasing the number of cores.
A compilation with the default values in reax_defs.h (but with IntelMPI 4.0) results in a normal memory consumption behavior.

Thanks again for your input.
Kind regards
Theo

Dr. Theo de Bruin
Research Engineer

IFP Energies nouvelles - Applied Chemistry and Physical Chemistry Division

1 et 4 avenue de Bois-Préau
92852 Rueil-Malmaison Cedex - France
Phone: +33 1 47 52 54 38 - Fax: +33 1 47 52 70 58
Email: [email protected]...
Web: www.ifpenergiesnouvelles.fr

You can also try the USER-REAXC package which does dynamic
memory allocation instead of the static mode used by the Fortran lib.

Steve