problems with increasing the number of cores

Dear all,

I have encountered a problem when increasing the number of cores
in my simulation. I ran the same input script on 64, 512, 2048,
and 8192 cores. The simulations worked fine for 64, 512, and 2048 cores,
but on 8192 cores the simulation did not pass the setup until the simulation
was killed by the scheduling system.

The system I am running is a bulk system of water with pbcs. The number of
atoms is approx. 5 Mio. The box lengths are 800 x 80 x 800 Angstroms^3.
Could the problem be related to the shape of the box?
Has anyone encountered similar problems before?
Does anyone have an idea of where I should start to search for the reason
of the problem?

The log files and the LAMMPS input file are attached.

Best,
Rolf

64cores.log (1.95 KB)

512cores.log (1.97 KB)

2048cores.log (1.99 KB)

8192cores.log (1.07 KB)

input.lammps (301 Bytes)

Dear all,

I have encountered a problem when increasing the number of cores
in my simulation. I ran the same input script on 64, 512, 2048,
and 8192 cores. The simulations worked fine for 64, 512, and 2048 cores,
but on 8192 cores the simulation did not pass the setup until the simulation
was killed by the scheduling system.

do you have the message from the scheduler, indicating the reason?

The system I am running is a bulk system of water with pbcs. The number of
atoms is approx. 5 Mio. The box lengths are 800 x 80 x 800 Angstroms^3.
Could the problem be related to the shape of the box?

unlikely.

Has anyone encountered similar problems before?

yes. not sure if this is still and issue, but a while back, many MPI
libraries for infiniband networks used pinned memory (i.e. memory with
a fixed relation between address space and physical address, which
makes it non-swappable) in O(N**2) fashion (with N being the number of
total MPI tasks), because of better performance for several
benchmarks. at over 512 cores this amount of inaccessible RAM would
get close to 1GB per core and thus forcing applications to swap
excessively or crash due to exceeding the preconfigured per process
memory limits. the solution was to set an option or environment
variable to use a "shared request queue" (SRQ) instead, which has O(N)
memory consumption.

Does anyone have an idea of where I should start to search for the reason
of the problem?

the first place to look is the logs of the batch system and the second
option is to ask the system manager of the cluster for assistance.
unless you accidentally hit a case, where lammps tries to allocate
memory based on an uninitialized variable (which can usually be
identified with valgrind), there is very little in LAMMPS, that could
cause problems.

the second, less likely case, would be if you have domains that
contain no atoms. a while back, we did some careful testing for a lot
of features, to make them compatible with this, but some changes to
the contrary may have crept in. quite a few features in LAMMPS are
developed for dense system and not tested on sparse simulations,
causing the occasional surprise. given the increasing popularity of
LAMMPS, these things are less likely to go unnoticed for inputs that
don't use any "exotic" features, tho.

HTH,
     axel.

Hi,

thanks for your help Axel.

I found the problem and also have a bug fix:
In atom->map_init(), there is an MPI_Allrecude. However,
the map_init() function is in my test case only called by a
few of the cores, which is why the code stops at this point.

I've attached a corrected version that fixes the problem. The
problem is labeled with REI. I'm think the solution that is
proposed in the attached file should work fine, but it would be
nice if someone would have a brief look at it to confirm the
modification.

Best,
Rolf

atom_map.cpp (10.9 KB)

Hi,

thanks for your help Axel.

I found the problem and also have a bug fix:
In atom->map_init(), there is an MPI_Allrecude. However,
the map_init() function is in my test case only called by a
few of the cores, which is why the code stops at this point.

right. this seems to be a side effect from the move to allow 64-bit atom tags.

I've attached a corrected version that fixes the problem. The
problem is labeled with REI. I'm think the solution that is
proposed in the attached file should work fine, but it would be
nice if someone would have a brief look at it to confirm the
modification.

yes, this seems reasonable. i'll clean it up a little bit and also
check for all other locations that call atom->map_init(), as they may
be affected as well, and then send the result to steve. i have a bunch
of stuff to (re)-send him anyway...

axel.

Rolf - can you test the 2 attached files and see if
they fix the problem. I think the correct logic
is to only do the MPI_Allreduce in map_init() when the max atomID
has changed, which is not the case from within map_set().

All the other calls to map_init(), elsewhere in the code,
should be fine.

Steve

atom.cpp (57 KB)

atom.h (12.8 KB)