Kokkos + 2080 GPU Runs out of Shared Memory with 3 Atomic Species

LAMMPS LAMMPS General Discussion
,
1

I’ve been running some AlGaN simulations using an ACE potential. I have no problems running the model on CPU, however, I have intermittent issues on a 2080 SUPER GPU and I am not sure how to debug them. I notice the error when I use an AlGaN structure, but not the GaN structure. So maybe related to the number of species.

When I run the attached script (with structure.data) on a single GPU with the following command:

lmp -in npt_equilirbate.in -k on g 1 -sf kk -pk kokkos newton on neigh half

I get the run out of shared memory error after 20-40 timesteps. A 2080 SUPER does not have the most shared memory (~64kB), but the shared memory size does not tend to increase that much generation to generation so I was hoping to at least understand if this actually is a hardware limitation or something incorrect with what I am doing.

Exception: Kokkos::Impl::ParallelFor< Cuda > insufficient shared memory
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

I noticed I do not get this error when I only use the gan_structure.data file. So is this just a fundamental limitation of me having a 2080 instead of something nicer, or am I doing something subtly incorrect?? Thanks!

algan_files.tar.gz (57.1 KB)

2

@ejmeitz Thanks for providing an input deck for reproducing the issue.

Warning: This reply was drafted with the help of an AI assistant (Claude) that did the research through the Kokkos source code and reviewed and edited by me before posting.

The problem is not the number of species but the maximum number of neighbors an atom has. Also, switching to a newer GPU won’t help because Kokkos uses the same size shared memory buffer on all NVIDIA GPUs. The shared memory with Kokkos is limited to 48kB per block. Newer NVIDIA GPU hardware has more available but it must be activated. These 48kB translate to a maximum of about 380 neighbors. For your inputs we have an average number of neighbors of 339 for the case that runs and 414 for the case that crashes. So you are right at the limit of what can be done and one of the two systems is under (but could randomly crash when one atom temporarily has more neighbors due to a density fluctuation) and the other over for most atoms. I have tested this on a wimpy AMD Radeon 780M embedded GPU and I don’t hit the same limit because for HIP the shared memory is limited to 64kB per block (or about 510 neighbors max.).

So it is neither a fundamental hardware limitation nor a mistake on your side, but a limit in either Kokkos (that could opt-in to enable the full 64kB shared memory on your card) or the KOKKOS pair style implementation (that could reduce the team size to not hit the limit). I will let the coding agent submit a feature request issue with this information.

Update: [Feature Request] Work around GPU scratch limits with ML-PACE pair styles · Issue #5063 · lammps/lammps · GitHub

1 Like
3

@akohlmey Great! Thanks for the information.

This is a pretty tiny system. Only 72 atoms. The cutoffs are roughly 4-6 Ang which lines up with the ~400 neighbors total (only capturing nearest neighbors).

I imagine larger systems would have far more neighbors than my system. Like if I ran LJ argon with 256 atom supercell, that has a cutoff around 8.5 Angstroms. LJ Argon must have more than 400 total neighbors. So will pair_style pace hit this shared memory wall for any moderately sized system? I’m a bit shocked I’m the first to run into this. Am I understanding the diagnosis wrong??

4

You do.
This is not about total number of neighbors, but about maximum number of neighbors of any atom. So system size has no impact. Also, it is specific to the ML-PACE pair styles.
Furthermore, the output from your input indicates a neighbor list cutoff of 10 angstrom (i.e. inclusive of neighbor list skin).

The LAMMPS output says 24408 neighbors

5

Ah yes, I was under the impression the ACE potential had a cutoff of 4-6, but it was in fact 8. This captures second-nearest neighbors and with the neighbor skin even more atoms.

Thanks for the help!

6

We’ve run into this before. The easy solution is to switch from scratch level 0 (shared memory) to scratch level 1 (global memory), either automatically or as an option in the input. I will look at this.