System & Simulation Details:
-
LAMMPS Version:
22Jul2025_update4(built with Kokkos CUDA enabled, double precision, cuFFT). -
Toolchain:
foss-2024a-kokkos-CUDA-12.6.0(EESSI 2025.06 repository). -
Hardware: NVIDIA GPU (managed via a 1g.20g MIG slice).
-
Model Size: ~30,900 atoms (Martini-like coarse-grained functionalized surfaces with polarizable water and graphene sheets).
More information about the system can be found here: reference_article. I’m trying to replicate the experiments reported there.
I’m pasting the output of lmp -h for a full list of compiling flags and available packages:
Large-scale Atomic/Molecular Massively Parallel Simulator - 22 Jul 2025
Usage example: lmp -var t 300 -echo screen -in in.alloy
List of command-line options supported by this LAMMPS executable:
-echo none/screen/log/both : echoing of input script (-e)
-help : print this help message (-h)
-in none/filename : read input from file or stdin (default) (-i)
-kokkos on/off ... : turn KOKKOS mode on or off (-k)
-log none/filename : where to send log output (-l)
-mdi '<mdi flags>' : pass flags to the MolSSI Driver Interface
-mpicolor color : which exe in a multi-exe mpirun cmd (-m)
-cite : select citation reminder style (-c)
-nocite : disable citation reminder (-nc)
-nonbuf : disable screen/logfile buffering (-nb)
-package style ... : invoke package command (-pk)
-partition size1 size2 ... : assign partition sizes (-p)
-plog basename : basename for partition logs (-pl)
-pscreen basename : basename for partition screens (-ps)
-restart2data rfile dfile ... : convert restart to data file (-r2data)
-restart2dump rfile dgroup dstyle dfile ...
: convert restart to dump file (-r2dump)
-restart2info rfile : print info about restart rfile (-r2info)
-reorder topology-specs : processor reordering (-r)
-screen none/filename : where to send screen output (-sc)
-skiprun : skip loops in run and minimize (-sr)
-suffix gpu/intel/kk/opt/omp: style suffix to apply (-sf)
-var varname value : set index style variable (-v)
OS: Linux "Rocky Linux 8.10 (Green Obsidian)" 4.18.0-553.27.1.el8_10.x86_64 x86_64
Compiler: GNU C++ 13.3.0 with OpenMP 4.5
C++ standard: C++17
Embedded fmt library version: 10.2.0
Embedded JSON class version: 3.12.0
MPI v3.1: Open MPI v5.0.3, package: Open MPI [email protected] Distribution, ident: 5.0.3, repo rev: v5.0.3, Apr 08, 2024
Accelerator configuration:
KOKKOS package API: CUDA OpenMP Serial
KOKKOS package precision: double
Kokkos library version: 4.6.2
OPENMP package API: OpenMP
OPENMP package precision: double
OpenMP standard: OpenMP 4.5
FFT information:
FFT precision = double
FFT engine = mpiFFT
FFT library = FFTW3 with threads
KOKKOS FFT engine = mpiFFT
KOKKOS FFT library = cuFFT
Active compile time flags:
-DLAMMPS_GZIP
-DLAMMPS_PNG
-DLAMMPS_JPEG
-DLAMMPS_FFMPEG
-DLAMMPS_SMALLBIG
sizeof(smallint): 32-bit
sizeof(imageint): 32-bit
sizeof(tagint): 32-bit
sizeof(bigint): 64-bit
Available compression formats:
Extension: .gz Command: gzip
Extension: .bz2 Command: bzip2
Extension: .zst Command: zstd
Extension: .xz Command: xz
Extension: .lzma Command: xz
Extension: .lz4 Command: lz4
Installed packages:
AMOEBA APIP ASPHERE ATC AWPMD BOCS BODY BPM BROWNIAN CG-DNA CG-SPICA CLASS2
COLLOID COLVARS COMPRESS CORESHELL DIELECTRIC DIFFRACTION DIPOLE DPD-BASIC
DPD-MESO DPD-REACT DPD-SMOOTH DRUDE EFF ELECTRODE EXTRA-COMPUTE EXTRA-DUMP
EXTRA-FIX EXTRA-MOLECULE EXTRA-PAIR FEP GRANULAR H5MD INTERLAYER KIM KOKKOS
KSPACE LATBOLTZ LEPTON MACHDYN MANIFOLD MANYBODY MC MDI MEAM MGPT MISC ML-IAP
ML-PACE ML-POD ML-RANN ML-SNAP MOFFF MOLECULE MOLFILE NETCDF OPENMP OPT ORIENT
PERI PHONON PLUGIN PLUMED POEMS PTM PYTHON QEQ QTB REACTION REAXFF REPLICA
RHEO RIGID SCAFACOS SHOCK SMTBQ SPH SPIN SRD TALLY UEF VORONOI VTK YAFF
Problem Description: When trying to run our production workload utilizing the KOKKOS package on the GPU, the simulation crashes immediately during the initial force evaluation (VerletKokkos::setup) before completing the first timestep. The execution finishes with a hardware protection fault: cudaErrorIllegalAddress: an illegal memory access was encountered.
Crucially, the exact same setup runs correctly using the standard non-accelerated CPU executable. If we run the CPU-only KOKKOS build (Kokkos::OpenMP backend), the simulation fails at the exact same execution point with a Segmentation fault.
We have isolated the crash to a pointer aliasing or indexing mismatch when pair_style hybrid/overlay interacts with explicit neighbor list exclusions (neigh_modify exclude) on sub-styles using long-range Coulombic lookup tables. Disabling Coulomb tables (pair_modify pair coul/long table 0) changes the trace slightly but does not prevent the out-of-bounds memory access.
Relevant Input Script Snippets:
pair_style hybrid/overlay lj/gromacs 9 12 coul/long 12
kspace_style pppm 1.0e-5
kspace_modify slab 3.0
pair_coeff * * coul/long
# Multiple individual per-type pairs mapped here, including ghost beads with zero-LJ epsilon
# Custom neighbor exclusions that filter the neighbor list topography:
neigh_modify exclude type 2 3
neigh_modify exclude type 5 6
# (Other custom pair exclusions through neigh_modify)
Execution command:
export CUDA_LAUNCH_BLOCKING=1
srun --mpi=pmix_v6 --ntasks=2 --cpus-per-task=3 \
lmp -k on g 1 -sf kk -pk kokkos neigh full newton off \
-in run.in.slide
Complete Standard Error / Backtrace:
(cuda_instance->cuda_event_record_wrapper( CudaInternal::constantMemReusablePerDevice[cuda_device])) error( cudaErrorIllegalAddress): an illegal memory access was encountered /dev/shm/easybuild/LAMMPS/22Jul2025/foss-2024a-kokkos-CUDA-12.6.0/lammps-stable_22Jul2025/lib/kokkos/core/src/Cuda/Kokkos_Cuda_KernelLaunch.hpp:608
Backtrace:
Kokkos::Impl::save_stacktrace()
Kokkos::Impl::host_abort(char const*)
Kokkos::Impl::cuda_internal_error_abort(cudaError, char const*, char const*, int)
Kokkos::Impl::ParallelReduceAdaptor<Kokkos::TeamPolicy<Kokkos::Cuda, Kokkos::IndexType<int> >, LAMMPS_NS::PairComputeFunctor<LAMMPS_NS::PairCoulLongKokkos<Kokkos::Cuda>, 1, false, 0, LAMMPS_NS::CoulLongTable<1> >, s_EV_FLOAT>::execute_impl(...)
s_EV_FLOAT LAMMPS_NS::pair_compute_neighlist<LAMMPS_NS::PairCoulLongKokkos<Kokkos::Cuda>, 1u, 0, LAMMPS_NS::CoulLongTable<1> >(...)
s_EV_FLOAT LAMMPS_NS::pair_compute<LAMMPS_NS::PairCoulLongKokkos<Kokkos::Cuda>, LAMMPS_NS::CoulLongTable<1> >(...)
LAMMPS_NS::PairCoulLongKokkos<Kokkos::Cuda>::compute(int, int)
LAMMPS_NS::PairHybridKokkos::compute(int, int)
LAMMPS_NS::VerletKokkos::setup(int)
LAMMPS_NS::Run::command(int, char**)
LAMMPS_NS::Input::execute_command()
main
Is this a known structural issue in the 22Jul2025 build where the thread team policy in PairCoulLongKokkos loops over pre-filtered sub-neighbor list views without proper index boundary mapping? Has this behavior been patched in a more recent stable release or develop branch commit?
Any suggestions or workarounds that don’t involve altering the physical potentials or removing the neighbor exclusions would be highly appreciated.
Objective:
My main objective is to run the charge calculation in the GPU to speedup the simulation while keeping the physics of the reference article. Currently I’m having a performance of around 17 ns/day for the system containing charged beads, with kspace taking 88% of the computing time. For the system without charged beads I managed to reach 90 ns/day using kspace_style pppm/omp 1.0e-5 and the following lmp command:
srun \
--mpi=pmix_v6 \
--ntasks=2 \
--cpus-per-task=3 \
--chdir="${SIM_DIR}" \
lmp \
-sf hybrid gpu omp \
-pk gpu 1 neigh no \
-pk omp 3 \
-in "${INPUT_FILE}" \
-l "${LOG_FILE}" &
Though 90 ns/day is a bit low compared to the performance I achieved with GROMACS for the same system (NOTE that with GROMACS I cannot exactly replicate the very specific thermostatting or the rigid body behavior but 180 ns/day were reached.) It makes the simulation feasible.
Thank you in advance for any possible help. I will be glad to provide any further information or input needed.
Alfonso