Problems when compiling Lammps with Kokkos on GPU

Dear all,

I am trying to compile LAMMPS with the Kokkos package on GPU to be able to run simulations employing a MACE ML trained model. From the MACE documentation (MACE in LAMMPS — mace 0.1.0 documentation) and from crosschecking with the documentation of LAMMPS I tried to compile LAMMPS using two presets “most” and “kokkos-cuda”. the command “cmake -C …/cmake/presets/most.cmake -C …/cmake/presets/kokkos-cuda.cmake …/cmake” works find and prepares the compilation, but after “cmake --build .” the compilation always crashed in the very first steps when building the voronoi package:
"
– extracting… done
[ 1%] No update step for ‘voro_build’
[ 1%] Performing patch step for ‘voro_build’
patching file src/Makefile
[ 1%] No configure step for ‘voro_build’
[ 1%] Performing build step for ‘voro_build’
/tmp/tmpxft_00005216_00000000-45_cell.cpp1.ii(1): error: this declaration has no storage class or type specifier

/tmp/tmpxft_00005216_00000000-45_cell.cpp1.ii(1): error: expected a “;”

2 errors detected in the compilation of “cell.cc”.
make[4]: *** [Makefile:19: depend] Error 1
make[3]: *** [Makefile:14: all] Error 2
gmake[2]: *** [CMakeFiles/voro_build.dir/build.make:86: voro_build-prefix/src/voro_build-stamp/voro_build-build] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:507: CMakeFiles/voro_build.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2
"

I manually removed the Voronoi package from the “most” preset whih led to:

"
[ 0%] Building CXX object lib/kokkos/simd/src/CMakeFiles/kokkossimd.dir/Kokkos_SIMD_dummy.cpp.o
[ 0%] Linking CXX static library libkokkossimd.a
[ 0%] Built target kokkossimd
[ 0%] Building CXX object CMakeFiles/lepton.dir/home/jabr/md-software/lammps-29Aug2024/lib/lepton/src/CompiledExpression.cpp.o
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘…’:
435 | function(_Functor&& __f)
| ^
/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘…’:
530 | operator=(_Functor&& __f)
| ^
/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’
gmake[2]: *** [CMakeFiles/lepton.dir/build.make:76: CMakeFiles/lepton.dir/home/jabr/md-software/lammps-29Aug2024/lib/lepton/src/CompiledExpression.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:503: CMakeFiles/lepton.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2
"
upon compiling.
I hope that maybe someone can provide some tips on how to resolve this.
For completeness, below I show the message after “cmake -C […] …/cmake”:

"
loading initial cache file …/cmake/presets/most.cmake
loading initial cache file …/cmake/presets/kokkos-cuda.cmake
CMake Deprecation Warning at CMakeLists.txt:18 (cmake_policy):
The OLD behavior for policy CMP0109 will be removed from a future version
of CMake.

The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.

  • The CXX compiler identification is GNU 11.4.0
    – Detecting CXX compiler ABI info
    – Detecting CXX compiler ABI info - done
    – Check for working CXX compiler: /home/jabr/md-software/lammps-29Aug2024/lib/kokkos/bin/nvcc_wrapper - skipped
    – Detecting CXX compile features
    – Detecting CXX compile features - done
    – Found Git: /usr/bin/git (found version “2.34.1”)
    – Running check for auto-generated files from make-based build system
    – Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version “3.1”)
    – Found MPI: TRUE (found version “3.1”)
    – Looking for C++ include omp.h
    – Looking for C++ include omp.h - found
    – Found OpenMP_CXX: -fopenmp (found version “4.5”)
    – Found OpenMP: TRUE (found version “4.5”) found components: CXX
    – The C compiler identification is GNU 11.4.0
    – Detecting C compiler ABI info
    – Detecting C compiler ABI info - done
    – Check for working C compiler: /usr/bin/cc - skipped
    – Detecting C compile features
    – Detecting C compile features - done
    – Looking for sgemm_
    – Looking for sgemm_ - not found
    – Performing Test CMAKE_HAVE_LIBC_PTHREAD
    – Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
    – Found Threads: TRUE
    – Looking for sgemm_
    – Looking for sgemm_ - found
    – Found BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so
    – Looking for cheev_
    – Looking for cheev_ - found
    – Found LAPACK: /usr/lib/x86_64-linux-gnu/libopenblas.so;-lm;-ldl
    – Found GZIP: /usr/bin/gzip
    – Could NOT find FFMPEG (missing: FFMPEG_EXECUTABLE)
    – Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
    – Could NOT find Eigen3 (missing: Eigen3_DIR)
    – Eigen3 download requested - we will build our own
    – Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version “1.2.11”)
    – Looking for C++ include cmath
    – Looking for C++ include cmath - found
    – Downloading external potential C_10_10.mesocnt from https://download.lammps.org/potentials
    – [download 0% complete]
    […]
    – [download 100% complete]
    – KOKKOS: Enabling CUDA LAMBDA function support
    – Setting default Kokkos CXX standard to 17
    – Kokkos version: 4.3.1
    – The project name is: Kokkos
    – Using internal gtest for testing
    – Compiler Version: 11.5.119
    – Using -std=c++17 for C++17 standard as feature
    – Built-in Execution Spaces:
    – Device Parallel: Kokkos::Cuda
    – Host Parallel: NoTypeDefined
    – Host Serial: SERIAL

    – Architectures:
    – PASCAL60
    – Found CUDAToolkit: /usr/include (found version “11.5.119”)
    – Found TPLCUDA: TRUE
    – Found TPLLIBDL: /usr/include
    – Using internal desul_atomics copy
    – Kokkos Backends: SERIAL;CUDA
    – Generating style headers…
    – Generating package headers…
    – Generating lmpinstalledpkgs.h…
    – Looking for a Fortran compiler
    – Looking for a Fortran compiler - /usr/bin/f95
    – The Fortran compiler identification is GNU 11.4.0
    – Detecting Fortran compiler ABI info
    – Detecting Fortran compiler ABI info - done
    – Check for working Fortran compiler: /usr/bin/f95 - skipped
    – Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
    – Could NOT find FFTW3 (missing: FFTW3_LIBRARY FFTW3_INCLUDE_DIR)
    – Found Python3: /home/jabr/miniconda3/bin/python3.9 (found version “3.9.12”) found components: Interpreter
    – Could NOT find ClangFormat (missing: ClangFormat_EXECUTABLE) (Required is at least version “11.0”)
    – The following tools and libraries have been found and configured:
  • MPI
  • OpenMP
  • LAPACK
  • BLAS
  • ZLIB
  • TPLCUDA
  • TPLLIBDL
  • Threads
  • CUDAToolkit
  • Python3

– <<< Build configuration >>>
LAMMPS Version: 20240829
Operating System: Linux Ubuntu" 22.04
CMake Version: 3.27.2
Build type: RelWithDebInfo
Install path: /home/jabr/.local
Generator: Unix Makefiles using /usr/bin/gmake
– Enabled packages: AMOEBA;ASPHERE;BOCS;BODY;BPM;BROWNIAN;CG-DNA;CG-SPICA;CLASS2;COLLOID;COLVARS;COMPRESS;CORESHELL;DIELECTRIC;DIFFRACTION;DIPOLE;DPD-BASIC;DPD-MESO;DPD-REACT;DPD-SMOOTH;DRUDE;EFF;ELECTRODE;EXTRA-COMMAND;EXTRA-COMPUTE;EXTRA-DUMP;EXTRA-FIX;EXTRA-MOLECULE;EXTRA-PAIR;FEP;GRANULAR;INTERLAYER;KOKKOS;KSPACE;LEPTON;MACHDYN;MANYBODY;MC;MEAM;MESONT;MISC;ML-IAP;ML-POD;ML-SNAP;ML-UF3;MOFFF;MOLECULE;OPENMP;OPT;ORIENT;PERI;PHONON;PLUGIN;POEMS;QEQ;REACTION;REAXFF;REPLICA;RIGID;SHOCK;SPH;SPIN;SRD;TALLY;UEF;YAFF
– <<< Compilers and Flags: >>>
– C++ Compiler: /home/jabr/md-software/lammps-29Aug2024/lib/kokkos/bin/nvcc_wrapper
Type: GNU
Version: 11.4.0
C++ Standard: 17
C++ Flags: -O2 -g -DNDEBUG
Defines: LAMMPS_SMALLBIG;LAMMPS_MEMALIGN=64;LAMMPS_OMP_COMPAT=4;LAMMPS_GZIP;FFT_KISS;EIGEN_NO_CUDA;LMP_OPENMP;$<BUILD_INTERFACE:LMP_KOKKOS>;FFT_KOKKOS_CUFFT;LMP_PLUGIN
Options: -Xcudafe;–diag_suppress=unrecognized_pragma
– Fortran Compiler: /usr/bin/f95
Type: GNU
Version: 11.4.0
Fortran Flags: -O2 -g -DNDEBUG
– C compiler: /usr/bin/cc
Type: GNU
Version: 11.4.0
C Flags: -O2 -g -DNDEBUG
– <<< Linker flags: >>>
– Executable name: lmp
– Static library flags:
– <<< MPI flags >>>
– MPI_defines: MPICH_SKIP_MPICXX;OMPI_SKIP_MPICXX;_MPICC_H
– MPI includes: /usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
– MPI libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so;/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so;
– Kokkos Devices: CUDA;CUDA_LAMBDA;SERIAL
– Kokkos Architecture: PASCAL60
– <<< FFT settings >>>
– Primary FFT lib: KISS
– Using double precision FFTs
– Using non-threaded FFTs
– Using builtin distributed FFT algorithms
– Kokkos FFT: CUFFT
– <<< Building Tools >>>
– Configuring done (74.5s)
– Generating done (0.2s)
– Build files have been written to: /home/jabr/md-software/lammps-29Aug2024/build
"

This package is not part of LAMMPS and thus you should be contacting the developer of that package for assistance.

If you have problems compiling with many packages, why don’t you switch to the “basic” preset first or only compile the packages of primary interest first and then enable additional packages as you need them. Compilation can be incremental. That will avoid chasing down compilation issues due to limitations of nvcc, the CUDA compiler, that are peripheral to or outside of the scope of what you are trying to achieve.

I would also try a gcc 12.x compiler instead of gcc 11.x but not 13.x . Nvcc is rather sensitive to which compiler it’s running on top of.

Are you sure you want to compile for PASCAL60, ie. nvidia P100 gpu from 2007 ?

Following both answers I tried to build lammps with only the kokkos package. with the kokkos-serial and kokkos-openmp preset everything works fine. When I try to compile with the kokkos-cuda preset the compilation crashes everytime at
"

Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Abort.cpp.o
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
gmake[2]: *** [lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/build.make:76: lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Abort.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2443: lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

"

I have installed gcc 12 with “sudo apt install gcc-12” but I seem to be unable to request it for the compilation. When I execute “which gcc” it shows “/usr/bin/gcc → gcc-11”. Also I switched to the TURING75 architecture, which seems to be the right choice for my PC. The output of cmake -C preset/kokkos-cuda is:
"

loading initial cache file ../cmake/presets/kokkos-cuda.cmake

- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/jabr/md-software/lammps-29Aug2024/lib/kokkos/bin/nvcc_wrapper - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.34.1")
-- Running check for auto-generated files from make-based build system
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Looking for C++ include omp.h
-- Looking for C++ include omp.h - found
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5") found components: CXX
-- Found GZIP: /usr/bin/gzip
-- Could NOT find FFMPEG (missing: FFMPEG_EXECUTABLE)
-- Looking for C++ include cmath
-- Looking for C++ include cmath - found
-- KOKKOS: Enabling CUDA LAMBDA function support
-- Setting default Kokkos CXX standard to 17
-- Kokkos version: 4.3.1
-- The project name is: Kokkos
-- Using internal gtest for testing
-- Compiler Version: 11.5.119
-- Using -std=c++17 for C++17 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
--
-- Architectures:
--  TURING75
-- Found CUDAToolkit: /usr/include (found version "11.5.119")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found TPLCUDA: TRUE
-- Found TPLLIBDL: /usr/include
-- Using internal desul_atomics copy
-- Kokkos Backends: SERIAL;CUDA
-- Generating style headers...
-- Generating package headers...
-- Generating lmpinstalledpkgs.h...
-- Found Python3: /home/jabr/miniconda3/bin/python3.9 (found version "3.9.12") found components: Interpreter
-- Could NOT find ClangFormat (missing: ClangFormat_EXECUTABLE) (Required is at least version "11.0")
-- The following tools and libraries have been found and configured:
 * MPI
 * OpenMP
 * Threads
 * CUDAToolkit
 * TPLCUDA
 * TPLLIBDL
 * Python3

-- <<< Build configuration >>>
   LAMMPS Version:   20240829
   Operating System: Linux Ubuntu" 22.04
   CMake Version:    3.27.2
   Build type:       RelWithDebInfo
   Install path:     /home/jabr/.local
   Generator:        Unix Makefiles using /usr/bin/gmake
-- Enabled packages: KOKKOS
-- <<< Compilers and Flags: >>>
-- C++ Compiler:     /home/jabr/md-software/lammps-29Aug2024/lib/kokkos/bin/nvcc_wrapper
      Type:          GNU
      Version:       11.4.0
      C++ Standard:  17
      C++ Flags:     -O2 -g -DNDEBUG
      Defines:       LAMMPS_SMALLBIG;LAMMPS_MEMALIGN=64;LAMMPS_OMP_COMPAT=4;LAMMPS_GZIP;$<BUILD_INTERFACE:LMP_KOKKOS>
      Options:       -Xcudafe;--diag_suppress=unrecognized_pragma
-- <<< Linker flags: >>>
-- Executable name:  lmp
-- Static library flags:
-- <<< MPI flags >>>
-- MPI_defines:      MPICH_SKIP_MPICXX;OMPI_SKIP_MPICXX;_MPICC_H
-- MPI includes:     /usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
-- MPI libraries:    /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so;/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so;
-- Kokkos Devices: CUDA;CUDA_LAMBDA;SERIAL
-- Kokkos Architecture: TURING75
-- Configuring done (11.7s)
-- Generating done (0.1s)
-- Build files have been written to: /home/jabr/md-software/lammps-29Aug2024/build_on_gpu

"
I could not yet figure out how to tell cmake to use the gcc 12 compiler, maybe this is the core issue at hand.

Thanks in advance for any help!

So what do you see with “which gcc-12” and (more importantly) “which g++-12”?

The issue is not CMake, but rather the nvcc_wrapper scripts used by Kokkos. It will default to using “g++” as compiled and you can overwrite this with the environment variable NVCC_WRAWPPER_DEFAULT_COMPILER. So, try wiping out your build folder, run

export NVCC_WRAPPER_DEFAULT_COMPILER=g++-12

and then rerun CMake to build with KOKKOS and CUDA.

1 Like