Compiling LAMMPS with Kokkos

ntiwari · February 10, 2022, 11:33pm

I posted yesterday about building a Docker image of LAMMPS with Kokkos support. However, I’m having some trouble compiling.

1.) I’m using the most recent, stable version of LAMMPS (29 Sep 2021)
2.) I’m compiling on the Nvidia HPC SDK Docker image. The OS is Ubuntu 20.04, the CUDA version is 11.2, and OpenMPI is version 3.1.5.
3.) I’ve made a single modification, and that is to Makefile.kokkos_cuda_mpi. I set KOKKOS_ARCH = Volta70. I also tried explicitly setting KOKKOS_ABSOLUTE_PATH to the location of the nvcc_wrapper executable included with LAMMPS. But that didn’t help.
4.) I tried using CMake (version 3.22), but I’m getting errors regarding the locations of header files. I more or less gave up on this and have put the majority of my time into regular make.

The error I get is: CUDA is enabled but the compiler is neither NVCC nor Clang(got version string nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Nov_30_19:08:53_PST_2020 Cuda compilation tools, release 11.2, V11.2.67 Build cuda_11.2.r11.2/compiler.29373293_0). Everything looks as it should from the documentation, so I’m stumped.

Thanks,
Nick

akohlmey · February 11, 2022, 12:55am

I cannot reproduce this. I am using the same stable version with no modifications to any file and start the container in the toplevel folder of the lammps source:

$ singularity shell --nv ./nvidia-hpc-sdk.sif 
Singularity> cd src
Singularity> make yes-basic
make[1]: Entering directory '/usr/local/lammps/src'
Installing package kspace
make[1]: Leaving directory '/usr/local/lammps/src'
make[1]: Entering directory '/usr/local/lammps/src'
Installing package manybody
make[1]: Leaving directory '/usr/local/lammps/src'
make[1]: Entering directory '/usr/local/lammps/src'
Installing package molecule
make[1]: Leaving directory '/usr/local/lammps/src'
make[1]: Entering directory '/usr/local/lammps/src'
Installing package rigid
make[1]: Leaving directory '/usr/local/lammps/src'
Singularity> make yes-kokkos
Installing package kokkos
Singularity> make KOKKOS_PATH=$PWD/../lib/kokkos KOKKOS_ARCH=Volta70 kokkos_cuda_mpi
Gathering installed package information (may take a little while)
make[1]: Entering directory '/usr/local/lammps/src'
Gathering git version information
make[1]: Leaving directory '/usr/local/lammps/src'
Compiling LAMMPS for machine kokkos_cuda_mpi
make[1]: Entering directory '/usr/local/lammps/src/Obj_kokkos_cuda_mpi'
make[1]: Leaving directory '/usr/local/lammps/src/Obj_kokkos_cuda_mpi'
make[1]: Entering directory '/usr/local/lammps/src/Obj_kokkos_cuda_mpi'
mpicxx -g -O3  -DLAMMPS_GZIP -DLMP_KOKKOS  -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -DFFT_CUFFT	  -I./ -I/usr/local/lammps/src/../lib/kokkos/core/src -I/usr/local/lammps/src/../lib/kokkos/containers/src -I/usr/local/lammps/src/../lib/kokkos/algorithms/src -std=c++14 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -expt-extended-lambda -arch=sm_70 -I./ -I/usr/local/lammps/src/../lib/kokkos/core/src -I/usr/local/lammps/src/../lib/kokkos/containers/src -I/usr/local/lammps/src/../lib/kokkos/algorithms/src  -c ../main.cpp
mpicxx -g -O3  -DLAMMPS_GZIP -DLMP_KOKKOS  -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -DFFT_CUFFT	  -I./ -I/usr/local/lammps/src/../lib/kokkos/core/src -I/usr/local/lammps/src/../lib/kokkos/containers/src -I/usr/local/lammps/src/../lib/kokkos/algorithms/src -std=c++14 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -expt-extended-lambda -arch=sm_70 -I./ -I/usr/local/lammps/src/../lib/kokkos/core/src -I/usr/local/lammps/src/../lib/kokkos/containers/src -I/usr/local/lammps/src/../lib/kokkos/algorithms/src  -c ../pair_deprecated.cpp
mpicxx -g -O3  -DLAMMPS_GZIP -DLMP_KOKKOS  -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -DFFT_CUFFT	  -I./ -I/usr/local/lammps/src/../lib/kokkos/core/src -I/usr/local/lammps/src/../lib/kokkos/containers/src -I/usr/local/lammps/src/../lib/kokkos/algorithms/src -std=c++14 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -expt-extended-lambda -arch=sm_70 -I./ -I/usr/local/lammps/src/../lib/kokkos/core/src -I/usr/local/lammps/src/../lib/kokkos/containers/src -I/usr/local/lammps/src/../lib/kokkos/algorithms/src  -c ../pair_lj_cut_coul_cut_kokkos.cpp

akohlmey · February 11, 2022, 1:25am

For compiling with CMake a little trickery is needed because nvidia sets up the paths in their containers in a way that confuses the script in the KOKKOS library that finds the location of the CUDA installation and its libraries. It worked the following way:

$ singularity shell --nv ./nvidia-hpc-sdk.sif 
Singularity> env PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/cuda/11.5/bin:$PATH cmake -S cmake -B build-nvidia -C cmake/presets/kokkos-cuda.cmake -C cmake/presets/basic.cmake -D Kokkos_ARCH_MAXWELL50=off -D Kokkos_ARCH_VOLTA70=on 
loading initial cache file cmake/presets/kokkos-cuda.cmake
loading initial cache file cmake/presets/basic.cmake
-- The CXX compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/local/lammps/lib/kokkos/bin/nvcc_wrapper
-- Check for working CXX compiler: /usr/local/lammps/lib/kokkos/bin/nvcc_wrapper -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- Appending /usr/local/gdrcopy/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/hcoll/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/nccl_rdma_sharp_plugin/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/sharp/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ucx/mt/lib: to CMAKE_LIBRARY_PATH: /usr/local/gdrcopy/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/hcoll/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/nccl_rdma_sharp_plugin/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/sharp/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ucx/mt/lib:
-- Running check for auto-generated files from make-based build system
-- Found MPI_CXX: /opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/lib/libmpi.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Looking for C++ include omp.h
-- Looking for C++ include omp.h - found
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Found GZIP: /usr/bin/gzip  
-- Could NOT find FFMPEG (missing: FFMPEG_EXECUTABLE) 
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
-- Looking for C++ include cmath
-- Looking for C++ include cmath - found
-- KOKKOS: Enabling CUDA LAMBDA function support
-- Setting default Kokkos CXX standard to 14
-- Setting policy CMP0074 to use <Package>_ROOT variables
-- The project name is: Kokkos
-- Compiler Version: 11.5.119
-- Using -std=c++14 for C++14 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: Kokkos::OpenMP
--       Host Serial: SERIAL
-- 
-- Architectures:
--  VOLTA70
-- Found CUDAToolkit: /opt/nvidia/hpc_sdk/Linux_x86_64/22.1/cuda/11.5/include (found version "11.5.119") 
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found TPLCUDA: TRUE  
-- Found TPLLIBDL: /usr/lib/x86_64-linux-gnu/libdl.so  
-- Kokkos Devices: OPENMP;CUDA;SERIAL, Kokkos Backends: OPENMP;CUDA;SERIAL
-- Generating style headers...
-- Generating package headers...
-- Generating lmpinstalledpkgs.h...
-- Could NOT find ClangFormat (missing: ClangFormat_EXECUTABLE) (Required is at least version "8.0")
-- The following tools and libraries have been found and configured:
 * Git
 * MPI
 * OpenMP
 * Threads
 * TPLCUDA
 * TPLLIBDL

-- <<< Build configuration >>>
   Operating System: Linux Ubuntu 20.04
   Build type:       RelWithDebInfo
   Install path:     /home/akohlmey/.local
   Generator:        Unix Makefiles using /usr/bin/make
-- Enabled packages: KOKKOS;KSPACE;MANYBODY;MOLECULE;RIGID
-- <<< Compilers and Flags: >>>
-- C++ Compiler:     /usr/local/lammps/lib/kokkos/bin/nvcc_wrapper
      Type:          GNU
      Version:       9.3.0
      C++ Flags:     -O2 -g -DNDEBUG
      Defines:       LAMMPS_SMALLBIG;LAMMPS_MEMALIGN=64;LAMMPS_OMP_COMPAT=4;LAMMPS_GZIP;FFT_KISS;$<BUILD_INTERFACE:LMP_KOKKOS>
-- <<< Linker flags: >>>
-- Executable name:  lmp
-- Static library flags:    
-- <<< MPI flags >>>
-- MPI_defines:      MPICH_SKIP_MPICXX;OMPI_SKIP_MPICXX;_MPICC_H
-- MPI includes:     /opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/include/openmpi;/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/include/openmpi/opal/mca/hwloc/hwloc201/hwloc/include;/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/include/openmpi/opal/mca/event/libevent2022/libevent;/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/include/openmpi/opal/mca/event/libevent2022/libevent/include;/opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/include
-- MPI libraries:    /opt/nvidia/hpc_sdk/Linux_x86_64/22.1/comm_libs/hpcx/hpcx-2.10.beta/ompi/lib/libmpi.so;
-- Kokkos Arch: 
-- <<< FFT settings >>>
-- Primary FFT lib:  KISS
-- Using double precision FFTs
-- Using non-threaded FFTs
-- Kokkos FFT: KISS
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/local/lammps/build-nvidia
Singularity> cmake --build build-nvidia          
Scanning dependencies of target kokkoscore
[  0%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o
[  0%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Error.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_ExecPolicy.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostBarrier.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostSpace.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostSpace_deepcopy.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostThreadTeam.cpp.o
[  1%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_MemoryPool.cpp.o
[  2%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_MemorySpace.cpp.o
[  2%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_NumericTraits.cpp.o
[  2%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Profiling.cpp.o
[  2%] Building CXX object lib/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Serial.cpp.o

ntiwari · February 17, 2022, 4:16pm

Hi Axel,

Thanks for your reply. The problem was that I was using an out-of-date NV HPC image. Once I updated, everything worked fine. Both regular make and cmake.

Here is a link to the Dockerfile I wrote.

Nick

ottocome · October 12, 2023, 12:17pm

This seems to be happening again:

echo $CXX
/opt/software/x86/nvhpc/nvhpc_23.5/Linux_x86_64/23.5/compilers/bin/nvc++

from src:

make yes-kokkos 
make kokkos_cuda_mpi
../../lib/kokkos/Makefile.kokkos:1118: *** Makefile.kokkos: CUDA is enabled but the compiler is neither NVCC nor Clang (got version string nvc++ 23.5-0 64-bit target on x86-64 Linux -tp zen3 NVIDIA Compilers and Tools Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.)

It seems nvc++ is not recognized as a possible compiler.

stamoor · October 12, 2023, 3:08pm

Need more info: LAMMPS version, CUDA version, etc.

ottocome · October 12, 2023, 3:46pm

NVHPC 23.5, CUDA 12, LAMMPS version 28Mar2023

Here is where it seems to fail:

ifeq ($(KOKKOS_INTERNAL_USE_CUDA_ARCH), 1)
  ifeq ($(KOKKOS_INTERNAL_COMPILER_NVCC), 1)
    KOKKOS_INTERNAL_CUDA_ARCH_FLAG=-arch
  else ifeq ($(KOKKOS_INTERNAL_COMPILER_CLANG), 1)
                KOKKOS_INTERNAL_CUDA_ARCH_FLAG=--cuda-gpu-arch
                KOKKOS_CXXFLAGS += -x cuda
  else
    $(error Makefile.kokkos: CUDA is enabled but the compiler is neither NVCC nor Clang (got version string $(KOKKOS_CXX_VERSION)) )
  endif
  KOKKOS_INTERNAL_USE_CUDA_ARCH = 1
endif

stamoor · October 12, 2023, 4:25pm

Can you try updating to Kokkos 4 which was released in LAMMPS in August: https://github.com/lammps/lammps/pull/3670