[lammps-users] Issues with building LAMMPS w/ KOKKOS & KSPACE packages

Dear LAMMPS users and developers,

I am currently trying to build LAMMPS with the KSPACE and KOKKOS packages on a cluster available to my research group:

  • Intel Xeon E5-2630 (lscpu)
  • 2 x NVIDIA GeForce GTX 1080 Ti’s + 1 x NVIDIA TITAN Xp (nvidia-smi)
  • Ubuntu 16.04.4 LTS, kernel 4.4.0-143-generic (uname -r)
  • NVIDIA driver 384.183 (nvidia-smi)
  • CUDA 9.0 (nvcc --version)
  • gcc 5.4.0 (gcc --version)
  • cmake 3.19.2

I have the following CMake presets in …/lammps_all.cmake (from the build directory):

get_filename_component(NVCC_WRAPPER {CMAKE_CURRENT_SOURCE_DIR}/../lib/kokkos/bin/nvcc_wrapper ABSOLUTE) set(BIN2C /usr/local/cuda-9.0/bin/bin2c CACHE FILEPATH "" FORCE) set(BUILD_MPI ON CACHE BOOL "" FORCE) set(BUILD_OMP ON CACHE BOOL "" FORCE) set(CMAKE_CXX_COMPILER {NVCC_WRAPPER} CACHE FILEPATH “” FORCE)
set(PKG_KOKKOS ON CACHE BOOL “” FORCE)
set(PKG_KSPACE ON CACHE BOOL “” FORCE)
set(Kokkos_ARCH_PASCAL61 ON CACHE BOOL “” FORCE)
set(Kokkos_ARCH_HSW ON CACHE BOOL “” FORCE)
set(Kokkos_ENABLE_CUDA ON CACHE BOOL “” FORCE)
set(Kokkos_ENABLE_DEBUG ON CACHE BOOL “” FORCE)
set(Kokkos_ENABLE_LIBRT ON CACHE BOOL “” FORCE)
set(Kokkos_ENABLE_OPENMP ON CACHE BOOL “” FORCE)
set(Kokkos_ENABLE_SERIAL ON CACHE BOOL " FORCE)

I call /home/bye/cmake-3.19.2/bin/cmake -C …/lammps_all.cmake …/cmake and this is the output:

loading initial cache file …/lammps_all.cmake
– The CXX compiler identification is GNU 5.4.0
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Check for working CXX compiler: /home/bye/lammps/lib/kokkos/bin/nvcc_wrapper - skipped
– Detecting CXX compile features
– Detecting CXX compile features - done
– Found Git: /usr/bin/git (found version “2.7.4”)
– Running check for auto-generated files from make-based build system
– Found MPI_CXX: /usr/lib/openmpi/lib/libmpi_cxx.so (found version “3.0”)
– Found MPI: TRUE (found version “3.0”)
– Looking for C++ include omp.h
– Looking for C++ include omp.h - found
– Found OpenMP_CXX: -fopenmp (found version “4.0”)
– Found OpenMP: TRUE (found version “4.0”)
– Found JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (found version “80”)
– Found GZIP: /bin/gzip
– Found FFMPEG: /usr/bin/ffmpeg
– Found PkgConfig: /usr/bin/pkg-config (found version “0.29.1”)
– Checking for module ‘fftw3’
– Found fftw3, version 3.3.4
– Found FFTW3: /usr/lib/x86_64-linux-gnu/libfftw3.so
– Looking for C++ include cmath
– Looking for C++ include cmath - found
– KOKKOS: Enabling CUDA LAMBDA function support
– Setting default Kokkos CXX standard to 11
– Setting policy CMP0074 to use _ROOT variables
– The project name is: Kokkos
– Compiler Version: 9.0.176
– Using -std=c++11 for C++11 standard as feature
– Execution Spaces:
– Device Parallel: CUDA
– Host Parallel: OPENMP
– Host Serial: SERIAL

– Architectures:
– HSW
– PASCAL61
– Found CUDAToolkit: /usr/local/cuda-9.0/include (found version “9.0.176”)
– Looking for C++ include pthread.h
– Looking for C++ include pthread.h - found
– Performing Test CMAKE_HAVE_LIBC_PTHREAD
– Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
– Found Threads: TRUE
– Found TPLCUDA: TRUE
– Found TPLLIBRT: /usr/lib/x86_64-linux-gnu/librt.so
– Found TPLLIBDL: /usr/lib/x86_64-linux-gnu/libdl.so
– Generating style headers…
– Generating package headers…
– Generating lmpinstalledpkgs.h…
– The following tools and libraries have been found and configured:

  • Git
  • MPI
  • OpenMP
  • JPEG
  • PkgConfig
  • FFTW3
  • Threads
  • CUDAToolkit
  • TPLCUDA
  • TPLLIBRT
  • TPLLIBDL

– <<< Build configuration >>>
Operating System: Linux
Build type: RelWithDebInfo
Install path: /home/bye/.local
Generator: Unix Makefiles using /usr/bin/make
– Enabled packages: KOKKOS;KSPACE
– <<< Compilers and Flags: >>>
– C++ Compiler: /home/bye/lammps/lib/kokkos/bin/nvcc_wrapper
Type: GNU
Version: 5.4.0
C++ Flags: -O2 -g -DNDEBUG
Defines: LAMMPS_SMALLBIG;LAMMPS_MEMALIGN=64;LAMMPS_OMP_COMPAT=3;LAMMPS_JPEG;LAMMPS_GZIP;LAMMPS_FFMPEG;FFT_FFTW3;FFT_FFTW_THREADS;LMP_KOKKOS;FFT_CUFFT
– <<< Linker flags: >>>
– Executable name: lmp
– Static library flags:
– <<< MPI flags >>>
– MPI_defines:
– MPI includes: /usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent;/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/include;/usr/lib/openmpi/include;/usr/lib/openmpi/include/openmpi
– MPI libraries: /usr/lib/openmpi/lib/libmpi_cxx.so;/usr/lib/openmpi/lib/libmpi.so;
– Kokkos Arch:
– <<< FFT settings >>>
– Primary FFT lib: FFTW3
– Using double precision FFTs
– Using threaded FFTs
– Kokkos FFT: cuFFT
– Configuring done
– Generating done
– Build files have been written to: /home/bye/lammps/build

I noticed that the check for a working C++ compiler was skipped. Then, I execute /home/bye/cmake-3.19.2/bin/cmake --build . --target install – -j 20 to install locally. Everything goes well until the LAMMPS dependencies start to get built (“Scanning dependencies of lammps”). I get a long string of errors starting with:

/usr/lib/gcc/x86_64-linux-gnu/5/include/stddef.h(436): error: identifier “nullptr” is undefined
/usr/lib/gcc/x86_64-linux-gnu/5/include/stddef.h(436): error: expected a “;”
/usr/lib/gcc/x86_64-linux-gnu/5/include/stddef.h(436): error: identifier “nullptr” is undefined
/usr/lib/gcc/x86_64-linux-gnu/5/include/stddef.h(436): error: expected a “;”
/usr/include/c++/5/bits/c++config.h(200): error: expected a “;”
/usr/include/c++/5/exception(63): error: expected a “;”
/usr/include/c++/5/exception(68): error: expected a “;”
/usr/include/c++/5/exception(76): error: expected a “;”
/usr/include/c++/5/exception(83): error: expected a “;”
/usr/include/c++/5/exception(93): error: expected a “{”
/usr/include/c++/5/bits/exception_ptr.h(64): error: function “std::current_exception” returns incomplete type “std::__exception_ptr::exception_ptr”
/usr/include/c++/5/bits/exception_ptr.h(64): error: expected a “{”
/usr/include/c++/5/bits/exception_ptr.h(79): error: expected a “;”
/usr/include/c++/5/bits/exception_ptr.h(81): error: expected a “;”
/usr/include/c++/5/bits/exception_ptr.h(82): error: expected a “;”
/usr/include/c++/5/bits/exception_ptr.h(84): error: expected a “;”
/usr/include/c++/5/bits/exception_ptr.h(86): error: declaration is incompatible with previous “std::current_exception”
(64): here
/usr/include/c++/5/bits/exception_ptr.h(86): error: use of a local type to declare a function

Quick online searches tell me this has to do with gcc not abiding by the C++11 standard. However, in the printout above, it states “Setting default Kokkos CXX standard to 11”. I was able to compile LAMMPS with the GPU and KSPACE packages normally on the same machine. How do I fix this?

Thank you,

Benjamin

you didn’t say which specific LAMMPS version you are using. and using ubuntu 16.04 is pushing the limits.
I would suspect the problem originates in nvcc_wrapper or CMake not being able to tell how to enable C++11 for it.
you may try to add -std=c++11 or --std=c++11 to CMAKE_TUNE_FLAGS

axel.

Could also try the traditional Makefile-based build system as a backup. This all should work–Kokkos has required the c++11 standard for years, and is switching to c++14 in the next update.

Stan

Dear Axel and Stan,

Thank you both for the quick response!

I forgot to mention that I am using the latest stable build of LAMMPS from October 29, 2020.

The errors I encountered originally occurred in the step

[ 14%] Building CXX object CMakeFiles/lammps.dir/home/bye/lammps/src/bond_deprecated.cpp.o

When I added “-D CMAKE_TUNE_FLAGS=”-std=c++11", I get a different error (truncated):

[ 14%] Building CXX object CMakeFiles/lammps.dir/home/bye/lammps/src/body.cpp.o
/home/bye/lammps/src/fmt/core.h(331): warning: integer conversion resulted in a change of sign
detected during:
instantiation of “__nv_bool fmt::v7_lmp::detail::is_unicode() [with Char=char]”
(1865): here
instantiation of “void fmt::v7_lmp::print(FILE *, const S &, Args &&…) [with S=char [44], Args=<char (&)[16], char *&>, Char=char]”
/home/bye/lammps/src/bond.cpp(284): here
/home/bye/lammps/src/fmt/core.h(331): warning: integer conversion resulted in a change of sign
detected during:
instantiation of “__nv_bool fmt::v7_lmp::detail::is_unicode() [with Char=char]”
(1865): here
instantiation of “void fmt::v7_lmp::print(FILE *, const S &, Args &&…) [with S=char [16], Args=<int64_t, double &, double &, double &, double &>, Char=char]”
/home/bye/lammps/src/atom_vec_line.cpp(504): here
/home/bye/lammps/src/fmt/core.h(331): warning: integer conversion resulted in a change of sign
detected during:
instantiation of “__nv_bool fmt::v7_lmp::detail::is_unicode() [with Char=char]”
(1865): here
instantiation of “void fmt::v7_lmp::print(FILE *, const S &, Args &&…) [with S=char [19], Args=<LAMMPS_NS::bigint &>, Char=char]”
/home/bye/lammps/src/balance.cpp(1167): here

Per Stan’s advice, I also tried using traditional make. I first tried “make kokkos_cuda_mpi” in /lammps/src but make was not able to find nvcc_wrapper. I fixed the issue by providing the absolute path to the Kokkos library. Then, I get the following error:

mpicxx -g -O3 -DLAMMPS_GZIP -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -DFFT_CUFFT -c …/main.cpp
In file included from …/pointers.h:24:0,
from …/input.h:17,
from …/main.cpp:15:
…/lmptype.h:34:2: error: error LAMMPS requires a C++11 (or later) compliant compiler. Enable C++11 compatibility or upgrade the compiler.
error LAMMPS requires a C++11 (or later) compliant compiler. Enable C++11 compatibility or upgrade the compiler.
^
In file included from /usr/include/c++/5/cstdint:35:0,
from …/lmptype.h:47,
from …/pointers.h:24,
from …/input.h:17,
from …/main.cpp:15:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
error This file requires compiler and library support
^
Makefile:114: recipe for target ‘main.o’ failed
make[1]: *** [main.o] Error 1
make[1]: Leaving directory ‘/home/bye/lammps/src/Obj_kokkos_cuda_mpi’
Makefile:195: recipe for target ‘kokkos_cuda_mpi’ failed
make: *** [kokkos_cuda_mpi] Error 2

Seeing that the issue had to do with C++11 compatibility, I added “-std=c++11” to CCFLAGS in the Makefile (saved separately as Makefile.machine) and tried again. I then got this error:

mpicxx -g -O3 -std=c++11 -DLAMMPS_GZIP -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -DFFT_CUFFT -c …/main.cpp
…/fmt/core.h(1449): error: expression must have a constant value
…/fmt/core.h(1519): warning: variable “zero_ptr” was declared but never referenced
1 error detected in the compilation of “/tmp/tmpxft_000078f5_00000000-4_main.cpp4.ii”.
Makefile:114: recipe for target ‘main.o’ failed
make[1]: *** [main.o] Error 2
make[1]: Leaving directory ‘/home/bye/lammps/src/Obj_machine’
Makefile:195: recipe for target ‘machine’ failed
make: *** [machine] Error 2

This seems to be a separate issue from the C++11 compatibility but I’m not sure what the cause is. Is the node I’m using simply too outdated to compile LAMMPS with the Kokkos package?

Thank you,

Benjamin

The problem appears to be with nvcc from the CUDA 9.0 version. That one is apparently not sufficiently C++11 compliant for all of LAMMPS as we have it now in the latest stable version. the hosting gcc compiler is compliant.

I have on my desktop a whole bunch of CUDA toolkit versions and tested compiling the core LAMMPS code (no packages) with an ubuntu16.04 container and using nvcc_wrapper instead of g++ (with -std=c++11) and the oldest of my local CUDA versions that was capable of successfully compiling the core LAMMPS code was version 10.2.

So you have the following options:

  • try an older version of LAMMPS (not recommended)
  • try a newer CUDA version (simplest option and can be done without superuser privilege provided the installed CUDA driver is compatible)
  • have the cluster upgraded to a newer version of the OS (and its tools). Please keep in mind that the end of life for ubuntu16.04LTS is at April 2021, so that will have to be done rather soon anyway. according to comments of people benchmarking and improving KOKKOS code in LAMMPS for use with CUDA, there is a performance gain from using CUDA 11.x for KOKKOS.

axel.