Part of our compute infrastructure (for test runs mostly) is heterogeneous in terms of Nvidia GPU generations (P100, V100, A100) and we are looking for ways to have LAMMPS support this, even if it means a slight drop in performance.
# preset that enables KOKKOS and selects CUDA compilation with OpenMP
# enabled as well. This preselects CC 5.0 as default GPU arch, since
# that is compatible with all higher CC, but not the default CC 3.5
[ ... ]
set(Kokkos_ARCH_PASCAL60 ON CACHE BOOL "" FORCE)
However, when running things built for V100 on an A100 GPU, we see the error
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture
which is also what the LAMMPS + Kokkos docs say would happen (no support across compute capability major versions).
Currently we build separate LAMMPS extension modules for different architectures to handle that, and just want to make sure we’re doing things by the book, so my questions would be:
The kokkos-cuda.cmake preset seems to suggest that support for multiple architectures exists, but the LAMMPS + Kokkos docs and our tests imply the opposite. Which one is correct? We want to make sure that we see the error because there is no cross-architecture support rather than a compile mistake on our end.
As a solution, is it possible to build a “fat binary” by compiling for multiple selected architectures? We already found that something like
doesn’t work (cmake says “use exactly one architecture”).
Besides using Kokkos to get GPU support, there is also LAMMPS’ own GPU package. Its docs say that it supports “all major GPU architectures supported by this [CUDA] toolkit. Thus the GPU_ARCH setting is merely an optimization, to have code for the preferred GPU architecture directly included rather than having to wait for the JIT compiler of the CUDA driver to translate it.” – That sounds exactly like the behavior we are looking for, so should we use that instead?
Technically, yes, but it won’t work with KOKKOS. I have managed to compile such an executable, but the behavior will be that it appears as if LAMMPS is stuck while the JIT compiler will recompile all GPU kernels, but then it will stop with an error regardless.This is a design decision of the Kokkos developers and thus limitation of the Kokkos library itself. The information that I got when asking about it was that it may work for minor architecture differences, but it is not supposed to work for major architecture differences, because of different architecture dependent code paths.
When compiling with CMake, the GPU package will always build “fat” binaries for CUDA and should thus create binaries that will run on different Nvidia GPU architectures. Also, you can compile it in OpenCL mode and then the same executable will even work on Intel or AMD GPUs in addition to Nvidia GPUs. It even has some internal heuristics to optimize the kernels for individual GPU architectures.
Whether KOKKOS or GPU package is the better choice depends on your application. In some cases only KOKKOS provides GPU support, in others only the GPU package. Also, there are performance differences and KOKKOS currently only support full double precision.
A pragmatic solution would be to compile multiple binaries under different names and then create a wrapper shell script that detects which GPU a machine has (e.g. by parsing the output of lspci -mm | grep VGA) and then selects which executable name to use based on that information and then just do an exec ${exename} "$@".
I talked to the Kokkos developers on their Slack channel and it appears that it should work, but the call is failing in CUDA, not Kokkos. They suggested that the necessary PTX may not be embedded in the binary, and tweaking the compile flags may help.
@elcorto Can you please post your compiler and linker flags here?
For reference, here is the check in Kokkos:
// Query what compute capability architecture a kernel executes:
Impl::CudaInternal::m_cudaArch = Impl::cuda_kernel_arch(cuda_device_id);
if (Impl::CudaInternal::m_cudaArch == 0) {
Kokkos::abort(
"Kokkos::Cuda::initialize ERROR: likely mismatch of architecture\n");
}
int compiled_major = Impl::CudaInternal::m_cudaArch / 100;
int compiled_minor = (Impl::CudaInternal::m_cudaArch % 100) / 10;
if ((compiled_major > cudaProp.major) ||
((compiled_major == cudaProp.major) &&
(compiled_minor > cudaProp.minor))) {
std::stringstream ss;
ss << "Kokkos::Cuda::initialize ERROR: running kernels compiled for "
"compute capability "
<< compiled_major << "." << compiled_minor
<< " on device with compute capability " << cudaProp.major << "."
<< cudaProp.minor << " is not supported by CUDA!\n";
std::string msg = ss.str();
Kokkos::abort(msg.c_str());
}
if (Kokkos::show_warnings() &&
(compiled_major != cudaProp.major || compiled_minor != cudaProp.minor)) {
std::cerr << "Kokkos::Cuda::initialize WARNING: running kernels compiled "
"for compute capability "
<< compiled_major << "." << compiled_minor
<< " on device with compute capability " << cudaProp.major << "."
<< cudaProp.minor
<< " , this will likely reduce potential performance."
<< std::endl;
}
That’s good to know, thanks. Unfortunately, the way our application (which is this) interfaces with LAMMPS requires GPU support to be built with Kokkos. I have to double-check with the other devs, though.
True, this is the first solution we had in mind, but wanted to be sure that this was the best option, hence the question here.
Thank you very much for your efforts! Christian Robert Trott of the Kokkos team was so kind to share his response via email, which I’ll post below:
I can gather and share compile output as suggested above if needed. In the meantime, here is how we build LAMMPS.
To provide more context, we actually do this as part of a docker image build, which we convert to an apptainer/singularity image and run on GPU nodes via apptainer exec --nv, I’m not sure if that is known to cause problems in terms of forward compatibility.
cuda_version=12.4.1
lammps_version=patch_4Feb2025
lammps_gpu_arch=VOLTA70
lammps_cpu_arch=HSW
py_version=3.11
sys_install_path=/usr/local/lib/python${py_version}/dist-packages
spack install cuda@$cuda_version arch=x86_64
# The lammps linker wants libcuda.so.1 which is not exposed by spack's cuda
# package and also not by the equiv module cuda/12.4 on the cluster:
# find /trinity/shared/pkg/devel/cuda/12.4/ -name "libcuda.so.1"
# There is only stubs/libcuda.so . So create a link here to make the linker
# happy. When running this image (after apptainer convert) with apptainer exec
# --nv, a libcuda.so.1 is mapped into the container.
spack load cuda@$cuda_version
cd $(spack location -i cuda@$cuda_version)/lib64
ln -s stubs/libcuda.so libcuda.so.1
cd /opt/soft/git
git clone --branch=$lammps_version --depth=1 https://github.com/lammps/lammps.git
export LD_LIBRARY_PATH=$CUDA_HOME/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
cd /opt/soft/git/lammps
mkdir -pv build
cd build
rm -rf *
cmake ../cmake \
-D PKG_KOKKOS=yes \
-D BUILD_MPI=yes \
-D PKG_ML-SNAP=yes \
-D Kokkos_ENABLE_CUDA=yes \
-D Kokkos_ARCH_${lammps_cpu_arch}=yes \
-D Kokkos_ARCH_${lammps_gpu_arch}=yes \
-D CMAKE_CXX_COMPILER=$(pwd)/../lib/kokkos/bin/nvcc_wrapper \
-D BUILD_SHARED_LIBS=yes
cmake --build . --parallel=8
# Build Python extension
cd /opt/soft/git/lammps/python
python3 -m venv __tmp_env
. ./__tmp_env/bin/activate
python install.py -p lammps -l ../build/liblammps.so -v ../src/version.h
deactivate
cp -rv __tmp_env/lib/python${py_version}/site-packages/lammps* $sys_install_path/
Note that the CUDA we use during build is not part of the final image. At runtime, we make the host CUDA available to the LAMMPS Python extension inside the container such that the two CUDA shared libs which are needed at runtime are present. So roughly
We made progress and (potentially) solved the issue.
Our machines have a fixed CUDA driver version (CUDA toolkit 12.1, libcuda.so.1 → libcuda.so.530.30.02. What we did wrong was to compile LAMMPS using a newer toolkit version (12.4). Using that, things do run, but only on the compiled-for target GPU architecture. On others we see the “mismatch of architecture” error.
If we compile LAMMPS with CUDA toolkit version 12.1 (same as host CUDA driver’s toolkit version), then we seem to have forward compatibility. For example, if we compile for PASCAL60, we can run on V100 (VOLTA70) or A100 (AMPERE80). Results are the same (up to numerical noise) compared “native” (e.g. VOLTA70 on V100).
There is one remaining question: Running in what we think is the forward compatibility mode, when LAMMPS is called for the first time, we notice a wait of maybe 1 min, before calculations continue. We assumed that this is where the host CUDA JIT is recompiling kernels for the current GPU architecture. However, this message from Kokkos (PASCAL60 on A100 example)
Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 6.0 on device with compute capability 8.0 , this will likely reduce potential performance.
seems to suggest that this not the case and instead the compute capability 6.0
(PASCAL60) machine code is used. Is that correct?
This is backward compatibility. The newer architecture is backward compatible with executable code for a previous architecture.
That is rather short, I’ve seen longer way back when I was experimenting.
Yes.
Yes and no. An older compute capability is missing features that a newer architecture has. Thus at (LAMMPS) compile time, the code has to be compiled to be compatible with the lowest common denominator. That makes optimizations that can only work on newer architectures unavailable. So there is a potential slowdown compared to a kernel compiled for that specific architecture.
Thank you very much, that answers all our questions.
In terms of performance, we’re OK with that for the target use case, which is small scale runs on whatever hardware happens to be available. For large scale production, we opt for homogeneous hardware and a LAMMPS compiled for that.