I am trying to build LAMMPS with Kokkos package to run on Intel PVC GPUs. From what I understand, I need to select the SYCL backend and I need to use Intel compilers (GNU compilers seem to lack support for SYCL). I picked the cmake build strategy because I have some experience with it. I am having difficulty getting cmake to play nicely with the intel compilers and I can’t find any relevant guidance. Has anyone here done this before? Is anyone here interested in helping me?
Richard Lawrence
Texas A&M University
Please provide more details on this.
Can you compile LAMMPS with the Intel compilers and withOUT KOKKOS?
What is your CMake command line for configuring LAMMPS?
I have not heard or seen anything. I’ve had success using Intel GPUs with OpenCL and the GPU package.
We are running LAMMPS on ALCF Aurora (Intel PVC GPUs) using the SYCL backend with Kokkos, in general it works quite well. However, I’ve only used the GNU Makefile build system to compile LAMMPS for Intel GPUs with SYCL. What errors are you getting specifically?
Howdy Alex,
I have compiled LAMMPS with intel compilers and the GPU package, but it was using make strategy (not cmake) and instructions were provided by an intel representative. I don’t know how to translate that success to building with kokkos, since I am not as familiar with the make strategy.
Cmake has trouble locating header files such as omp.h provided by the intel compilers and has trouble correctly identifying which intel compiler executable is for what purpose (icpc, icx, dpcpp). To be fair, intel is regularly renaming and relocating things so this kind of difficulty is expected.
With cmake, I am doing it like this (roughly)
cmake -DKokkos_ARCH_INTEL_PVC=yes -C ../cmake/presets/<preset> ...
where I picked the kokkos-sycl preset the oneapi preset and the basic preset
Howdy Stan,
Any chance you could share your notes with me? I wouldn’t mind trying the makefile build system.
Richard
This was just reported today as having a problem with recent intel compiler and CMake versions. Please try the replacement preset files as attached to the bug report issue.
On Aurora it is as simple as:
cd lammps/src
make yes-kokkos
make -j64 aurora_kokkos
This uses lammps/src/MAKE/MACHINES/Makefile.aurora_kokkos
Howdy Alex,
Thank you for the tip. That cmake preset was very helpful. The build seems to complete Update: build not complete
Another problem that might just be a local site problem is that the executable clang++ wasn’t found so I had to specify the path to it in CMakeCache.txt. Intel put it in bin-llvm for some reason. I suspect that I just need to modify the compiler module to add that to the PATH variable.
Richard Lawrence
But you didn’t specify the gpu arch or the backend? Is the makefile supposed to automatically detect those?
It is specified inside the Makefile, see https://github.com/lammps/lammps/blob/35214120ad350b7a7fb6e929f6abf467e21fcf8d/src/MAKE/MACHINES/Makefile.aurora_kokkos#L22-L23
Sorry Alex, but I am still having problems. When I use kokkos-sycl.cmake, it sets the compiler to clang++ which causes cmake to erase what it knows about openmp and then it cannot find openmp which it just knew about moments before. I am able to make some progress by disabling openmp, but I love it.
Well even without openmp, cmake --build . doesn’t compile correctly.
../include/sycl/types.hpp:2355:17: error: ambiguous partial specializations of
'is_device_copyable<const Kokkos::Experimental::Impl::SYCLFunctionWrapper<Kokkos::Impl::BitsetCount<Kokkos::Bitset<>>, Kokkos::Experimental::Impl::SYCLInternal::USMObjectMem<sycl::usm::alloc::host>>>'
The aurora makefile doesn’t work for me. It doesn’t know that my g++ executable isn’t part of my intel compilers and doesn’t support sycl.
First of all, make sure you are using the very latest LAMMPS feature release 21 Nov 2023 (or use the “release” branch from the git repository). If we have to make adjustments to the source code, you may even need to check out the “develop” branch or create a local feature branch into which you need to collect some commits with git cherry-pick.
Second, make sure you are using the most recent CMake module.
Don’t worry about OpenMP. Its use for KOKKOS with GPU acceleration is rather limited.
The the kokkos-sycl.cmake file is almost a year old and likely needs some updating since I have heard stories that SYCL support in KOKKOS is a bit of a moving target and was a bit experimental before Kokkos 4 (the stable release ships with Kokkos 3.x). Based on the settings for the GNU make build system, I would try with an adapted preset file:
kokkos-sycl-v1.cmake (1.0 KB)
There should be no need to use the “oneapi.cmake” preset file. It is primarily meant for compiling with the LLVM based intel compiler with the INTEL package or without accelerators in the presence of other compilers (CMake looks for “cc” and “c++” by default). When using KOKKOS and a sufficiently modern CMake all required settings should be autodetected and enabled by the Kokkos CMake scripting.
Howdy Alex,
At the direction of the Kokkos developers, I switch to a later release of LAMMPS (feature 21Nov) that bundles a more recent release of Kokkos (4.1). Building seems to be going well; at least, configuration was much easier. But now I have this error. Do you recognize it?
In file included from lammps/src/KOKKOS/fft3d_kokkos.cpp:18:
In file included from lammps/src/KOKKOS/fft3d_kokkos.h:19:
In file included from lammps/src/KOKKOS/remap_kokkos.h:20:
In file included from lammps/src/KOKKOS/fftdata_kokkos.h:220:
lammps/src/KOKKOS/kissfft_kokkos.h:433:15: error: SYCL kernel cannot call a recursive function
kf_work(d_Fout, d_f, fstride*p, in_stride, d_factors, st, Fout_count, f_count, factors_count);
^
lammps/src/KOKKOS/kissfft_kokkos.h:410:3: note: function implemented using recursion declared here
KOKKOS_INLINE_FUNCTION
^
lammps/lib/kokkos/core/src/Kokkos_Macros.hpp:362:32: note: expanded from macro 'KOKKOS_INLINE_FUNCTION'
#define KOKKOS_INLINE_FUNCTION KOKKOS_IMPL_INLINE_FUNCTION
^
lammps/lib/kokkos/core/src/Kokkos_Macros.hpp:319:37: note: expanded from macro 'KOKKOS_IMPL_INLINE_FUNCTION'
#define KOKKOS_IMPL_INLINE_FUNCTION inline
Also, is it worth it to submit a bug report for this?
Probably I should just try a different FFT build. That would avoid this problem.
This is an issue for @stamoor to look at.
error: SYCL kernel cannot call a recursive function
I’ve seen this too. This is a limitation of SYCL, not much we can do to make KISS FFT work with SYCL. In theory we should be able to use MKL for the 1D FFTs but some of the compiler macros may need to be tweaked in the code. You are at the bleeding edge here… Do you absolutely need to compile with KSpace enabled?
Yes, I need kspace. I want to run the rhodopsin benchmark which uses the pppm solver.
OK this will probably require surgery into the code to allow MKL to run on the Intel GPUs. Let me consult with my ANL collaborators and we’ll try to get it working.