FW: [EXTERNAL] How to build LAMMPS for GPU with Kokkos

It looks like your compile line in missing “-DLMP_KOKKOS”, which is set automatically in /src/Makefile.package when you type “make yes-kokkos”. Can you check that?

Also it looks like you are still using mpicxx as the compiler instead of nvcc_wrapper.

Stan

The Makefile.kokkos_cuda_mpi will only work with OpenMPI or MPICH because a special environment variable is set:

export MPICH_CXX = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper

export OMPI_CXX = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper

If you could find a similar variable for SGI MPT then you could use mpicxx, otherwise you have to use nvcc_wrapper as the compiler/linker and explicitly set the MPI library in the makefile (i.e. can’t use mpicxx).

Stan

HEUREKA!

Thanks so much, Stan. For all you guidance.

Here are my comments:

  • The Makefile.kokkos_cuda_mpi will only work with OpenMPI or MPICH because a special environment variable is set:

  • export MPICH_CXX = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper

  • export OMPI_CXX = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper

I had added :

export MPICXX_CXX = /LAMMPS/lib/kokkos/bin/nvcc_wrapper

which is the setting to specify the compiler used by mpicxx.

I have verified that cxx does call the nvcc_wrapper. No problem here.

I have tried several versions of gcc (4.8 and 6).

  • It looks like your compile line in missing “-DLMP_KOKKOS”, which is set automatically in /src/Makefile.package when you type “make yes-kokkos”. Can you check that?

THAT WAS IT. I had an old Makefile.package sitting around (despite make clean) , it always got updated, but did not contain the critical PATH_INC setting to -DLMP_KOKKOS.

Removing the file lead to the correct setting.

I now do have a lmp executable for kokkos+GPU+MPT when building without user-omp. Thanks so much.

Is the user-omp necessary? I am just rebuilding with OpenMP and see if it works this time.

Many thanks again, Stan!

Great. Kokkos supports its own version of OpenMP, so you really can’t use Kokkos and user-omp at the same time, but you can install both into a single executable for convenience. For more info see the “Using OpenMP threading and CUDA together” section here: http://lammps.sandia.gov/doc/accelerate_kokkos.html.

We can add “MPICXX_CXX” to Makefile.kokkos_cuda_mpi in the future.

Stan

Thanks! Without the LAMMPS user group feedback I would not have been able to get this done. If there is anything I can do for your regarding testing of LAMMPS with HPE-MPT, I would be glad to be of service.