How to build LAMMPS for GPU with Kokkos

Hi Axel and all,

I am trying to build to build LAMMPS with KOKKOS for the GPU.

The problem is, that I am not used OpenMPI, but rather SGI MPT, but I did use the

Makefile.kokkos_cuda_openmpi,

It is not clear to me what changes I need to make.

I am using gcc as the CC compiler. For the regular GPU build I used the Intel compiler for building the packages, using nvcc for the gpu library.

I went through the following steps for Kokkos

$ make clean-all

$ make no-all

$ make no-lib

$ make yes-manybody yes-molecule yes-replica yes-kspace yes-asphere yes-misc yes-user-omp

make yes-kokkos

make kokkos_cuda_openmpi

but the build fails because nvcc can’t be found by the mpicc wrapper.

Do you have a suggestion to work around this?

Gabriele

two comments.

  1. if you had a reasonably up-to-date version of LAMMPS, the make target would be kokkos_cuda_mpi
  2. the nvcc wrapper depends on finding nvcc in your path. if it cannot find it, you have not set up your cuda environment properly

there is no indication here that this is a LAMMPS issue.

axel.

Gabriele,

You can also use nvcc_wrapper as the compiler and linker and then manually specify the MPI libraries. I don’t have an SGI MPT machine to test with, but for Titan using Cray MPI we use something similar:

KOKKOS_ABSOLUTE_PATH = (shell cd (KOKKOS_PATH); pwd)

CC = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper

LINK = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper

MPI_INC = -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -I//opt/cray/mpt/7.3.1/gni/mpich-gnu/5.1/include

MPI_PATH =

MPI_LIB = -L/opt/cray/mpt/7.3.1/gni/mpich-gnu/5.1/lib -lmpich

Thanks, Stan. Yes, I do the same thing, but I get these issues with multiple defined’s

mpicxx -g -O3 -DLAMMPS_GZIP -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 -I./ -I…/…/lib/kokkos/core/src -I…/…/lib/kokkos/containers/src -I…/…/lib/kokkos/algorithms/src -I /cuda/9.0/include --std=c++11 -expt-extended-lambda -arch=sm_35 -c …/compute_temp_chunk.cpp

…/accelerator_kokkos.h(64): error: invalid redeclaration of type name “LAMMPS_NS::AtomKokkos”

…/atom_kokkos.h(22): here

…/accelerator_kokkos.h(73): error: invalid redeclaration of type name “LAMMPS_NS::CommKokkos”

…/comm_kokkos.h(22): here

…/accelerator_kokkos.h(86): error: invalid redeclaration of type name “LAMMPS_NS::DomainKokkos”

…/domain_kokkos.h(28): here

…/accelerator_kokkos.h(92): error: invalid redeclaration of type name “LAMMPS_NS::NeighborKokkos”

…/neighbor_kokkos.h(31): here

I am using this version of LAMMPS:

LAMMPS (20 Apr 2018)

Hmm, not sure. What compiler are you using? Other details about your system?

Thanks,

Stan

Stan,

I am using gcc +nvcc. I waded through the code in accelerator_kokkos.h

#ifdef LMP_KOKKOS

#include “kokkos.h”

#include “atom_kokkos.h”

#include “comm_kokkos.h”

#include “comm_tiled_kokkos.h”

#include “domain_kokkos.h”

#include “neighbor_kokkos.h”

#include “memory_kokkos.h”

#include “modify_kokkos.h”

#define LAMMPS_INLINE KOKKOS_INLINE_FUNCTION

#else

// dummy interface to KOKKOS

// needed for compiling when KOKKOS is not installed

#include “atom.h”

#include “comm_brick.h”

#include “comm_tiled.h”

#include “domain.h”

#include “neighbor.h”

#include “memory.h”

#include “modify.h”

you wrote that you are using “make kokkos_cuda_openmpi”. that would be a makefile for an old version of LAMMPS. if you are using a cloned git repository and had modified the file, it may linger around after an update. the correct command is “make kokkos_cuda_mpi”.

axel.

That was a typo.

Kokkos is bundled already with LAMMPS in /lib/kokkos, so it should already be available. What version of g++ are you using?

Stan