LAMMPS segfaults when built with Kokkos and CUDA support

LAMMPS LAMMPS Mailing List Mirror
1

If I remove -shared I get this:

# mpicxx \-std=c\+\+11 \-D\_\_CUDA\_ARCH\_\_=350 \-E \-x c\+\+ \-DCUDA\_DOUBLE\_MATH\_FUNCTIONS \-\-std=c\+\+11 \-fopenmp \-O3 \-D\_\_CUDA\_PREC\_DIV \-D\_\_CUDA\_PREC\_SQRT \-I"\.\./\.\./lib/kokkos/core/src" \-I"\.\./\.\./lib/kokkos/containers/src" \-I"\.\./\.\./lib/kokkos/algorithms/src" \-I"\.\./\.\./lib/kokkos/linalg/src" \-I"\.\./" "\-I/opt/cuda\-7\.0/bin/\.\.//include" \-m64 \-g \-gdwarf\-2 "/tmp/tmpxft\_0001ae4d\_00000000\-4\_kokkos\_depend\.cudafe1\.cpp" > "/tmp/tmpxft\_0001ae4d\_00000000\-14\_kokkos\_depend\.ii" \# mpicxx -std=c++11 -c -x c++ --std=c++11 -fopenmp -O3
-I"../../lib/kokkos/core/src" -I"../../lib/kokkos/containers/src"
-I"../../lib/kokkos/algorithms/src" -I"../../lib/kokkos/linalg/src"
-I"../" "-I/opt/cuda-7.0/bin/..//include" -fpreprocessed -m64 -g
-gdwarf-2 -o "kokkos_depend.o"
"/tmp/tmpxft_0001ae4d_00000000-14_kokkos_depend.ii"
/usr/local/bin/ld: -f may not be used without -shared
collect2: error: ld returned 1 exit status

which is why I tried adding it.

which doesn't make sense. so the problem is something else. so you
have to ask yourself, why is ld complaining about a -f flag? or rather
what other flag, starts with -f. this requires a little digging, but
as it turns out, the problem is that when you compile with mpicxx, the
KOKKOS settings will divert *all* code to be compiled with nvcc and
then have nvcc pass the C++ code to the underlying gcc. however, since
nvcc is derived from the open64 compiler, it requires a different
syntax for compiler flags than gcc. the ultimate culprit is that
KOKKOS by default compiles with OpenMP enabled (together with CUDA)
and that puts the -fopenmp flag into the command line. for individual
file compilations, this is protected by -Xcompiler, but for the linker
it is not. hence the error message.\

thus the proper solution is the following change.

diff --git a/lib/kokkos/Makefile.lammps b/lib/kokkos/Makefile.lammps
index dd2af5c..00b55f4 100755
--- a/lib/kokkos/Makefile.lammps
+++ b/lib/kokkos/Makefile.lammps
@@ -111,11 +111,12 @@ SRC_KOKKOS += \(wildcard (KOKKOS_PATH)/core/src/OpenMP/*.cpp)
KOKKOS_HEADERS += \(wildcard (KOKKOS_PATH)/core/src/OpenMP/*.hpp)
ifeq ($(CUDA), yes)
KOKKOS_INC += -Xcompiler -fopenmp
+KOKKOS_LINK += -Xcompiler -fopenmp
else
KOKKOS_INC += -fopenmp
-endif
KOKKOS_LINK += -fopenmp
endif
+endif

ifeq (\(HWLOC\),yes\) KOKKOS\_INC \+= \-DKOKKOS\_HAVE\_HWLOC \-I(HWLOC_PATH)/include

or to use OMP=no when compiling with CUDA=yes

axel.

2

It compiles and runs fine now when built without OMP. If I make the
change you suggested, with OMP, I get a new error:

../pair_comb_omp.cpp: In member function ‘virtual double
LAMMPS_NS::PairCombOMP::yasu_char(double*, int&)’:
../pair_comb_omp.cpp:520:1: error: expected primary-expression before ‘}’ token
#endif
^

I can't see any obvious errors around that line. If I comment out the
#if and #pragma lines (even though they must be important) this error
goes away, and the previous error about -shared reappears.

3

It compiles and runs fine now when built without OMP. If I make the
change you suggested, with OMP, I get a new error:

../pair_comb_omp.cpp: In member function ‘virtual double
LAMMPS_NS::PairCombOMP::yasu_char(double*, int&)’:
../pair_comb_omp.cpp:520:1: error: expected primary-expression before ‘}’ token
#endif
^

I can't see any obvious errors around that line. If I comment out the

this is not very smart to have known to be broken code in your
executable, particularly when you obviously don't know what that code
is about. why don't you just delete the comb/omp pair style sources or
uninstall the USER-OMP package altogether? after all, you are not
interested in them.

the culprit is nvcc. it is not compatible with all the OpenMP
directives in the USER-OMP package (in particular "omp atomic" ).

#if and #pragma lines (even though they must be important) this error
goes away, and the previous error about -shared reappears.

i have explained in detail why the error appears. i also explained why
you need to have suitable experience as a developer to compile/use the
KOKKOS package.
i have no time to make up for your lack of experience in this regard
and babysit you through the procedure. i have been able to compile and
run LAMMPS with KOKKOS using CUDA with and without OpenMP support and
can confirm that outside the small makefile mishap, there is nothing
wrong in KOKKOS. i recommend that you stop wasting your and our time
and stop those - rather pointless it seems - experiments with KOKKOS.

axel.