ML-IAP + KOKKOS + PYTHON Failed to call Kokkos::Cuda::finalize()

LAMMPS LAMMPS Development
1

Hi all,

I installed and built LAMMPS (28 Mar 2023 - Development) to run python examples of MLIAP and Kokkos (for example ~/mylammps/examples/mliap/mliap_pytorch_Ta06A_kokkos.py). When running this script, the execution of the NVE MD run goes smoothly but I noticed a Kokkos::Cuda error at the very end of the log:

“”"
…
…
Performance: 3.402 ns/day, 7.056 hours/ns, 78.740 timesteps/s, 10.079 katom-step/s
78.0% CPU use with 1 MPI tasks x 1 OpenMP threads

MPI task timing breakdown:
Section | min time | avg time | max time |%varavg| %total

Pair | 11.546 | 11.546 | 11.546 | 0.0 | 90.92
Neigh | 0 | 0 | 0 | 0.0 | 0.00
Comm | 0.028056 | 0.028056 | 0.028056 | 0.0 | 0.22
Output | 0.021323 | 0.021323 | 0.021323 | 0.0 | 0.17
Modify | 0.39519 | 0.39519 | 0.39519 | 0.0 | 3.11
Other | | 0.7091 | | | 5.58

Nlocal: 128 ave 128 max 128 min
Histogram: 1 0 0 0 0 0 0 0 0 0
Nghost: 727 ave 727 max 727 min
Histogram: 1 0 0 0 0 0 0 0 0 0
Neighs: 0 ave 0 max 0 min
Histogram: 1 0 0 0 0 0 0 0 0 0
FullNghs: 7424 ave 7424 max 7424 min
Histogram: 1 0 0 0 0 0 0 0 0 0

Total # of neighbors = 7424
Ave neighs/atom = 58
Neighbor list builds = 0
Dangerous builds = 0
Total wall time: 0:00:14
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
“”"

I thought it would be worth bringing it up on the forum and see if there was a solution. Considering the MD simulation runs, it is not an immediate issue for me, but I figured I would go with the “see an error, report an error” approach.

Extra information:
-I am building and installing LAMMPS in a miniconda environment.
-The only changes I made to the mliap_pytorch_Ta06A_kokkos.py file was to increase the simulation duration and I moved all the python imports together.
-My cmake commands were as follows:
cmake
-C …/cmake/presets/most.cmake
-D CMAKE_CXX_COMPILER=mpicxx
-D PKG_VORONOI=no
-D PKG_ML-IAP=yes
-D PKG_ML-SNAP=yes
-D PKG_PYTHON=yes
-D PKG_GPU=cuda
-D PKG_KOKKOS=yes
-D GPU_ARCH=sm_86
-D Kokkos_ARCH_AMPERE86=yes
-D Kokkos_ENABLE_CUDA=yes
-D BUILD_SHARED_LIBS=ON
-D MLIAP_ENABLE_PYTHON=yes
…/cmake

  • nvcc --version returns this:
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2022 NVIDIA Corporation
    Built on Tue_May__3_18:49:52_PDT_2022
    Cuda compilation tools, release 11.7, V11.7.64
    Build cuda_11.7.r11.7/compiler.31294372_0

-The GPUs on my machine are NVIDIA RTX A6000s

Please let me know if any additional information is needed. Thanks!

2

You should be able to avoid the warning by adding the lines

lmp.close()
lmp.finalize()

at the end of the script.

1 Like
3

Thanks Axel. It worked just fine.