Hi people,
I’m doing a simulation of a sheet being dragged on top of a substrate, where I run into trouble when the sheet transitions through the periodic boundary. This is only a problem when running it with KOKKOS.
My problem can be reproduced by the following simplified code, where I drag a free floating graphene sheet by its end. That is, I apply a fix move on a dedicated part of the sheet (pull block: PB), while the remaining part is being modeled by the Tersoff potensial fitted for graphene. I use periodic boundary conditions in the plane of the sheet (x,y), and as soon as the non-fixed part of the sheet hits the boundary, it rips apart from the PB and scatters atoms from the connections. The reproduction code reads as follows:
units metal
newton on
boundary p p m
atom_style atomic
# Graphene lattice
lattice custom 2.419 &
a1 0 1.0 0 &
a2 $(sqrt(3)/2) 0.5 0 &
a3 $(1/(2*sqrt(3))) 0.5 0.83 &
basis 0 0 0 &
basis $(1/3) $(1/3) 0.0
# Simulation box & atoms
region simreg block 5 15 5 15 0 0.5
region boxreg block 0 20 0 20 -5 5
create_box 1 boxreg
create_atoms 1 region simreg basis 1 1
# Pull part of the sheet
region PB_region block INF INF 65.0 INF INF 1 units box # Pull block
group PB region PB_region
group integrate subtract all PB
fix move_PB PB move linear 0 2 0 units box
# Dynamics
mass 1 12.0107
velocity integrate create 1.0 5432373 dist gaussian
pair_style tersoff
pair_coeff * * C.tersoff C
timestep 0.001
fix nve integrate nve
# Output
thermo 1000
dump 1 all custom 100 dump_reproduce.data id type x y z vx vy vz
run 25000
where the file C.tersoff reads
C C C 3.0 1.0 0.0 3.8049e4 4.3484 -0.57058 0.72751 1.5724e-7 2.2119 346.74 1.95 0.15 3.4879 1393.6
This runs perfectly fine on CPU but when running it with KOKKOS, using the following job script, it fails a described initially.
#!/bin/bash
#SBATCH --job-name=NG4_GPU
#
#SBATCH --partition=normal
#
#SBATCH --ntasks=1
#
#SBATCH --cpus-per-task=2
#
#SBATCH --gres=gpu:1
#
#SBATCH --output=slurm.out
#
mpirun -n 1 lmp -pk kokkos newton on neigh half -k on g 1 -sf kk -in reproduce.in
My main suspicion is that KOKKOS might not support this due to the warning
WARNING: Fixes cannot yet send exchange data in Kokkos communication, switching to classic exchange/border communication (src/KOKKOS/comm_kokkos.cpp:651)
In order to investigate the problem further I ran a simulation where I unfixed the ‘fix move’ before going through the periodic boundary, and also a simulation where I applied the fix move to the whole sheet (no interatomic potential). Both ran perfectly fine, and thus it is only in the combination of a fix move on the PB with the remaining part being modeled by the interatomic potential that I cannot pass the periodic boundary successfully.
I hope that you might be able to guide me in the right direction here.
Thanks in advance.
Best regards
Mikkel