Error in AMOEBA GPU

Hello folks:
I have successfully compiled LAMMPS on NERSC (Perlmutter) and enabled GPU acceleration using CUDA. I’ve verified that this works by performing some benchmarking with a LJ fluid (I do this by first requesting and login into an interactive GPU node).

I’m using the latest development branch, obtained from git:
LAMMPS (15 Jun 2023 - Development - patch_15Jun2023-30-gc5d9f901d9-modified)

I am also able to run the CPU version of the AMOEBA water_box example, with no errors:
srun -n 2 --cpu_bind=cores …/…/build_gpu/lmp -in in.water_box.amoeba
works
However, when I try the GPU version:
srun -n 2 --cpu_bind=cores …/…/build_gpu/lmp -sf gpu -pk gpu 1 -in in.water_box.amoeba
I get the following errors:

MPICH ERROR [Rank 0] [job id 11017187.8] [Mon Jul 3 09:04:37 2023] [nid001004] - Abort(-1) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
Cuda driver error 4 in call at file ‘/global/homes/t/mike2mitch/codes/lammps/lammps-dev2/lib/gpu/geryon/nvd_device.h’ in line 429.
MPICH ERROR [Rank 0] [job id 11017187.8] [Mon Jul 3 09:04:37 2023] [nid001004] - Abort(-1) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0

Is this is bug or is there anything special that I would need to get the GPU version of AMOEBA working?

Many thanks
Mike

@MikeMitch thanks for your report. This was working not so long ago. We tested on a variety of options (double, mixed, or single precision and CUDA vs OpenCL and GPUs with shared and dedicated memory and Intel vs. Nvidia GPUs).

@ndtrung is looking into correcting this issue.

What he found out so far is, that this seems to be a bug that was introduced or unmasked with the changes in Misc Improvements to GPU Package by wmbrownIntel · Pull Request #3675 · lammps/lammps · GitHub

Thus reverting to the 8 February 2023 version should restore the code back to a working GPU acceleration for the AMOEBA package. Unfortunately, that will also undo several bugfixes to the GPU package that were applied since.
We are aiming to have this resolved for the next LAMMPS release (which is supposed to become a new stable release).