Kokkos error: Constructing View and initializing data with uninitialized execution space

I am trying to use the Kokkos package so as to utilize the GPU in some ReaxFF calculations. When I try to run an example calculation (examples/reaxff/RDX), with the following command:
lmp -in in.RDX -suffix kk > out.RDX
I get the following error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Constructing View and initializing data with uninitialized execution space
Traceback functionality not available

[Heisenberg:10160] *** Process received signal ***
[Heisenberg:10160] Signal: Aborted (6)
[Heisenberg:10160] Signal code:  (-6)
[Heisenberg:10160] [ 0] /lib64/libpthread.so.0(+0xf370)[0x7f0c533ae370]
[Heisenberg:10160] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f0c527771d7]
[Heisenberg:10160] [ 2] /lib64/libc.so.6(abort+0x148)[0x7f0c527788c8]
[Heisenberg:10160] [ 3] /home/rashid/Apps/spack/opt/spack/linux-centos7-broadwell/gcc-11.2.0/gcc-7.5.0-eoiw7jz56uxm2vul7osvg4xhrfdwnwk5/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_ter
minate_handlerEv+0x125)[0x7f0c530ad7d5]
[Heisenberg:10160] [ 4] /home/rashid/Apps/spack/opt/spack/linux-centos7-broadwell/gcc-11.2.0/gcc-7.5.0-eoiw7jz56uxm2vul7osvg4xhrfdwnwk5/lib64/libstdc++.so.6(+0x8f5b6)[0x7f0c530ab5b6]
[Heisenberg:10160] [ 5] /home/rashid/Apps/spack/opt/spack/linux-centos7-broadwell/gcc-11.2.0/gcc-7.5.0-eoiw7jz56uxm2vul7osvg4xhrfdwnwk5/lib64/libstdc++.so.6(+0x8f601)[0x7f0c530ab601]
[Heisenberg:10160] [ 6] /home/rashid/Apps/spack/opt/spack/linux-centos7-broadwell/gcc-11.2.0/gcc-7.5.0-eoiw7jz56uxm2vul7osvg4xhrfdwnwk5/lib64/libstdc++.so.6(+0x8f843)[0x7f0c530ab843]
[Heisenberg:10160] [ 7] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x16db7c0]
[Heisenberg:10160] [ 8] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x53a280]
[Heisenberg:10160] [ 9] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0xd73889]
[Heisenberg:10160] [10] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x9293b1]
[Heisenberg:10160] [11] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x9260bf]
[Heisenberg:10160] [12] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x91e0a4]
[Heisenberg:10160] [13] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x4781f4]
[Heisenberg:10160] [14] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x4746d6]
[Heisenberg:10160] [15] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x412ddf]
[Heisenberg:10160] [16] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0c52763b35]
[Heisenberg:10160] [17] /home/rashid/Apps/lammps-30Jul2021/build_gcc_kokkos_cuda11/lmp[0x44c1a3]
[Heisenberg:10160] *** End of error message ***

Any pointers on how to solve this so as to get the Kokkos package running would be very helpful.

I am using LAMMPS-30Jul2021, compiled with GCC 7.5.0 using openmpi and cuda 11.4.0. The cmake command I used to build LAMMPS is:

cmake -C ../cmake/presets/basic.cmake -D PKG_KOKKOS=on -DPKG_REAXFF=on -D Kokkos_ARCH_PASCAL60=yes -D Kokkos_ENABLE_CUDA=yes -D Kokkos_ENABLE_OPENMP=no -D CMAKE_CXX_COMPILER=/home/rashid/Apps/lammps-30Jul2021/lib/kokkos/bin/nvcc_wrapper ../cmake

Please try again with the current stable release (29 Sep 2021). We just updated it with a few bugfixes including a couple KOKKOS related ones.

Okay thanks. I’ll try that and update here.

Please also note that your quoted command line does not enable Kokkos and select how many gpus and threads should be used.

Under normal circumstances there should be an error about this before the one you quote is possible.

1 Like

I forgot to update the results here after my previous message.

This was the issue. I had not read the Kokkos package page fully. After fixing my command based on the instructions there, the issue was solved. The command which worked was this:

lmp -in in.RDX -k on g 1 -suffix kk -pk kokkos newton on neigh half > out.RDX

Even though the calculation completes successfully, the following error gets displayed after the calculation each time I run LAMMPS using the above command:

Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()

Hope this is not a big issue as this happens after the calculation.

This is with LAMMPS-30Jul2021.

Thanks for the pointers.

I also tried the current stable release (29 Sep 2021) compiled with same parameters mentioned above. But with it I am getting an error when I try to run the same calculation as above:

*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: (nil)
[ 0] /lib64/libpthread.so.0(+0xf370)[0x7f10c3d5b370]
[ 1] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x610a4b]
[ 2] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x85d115]
[ 3] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x6c6e0b]
[ 4] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x8acc61]
[ 5] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x52ce5c]
[ 6] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x46e0df]
[ 7] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x46e7db]
[ 8] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x411b18]
[ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f10c3412b35]
[10] /home/rashid/GitProjects/lammps/build_kokkos/lmp[0x44ca03]
*** End of error message ***

This is an issue that has since been corrected.

Please provide the output of lmp -h

The output is a bit long but I’m including it fully:

[[email protected] build_kokkos]$ ./lmp -h                                                                                            
                                                                                                                                      
Large-scale Atomic/Molecular Massively Parallel Simulator - 27 Oct 2021                                                               
Git info (release / patch_27Oct2021)                                                                                                  
                                                                                                                                      
Usage example: ./lmp -var t 300 -echo screen -in in.alloy                                                                             
                                                                                                                                      
List of command line options supported by this LAMMPS executable:                                                                     
                                                                                                                                      
-echo none/screen/log/both  : echoing of input script (-e)                                                                            
-help                       : print this help message (-h)                                                                            
-in none/filename           : read input from file or stdin (default) (-i)                                                            
-kokkos on/off ...          : turn KOKKOS mode on or off (-k)                                                                         
-log none/filename          : where to send log output (-l)                                                                           
-mdi '<mdi flags>'          : pass flags to the MolSSI Driver Interface                                                               
-mpicolor color             : which exe in a multi-exe mpirun cmd (-m)                                                                
-cite                       : select citation reminder style (-c)                                                                     
-nocite                     : disable citation reminder (-nc)                                                                         
-package style ...          : invoke package command (-pk)                                                                            
-partition size1 size2 ...  : assign partition sizes (-p)                                                                             
-plog basename              : basename for partition logs (-pl)                                                                       
-pscreen basename           : basename for partition screens (-ps)                                                                    
-restart2data rfile dfile ... : convert restart to data file (-r2data)                                                                
-restart2dump rfile dgroup dstyle dfile ...                                                                                           
                            : convert restart to dump file (-r2dump)                                                                  
-reorder topology-specs     : processor reordering (-r)                                                                               
-screen none/filename       : where to send screen output (-sc)                                                                       
-skiprun                    : skip loops in run and minimize (-sr)                                                                    
-suffix gpu/intel/opt/omp   : style suffix to apply (-sf)                                                                             
-var varname value          : set index style variable (-v)                                                                           
                                                                                                                                      
OS: Linux "CentOS Linux 7 (Core)" 3.10.0-514.2.2.el7.x86_64 x86_64                                                                    
                                                                                                                                      
Compiler: GNU C++ 7.5.0 with OpenMP 4.5                                                                                               
C++ standard: C++14                                                                                                                   
MPI v3.1: Open MPI v4.1.1, package: Open MPI [email protected] Distribution, ident: 4.1.1, repo rev: v4.1.1, Apr 24, 2021 
                                                                                                                                      
Accelerator configuration:                                                                                                            
                                                                                                                                      
KOKKOS package API: CUDA Serial                                                                                                       
KOKKOS package precision: double                                                                                                      
                                                                                                                                      
Active compile time flags:                                                                                                            
                                                                                                                                      
-DLAMMPS_GZIP                                                                                                                         
-DLAMMPS_PNG                                                                                                                          
-DLAMMPS_FFMPEG                                                                                                                       
-DLAMMPS_SMALLBIG                                                                                                                     
sizeof(smallint): 32-bit                                                                                                              
sizeof(imageint): 32-bit                                                                                                              
sizeof(tagint):   32-bit                                                                                                              
sizeof(bigint):   64-bit                                                                                                              
                                                                                                                                      
Available compression formats:                                                                                                        
                                                                                                                                      
Extension: .gz     Command: gzip                                                                                                      
Extension: .bz2    Command: bzip2                                                                                                     
Extension: .xz     Command: xz                                                                                                        
Extension: .lzma   Command: xz                                                                                                        
                                                                                                                                      
                                                                                                                                      
Installed packages:                                                                                                                   
                                                                                                                                      
KOKKOS KSPACE MANYBODY MOLECULE REAXFF RIGID                                                                                          
                                                                                                                                      
List of individual style options included in this LAMMPS executable                                                                   
                                                                                                                                      
* Atom styles:                                                                                                                        

angle           angle/kk        angle/kk/device angle/kk/host   atomic                                                                
atomic/kk       atomic/kk/device                atomic/kk/host  body                                                                  
bond            bond/kk         bond/kk/device  bond/kk/host    charge                                                                
charge/kk       charge/kk/device                charge/kk/host  ellipsoid                                                             
full            full/kk         full/kk/device  full/kk/host    hybrid                                                                
hybrid/kk       line            molecular       molecular/kk                                                                          
molecular/kk/device             molecular/kk/host               sphere                                                                
sphere/kk       sphere/kk/device                sphere/kk/host  template                                                              
tri                                                                                                                                   
                                                                                                                                      
* Integrate styles:                                                                                                                   
                                                                                                                                      
respa           verlet          verlet/kk       verlet/kk/device                                                                      
verlet/kk/host                                                                                                                        
                                                                                                                                      
* Minimize styles:                                                                                                                    
                                                                                                                                      
cg              cg/kk           cg/kk/device    cg/kk/host      fire                                                                  
fire/old        hftn            quickmin        sd                                                                                    

* Pair styles:                                                                                                                        

adp             airebo          airebo/morse    atm             bop                                                                   
born            born/coul/long  born/coul/msm   buck            buck/coul/cut                                                         
buck/coul/cut/kk                buck/coul/cut/kk/device                                                                               
buck/coul/cut/kk/host           buck/coul/long  buck/coul/long/kk                                                                     
buck/coul/long/kk/device        buck/coul/long/kk/host          buck/coul/msm                                                         
buck/kk         buck/kk/device  buck/kk/host    buck/long/coul/long                                                                   
comb            comb3           coul/cut        coul/cut/kk                                                                           
coul/cut/kk/device              coul/cut/kk/host                coul/debye                                                            
coul/debye/kk   coul/debye/kk/device            coul/debye/kk/host                                                                    
coul/dsf        coul/dsf/kk     coul/dsf/kk/device              coul/dsf/kk/host                                                      
coul/long       coul/long/kk    coul/long/kk/device                                                                                   
coul/long/kk/host               coul/msm        coul/streitz    coul/wolf                                                             
coul/wolf/kk    coul/wolf/kk/device             coul/wolf/kk/host                                                                     
reax            eam             eam/alloy       eam/alloy/kk                                                                          
eam/alloy/kk/device             eam/alloy/kk/host               eam/cd                                                                
eam/cd/old      eam/fs          eam/fs/kk       eam/fs/kk/device                                                                      
eam/fs/kk/host  eam/he          eam/kk          eam/kk/device   eam/kk/host                                                           
edip            edip/multi      eim             extep           gw                                                                    
gw/zbl          hbond/dreiding/lj               hbond/dreiding/morse                                                                  
hybrid          hybrid/kk       hybrid/overlay  hybrid/overlay/kk                                                                     
hybrid/scaled   lcbop           lj/charmm/coul/charmm                                                                                 
lj/charmm/coul/charmm/implicit  lj/charmm/coul/charmm/implicit/kk                                                                     
lj/charmm/coul/charmm/implicit/kk/device                                                                                              
lj/charmm/coul/charmm/implicit/kk/host          lj/charmm/coul/charmm/kk                                                              
lj/charmm/coul/charmm/kk/device lj/charmm/coul/charmm/kk/host                                                                         
lj/charmm/coul/long             lj/charmm/coul/long/kk                                                                                
lj/charmm/coul/long/kk/device   lj/charmm/coul/long/kk/host                                                                           
lj/charmm/coul/msm              lj/charmmfsw/coul/charmmfsh                                                                           
lj/charmmfsw/coul/long          lj/cut          lj/cut/coul/cut                                                                       
lj/cut/coul/cut/kk              lj/cut/coul/cut/kk/device                                                                             
lj/cut/coul/cut/kk/host         lj/cut/coul/long                                                                                      
lj/cut/coul/long/kk             lj/cut/coul/long/kk/device                                                                            
lj/cut/coul/long/kk/host        lj/cut/coul/msm lj/cut/kk       lj/cut/kk/device                                                      
lj/cut/kk/host  lj/cut/tip4p/cut                lj/cut/tip4p/long                                                                     
lj/expand       lj/expand/kk    lj/expand/kk/device                                                                                   
lj/expand/kk/host               lj/long/coul/long                                                                                     
lj/long/tip4p/long              local/density   meam/spline     meam/sw/spline                                                        
morse           morse/kk        morse/kk/device morse/kk/host   nb3b/harmonic                                                         
polymorphic     reaxff          reax/c          reaxff/kk       reaxff/kk/device                                                      
reaxff/kk/host  reax/c/kk       reax/c/kk/device                reax/c/kk/host                                                        
rebo            soft            sw              sw/kk           sw/kk/device                                                          
sw/kk/host      table           table/kk        table/kk/device table/kk/host                                                         
tersoff         tersoff/kk      tersoff/kk/device               tersoff/kk/host                                                       
tersoff/mod     tersoff/mod/c   tersoff/mod/kk  tersoff/mod/kk/device                                                                 
tersoff/mod/kk/host             tersoff/table   tersoff/zbl     tersoff/zbl/kk                                                        
tersoff/zbl/kk/device           tersoff/zbl/kk/host             tip4p/cut                                                             
tip4p/long      vashishta       vashishta/kk    vashishta/kk/device                                                                   
vashishta/kk/host               vashishta/table yukawa          yukawa/kk                                                             
yukawa/kk/device                yukawa/kk/host  zbl             zbl/kk                                                                
zbl/kk/device   zbl/kk/host     zero                                                                                                  

* Bond styles:                                                                                                                        

fene            fene/expand     fene/kk         fene/kk/device  fene/kk/host                                                          
gromos          harmonic        harmonic/kk     harmonic/kk/device                                                                    
harmonic/kk/host                hybrid          morse           quartic                                                               
table           zero                                                                                                                  

* Angle styles:                                                                                                                       

charmm          charmm/kk       charmm/kk/device                charmm/kk/host                                                        
cosine          cosine/kk       cosine/kk/device                cosine/kk/host                                                        
cosine/squared  harmonic        harmonic/kk     harmonic/kk/device                                                                    
harmonic/kk/host                hybrid          table           zero                                                                  

* Dihedral styles:                                                                                                                    

charmm          charmm/kk       charmm/kk/device                charmm/kk/host                                                        
charmmfsw       harmonic        harmonic/kk     harmonic/kk/device                                                                    
harmonic/kk/host                hybrid          multi/harmonic  opls                                                                  
opls/kk         opls/kk/device  opls/kk/host    table           zero                                                                  

* Improper styles:                                                                                                                    

cvff            harmonic        harmonic/kk     harmonic/kk/device                                                                    
harmonic/kk/host                hybrid          umbrella        zero                                                                  
                                                                                 
* KSpace styles:                                                                 
                                                                                 
ewald           ewald/dipole    ewald/dipole/spin               ewald/disp       
ewald/disp/dipole               msm             msm/cg          pppm             
pppm/cg         pppm/dipole     pppm/dipole/spin                pppm/disp        
pppm/disp/tip4p pppm/kk         pppm/kk/device  pppm/kk/host    pppm/stagger     
pppm/tip4p                                                                       
                                                                                 
* Fix styles                                                                     

acks2/reax      acks2/reaxff    acks2/reaxff/kk acks2/reaxff/kk/device           
acks2/reaxff/kk/host            acks2/reax/kk   acks2/reax/kk/device             
acks2/reax/kk/host              adapt           addforce        ave/atom         
ave/chunk       ave/correlate   ave/histo       ave/histo/weight                 
ave/time        aveforce        balance         box/relax       cmap             
deform          deform/kk       deform/kk/device                deform/kk/host   
deposit         ave/spatial     ave/spatial/sphere              dt/reset         
efield          ehex            enforce2d       enforce2d/kk                     
enforce2d/kk/device             enforce2d/kk/host               evaporate        
external        gravity         gravity/kk      gravity/kk/device                
gravity/kk/host halt            heat            indent          langevin         
langevin/kk     langevin/kk/device              langevin/kk/host                 
lineforce       momentum        momentum/kk     momentum/kk/device               
momentum/kk/host                move            nph             nph/kk           
nph/kk/device   nph/kk/host     nph/sphere      npt             npt/kk           
npt/kk/device   npt/kk/host     npt/sphere      nve             nve/kk           
nve/kk/device   nve/kk/host     nve/limit       nve/noforce     nve/sphere       
nve/sphere/kk   nve/sphere/kk/device            nve/sphere/kk/host               
nvt             nvt/kk          nvt/kk/device   nvt/kk/host     nvt/sllod        
nvt/sllod/kk    nvt/sllod/kk/device             nvt/sllod/kk/host                
nvt/sphere      planeforce      press/berendsen print           property/atom    
property/atom/kk                qeq/comb        qeq/reaxff      qeq/reax         
qeq/reaxff/kk   qeq/reaxff/kk/device            qeq/reaxff/kk/host               
qeq/reax/kk     qeq/reax/kk/device              qeq/reax/kk/host                 
rattle          reaxff/bonds    reax/c/bonds    reaxff/bonds/kk                  
reaxff/bonds/kk/device          reaxff/bonds/kk/host            reax/c/bonds/kk  
reax/c/bonds/kk/device          reax/c/bonds/kk/host            reaxff/species   
reax/c/species  reaxff/species/kk               reaxff/species/kk/device         
reaxff/species/kk/host          reax/c/species/kk                                
reax/c/species/kk/device        reax/c/species/kk/host          recenter         
restrain        rigid           rigid/nph       rigid/nph/small rigid/npt        
rigid/npt/small rigid/nve       rigid/nve/small rigid/nvt       rigid/nvt/small  
rigid/small     setforce        setforce/kk     setforce/kk/device               
setforce/kk/host                shake           shake/kk        shake/kk/device  
shake/kk/host   spring          spring/chunk    spring/self     store/force      
store/state     temp/berendsen  temp/rescale    thermal/conductivity             
tune/kspace     vector          viscous         wall/harmonic   wall/lj1043      
wall/lj126      wall/lj93       wall/lj93/kk    wall/lj93/kk/device              
wall/lj93/kk/host               wall/morse      wall/reflect    wall/reflect/kk  
wall/reflect/kk/device          wall/reflect/kk/host            wall/region      
                                                                                 
* Compute styles:                                                                
                                                                                 
aggregate/atom  angle           angle/local     angmom/chunk    bond             
bond/local      centro/atom     centroid/stress/atom            chunk/atom       
chunk/spread/atom               cluster/atom    cna/atom        com              
com/chunk       coord/atom      coord/atom/kk   coord/atom/kk/device             
coord/atom/kk/host              dihedral        dihedral/local  dipole           
dipole/chunk    displace/atom   erotate/rigid   erotate/sphere                   
erotate/sphere/atom             fragment/atom   global/atom     group/group      
gyration        gyration/chunk  heat/flux       improper        improper/local   
inertia/chunk   ke              ke/atom         ke/rigid        msd              
msd/chunk       omega/chunk     orientorder/atom                                 
orientorder/atom/kk             orientorder/atom/kk/device                       
orientorder/atom/kk/host        pair            pair/local      pe               
pe/atom         pressure        property/atom   property/chunk  property/local   
rdf             reduce          reduce/chunk    reduce/region   rigid/local      
slice           stress/atom     temp            temp/chunk      temp/com         
temp/deform     temp/deform/kk  temp/deform/kk/device                            
temp/deform/kk/host             temp/kk         temp/kk/device  temp/kk/host     
temp/partial    temp/profile    temp/ramp       temp/region     temp/sphere      
torque/chunk    vacf            vcm/chunk                                        
                                                                                 
* Region styles:                                                                 
                                                                                 
block           block/kk        block/kk/device block/kk/host   cone             
cylinder        intersect       plane           prism           sphere           
union                                                                            

* Dump styles:                                                                   
                                                                                 
atom            cfg             custom          image           local            
movie           xyz                                                              

* Command styles                                                                 

balance         change_box      create_atoms    create_bonds    create_box       
delete_atoms    delete_bonds    reset_ids       kim_init        kim_interactions 
kim_param       kim_property    kim_query       displace_atoms  info             
minimize        read_data       read_dump       read_restart    replicate        
rerun           reset_atom_ids  reset_mol_ids   run             set              
velocity        write_coeff     write_data      write_dump      write_restart    

As might have been clear from the above post, the error mentioned previously comes not with the stable release (29 Sep 2021), but with a pre-release version (patch_27Oct2021) and it was a mistake from my part to say so.

I just tried the same with the stable release:

$ ./lmp -h

Large-scale Atomic/Molecular Massively Parallel Simulator - 29 Sep 2021 - Update 1
Git info (HEAD / stable_29Sep2021_update1)

and it doesn’t give any of the errors above. Thanks.

@rashidrafeek thanks for the update and the clarification. It is reassuring to see that the stable release is working as expected.

If you have the time, please also repeat the test with the “develop” branch. There was one bugfix applied to LAMMPS since the 27 October 2021 release that is related to ReaxFF and thus is likely the fix for the segmentation fault you have observed, but it would be nice to have the confirmation. Thanks.

Yes, you are right. I tried the test with the “develop” branch and it did fix the issues.

./lmp -h

Large-scale Atomic/Molecular Massively Parallel Simulator - 27 Oct 2021
Git info (develop / patch_27Oct2021-107-ga4ceda9)

Thanks for confirming.

1 Like