running OMP?

On previous versions of lammps (as recent as May) I have been able to run omp accelerated pair styles with no problems. I reinstalled and recompiled (thanks Axel for the help) lammps today and now I can’t seem to get the omp styles to work properly.

I am running with:

env OMP_NUM_THREADS=2 mpirun -np 8 ~/bin/lmp_openmpi -sf omp -in start.lmp

FIrst little bit of output is:
LAMMPS (6 Jun 2012)
XXXXX Typically I would see Using X 2 OpenMP thread(s) per MPI task here, but today I am not XXXX
using OpenMP capable neighbor list subroutines
Scanning data file …
Reading data file …
orthogonal box = (-75.4443 -39.202 -20) to (75.4443 39.202 170.847)
2 by 1 by 4 MPI processor grid

…snip…
Later in the output I see:
Last active /omp style is pair_style airebo/omp

When I run top to see what the machine is doing it’s only running at 50% of what I think it should be running, in my case 8 procs vs 16.

dave,

On previous versions of lammps (as recent as May) I have been able to run
omp accelerated pair styles with no problems. I reinstalled and recompiled
(thanks Axel for the help) lammps today and now I can't seem to get the omp
styles to work properly.

I am running with:

env OMP_NUM_THREADS=2 mpirun -np 8 ~/bin/lmp_openmpi -sf omp -in start.lmp

this is not quite correct. with OpenMPI you need to use either:

env OMP_NUM_THREADS=2 mpirun -x OMP_NUM_THREADS -np 8
~/bin/lmp_openmpi -sf omp -in start.lmp

or (better):

mpirun -x OMP_NUM_THREADS=2 -np 8 ~/bin/lmp_openmpi -sf omp -in start.lmp

FIrst little bit of output is:
LAMMPS (6 Jun 2012)
XXXXX Typically I would see Using X 2 OpenMP thread(s) per MPI task here,
but today I am not XXXX
using OpenMP capable neighbor list subroutines
Scanning data file ...
Reading data file ...
orthogonal box = (-75.4443 -39.202 -20) to (75.4443 39.202 170.847)
2 by 1 by 4 MPI processor grid

....snip...
Later in the output I see:
Last active /omp style is pair_style airebo/omp

When I run top to see what the machine is doing it's only running at 50% of
what I think it should be running, in my case 8 procs vs 16.

env OMP_NUM_THREADS=2 mpirun -x OMP_NUM_THREADS -np 8
~/bin/lmp_openmpi -sf omp -in start.lmp

have a look at your file src/Makefile.package
it should look like this:

# Settings for libraries used by specific LAMMPS packages
# this file is auto-edited when those packages are included/excluded

PKG_INC = -DLMP_USER_OMP -I../../lib/colvars -I../../lib/poems
-I../../lib/meam
PKG_PATH = -L../../lib/colvars -L../../lib/poems -L../../lib/meam
PKG_LIB = -lcolvars -lpoems -lmeam

PKG_SYSINC = \(colvars\_SYSINC\) (meam_SYSINC)
PKG_SYSLIB = \(colvars\_SYSLIB\) (meam_SYSLIB)
PKG_SYSPATH = \(colvars\_SYSPATH\) (meam_SYSPATH)

my guess is that you don't have "-DLMP_USER-OMP" in there.

axel.

Back to the CUDA errors. Ahhhh!

dave,

I am running with:

env OMP_NUM_THREADS=2 mpirun -np 8 ~/bin/lmp_openmpi -sf omp -in start.lmp

this is not quite correct. with OpenMPI you need to use either:

env OMP_NUM_THREADS=2 mpirun -x OMP_NUM_THREADS -np 8

~/bin/lmp_openmpi -sf omp -in start.lmp

or (better):

mpirun -x OMP_NUM_THREADS=2 -np 8 ~/bin/lmp_openmpi -sf omp -in start.lmp

This may ultimately be the solution. I was getting errors when I tried running this command so I went back and looked and I had some weird hybrid of openmpi and mpich2 installed. I cleaned this up and went to recompile and I started getting all kinds of CUDA errors again.

my make sequence is generally:
make no-all
make yes-standard
make no-gpu
make no-kim
rm *cuda.{cpp,h} (I tried with and with out this command)
make openmpi

atom.cpp: In member function ‘void LAMMPS_NS::atom::sort()’:
atom.cpp:1366:30: error: invalid use of incomplete type ‘struct LAMMPS_NS::Cuda’
lammps.h:47:9: error: forward declaration of ‘struct LAMMPS_NS::Cuda’
atom.cpp:1366:48: error: invalid use of incomplete type ‘struct LAMMPS_NS::Cuda’
lammps.h:47:9: error: forward declaration of ‘struct LAMMPS_NS::Cuda’
atom.cpp:1445:30: error: invalid use of incomplete type ‘struct LAMMPS_NS::Cuda’
lammps.h:47:9: error: forward declaration of ‘struct LAMMPS_NS::Cuda’
atom.cpp:1445:48: error: invalid use of incomplete type ‘struct LAMMPS_NS::Cuda’
lammps.h:47:9: error: forward declaration of ‘struct LAMMPS_NS::Cuda’
atom.cpp: In member function ‘int LAMMPS_NS::atom::memcheck(const char*)’:
atom.cpp:1679:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
In file included from atom.cpp:35:0:
memory.h: At global scope:
memory.h: In instantiation of ‘LAMMPS_NS::bigint LAMMPS_NS::Memory::usage(TYPE*, int) [with TYPE = int, LAMMPS_NS::bigint = long int]’:
atom.cpp:1648:51: instantiated from here
memory.h:512:12: warning: unused parameter ‘array’ [-Wunused-parameter]
make[1]: *** [atom.o] Error 1
make[1]: Leaving directory `/home/schall2/Code/Lammps/lammps6-7-2012/src/Obj_openmpi’
make: *** [openmpi] Error 2

so 1 step forward 2 steps back I guess.

i am sorry, but i cannot help it to think that
this seems to be a largely self-inflicted mess.

all i can say to this, that i don't have these
kinds of problems, but i am using my own
branch with a few (but currently *very* few
changes relative to the mainline code) and
i have not run into any of these issues on
multiple and rather different machines.
i am a bit more paranoid in how i do things.
so i manually install packages with dependencies
in a particular order (e.g. first molecule, then
class2, then kspace, then manybody, then opt,
then user-cg-cmm, and so on and user-omp last)
without any GPU support. get these working
first. (first with make serial, then make openmpi,
then make openmpi-omp) then configure the gpu
package and compile one binary, then uninstall
gpu and install user-cuda and compile another
binary (both with openmpi support). this may
be too careful, but it works well for me.

there are a few things that can get broken
when using git or svn with LAMMPS using
the current installation mechanism, e.g. the
Makefile.package or Makefile.package.settings
components.
if they are corrupted it is best to uninstall
all packages, then delete them and have
them overwritten with empty templates.
also, if you update from the svn or git repo,
you have to watch out for files from packages
that get renamed or removed. they won't
get uninstalled and can make problems.
the last thing are customized makefiles.
sometimes the makefiles get additional
entries; those may be missing in your
own customization. it is recommended
to watch those files carefully and compare
to the original. with git it is also prudent
to maintain local modifications as a branch
and leave the master branch untouched.

in short, i don't know what else to advise
outside of 'try a checkout of LAMMPS-ICMS
into a separate directory' since too much
information is missing...

sorry,
    axel.

rm *cuda.{cpp,h} (I tried with and with out this command)

You should not do this. There is a file accelerator_cuda.h
that lives in src and is not part of the USER-CUDA package.
It includes a dummy CUDA class so LAMMPS can build w/out
the USER-CUDA package. If you whack it, you probably
cannot build LAMMPS at all.

Steve

i am sorry, but i cannot help it to think that
this seems to be a largely self-inflicted mess.

No doubt. I am pretty good at making messes.

I tried your paranoid approach and was able to compile both lammps and lammps-icms with new checkouts with no hiccups. It also fixed my omp problem. So I guess the problem was related to dependencies and my not so careful approach of trying the standard make instead of just installing what I need. I haven’t yet tried either gpu / cuda. I’ll take that on later. But at least I now know where and how to start.

Thanks for the help.

Dave

Did you see my message about restoring the src/accelerator_cuda.cpp file
that you whacked? I think if you do that your problems may go away.

Steve

Yeah, I saw it. I had already tried with and with out the accelerator_cuda.cpp file and it did not work. What finally worked was to start with a fresh image and then follow Axel’s stepwise “paranoid” approach. Once I did that, I had no problems and was successful in installing on two different machines. I’ll try our gpu cluster next.

Thanks,

Dave