Number of OpenMP threads in log file when running from PBS

LAMMPS LAMMPS General Discussion
1

LAMMPS version: 290824

Hi, I found something strange in the way the number of OpenMP threads are reported in the log file.
I am running on a cluster that uses PBS as scheduler, with the following settings:

#PBS -l select=4:ncpus=48:mpiprocs=48:ompthreads=96

Hence, I am using 4 nodes, 48x4=192 MPI ranks and 96x4=384 OpenMP threads (i.e. 2 threads per rank).
In the LAMMPS log file I read:

Performance: 2.997 ns/day, 8.007 hours/ns, 34.690 timesteps/s, 1.110 Matom-step/s
92.7% CPU use with 192 MPI tasks x 96 OpenMP threads

which I am almost 100% sure cannot be. Not only this is inconsistent with the PBS settings, but I think it’s physically impossible to run 96 OpenMP threads per MPI rank, when all the physical cores are occupied by MPI processes (the CPUs on the cluster have 24 physical cores and 48 threads, 2 CPUs per node).
So there’s either a bug in the way the log file is written, or PBS is ‘confusing’ LAMMPS for some reason. I would expect to read:

[...] 192 MPI tasks x 2 OpenMP threads

right?

2

Wrong! OpenMP doesn’t care how many physical cores you have. If you request 96 threads per MPI task, you get them. Even if that will make you dead slow.
BTW: the same is true for MPI.

LAMMPS uses the information that is provided by the standard OpenMP API to determine the number of threads. I suspect you don’t understand the logic of the PBS directives. You do request 96 OpenMP threads there, and I am fairly certain that that number is the number of threads per MPI task. You should easily confirm that by inspecting the environment variable
OMP_NUM_THREADS. In LAMMPS you can do this with the following two lines:

variable nthreads getenv OMP_NUM_THREADS
print "Threads per MPI: ${nthreads}"

It should be consistent to what LAMMPS sees.

3

Hi you’re right: ompthreads is actually per rank, not per node.
So the log output is correct.
Problem is: my simulation is faster with ompthreads=96 that with ompthreads=2, which doesn’t make any sense. But that’s a different problem.