Unexpected performance issues with PPPM K-space solver

LAMMPS
1

Hello everyone,

I am running LAMMPS simulations of bulk sodium chloride to compute the thermal conductivity. I have used the standard setup of [fixed region, thermostat1, device region, thermostat2, fixed region]. I am using the born/coul/long inter-ion potentials with a 15 A cutoff and a PPPM 1e-6 K-space solver to compute inter ion forces beyond that distance.

I ran 6 simulations, incrementing the size of the device region from 25, 30, 35 , … , 50 unit cells along the transport direction. All other parameters (other system dimensions, potentials, solvers, fixes, dumps, everything) remained the same between sims except the device region length. This corresponded to incrementing the number of atoms in each sim as 22100, 23300, 24500, … i.e. by small percentage changes. My problem is that the performance nontheless varies significantly amongst the sims, and checking the performance breakdown in the log file confirms it is only the k-space solver contributing to the change in performance across sims. My performance times were:

Device Region Length (u.c.) Performance (ns/day)

25 ____________________ 4.34
30 ____________________ 3.44
35 ____________________ 4.21
40 ____________________ 4.25
45 ____________________ 2.94
50 ____________________ 2.81

Each sim was run on an identical node (64 cores, same clock speeds, memory etc…) on an HPC cluster. Furthermore, sims with 45 u.c. and 25 u.c. were run on the exact same node suggesting this is not the source of the difference. The average times for Pair Potential evaluation, fixes, modification, communications, output, and neighbour list building all remained approximately constant across sims with only the K-space solver varying. The K-space solver generally takes 80% (I need accurate forces) of the computation time.

Apparently, as a new user, I may not attach the input script for the 25 u.c. sim as an example. If you would like to view the script I am happy to send it by another means, or copy it into the thread. Any suggested explanations for this phenomenon / solutions would be greatly appreciated. Thanks

2

PPPM works by overlaying the entire cell with a grid, then accumulating the charge density from the atoms according to a spread function, then computing the electric field caused by the charge density through a convolution in reciprocal space, and finally applying the force from the field. This accumulation and the application of the forces depend on then number of atoms and the width of the spread function and its interpolation order, but the remainder depends on the size of the grid, and the most time consuming steps are the six Fourier transforms and the corresponding transposes of the grid of the forward and backward 3d-FFT, which roughly scale O(N log(N)) with N grid points, until you use so many MPI processes so that the grid transpose becomes dominant due to the growing amount of time spent on MPI_Alltoall() calls.

This is consistent with your observation that the computational cost goes up with increasing the volume and that this cost is mostly in the KSpace part of the calculation.

3

Then you should consider turning off the use of tabulation for the Coulomb forces, which will lead to an additional slowdown in exchange for a more accurate force computation, but “only” in the real space pair force computation.

4

I appreciate these suggestions, but unless I am missing something they do not seem to account for the observed performance trend at all.

The average computational time required by the K-space solver increases by 33% when increasing the device length from 25 to 30 unit cells. This corresponds to increasing number of atoms in the sim from 22100 to 23300 or by 5%. The computational time then drops back to the expected O(NlogN) trend for the 35 and 40 u.c. sims (24500, 25700 atoms respectively) despite the number of atoms having increased. It then, after a further increase by 4-5% of the number of atoms (final two sims), increases by 50%. It thus appears that:

  1. The time for the K-space solver to run does not increase monotonically with number of atoms here

  2. The increase is certainly not O(NlogN), with 5% increases in number of atoms resulting in 33-50% increases in K-space solver time for certain sims.

The 25 u.c., 35 u.c., and 40 u.c. sims are consistent with the O(NlogN) trend in computational time. The other three are evidently not. My question is thus: do you know of any other effect that might be contributing to this surprising performance drop? I am using PBCs in x and y, fixed boundary conditions in z, and have k_space modify set to slab 3.0

Thank you very much for your time and advice

5

You are missing that the times for the PPPM steps are additive, you seem to be assuming that they are multiplicative.

You are also missing that I wrote “O(N log(N)) with N grid points” which applies only to the Fourier transform steps and is thus dependent on the volume of the system, but since fast fourier transforms only support decomposition into certain prime factors, the increase of grid points is not monotonic with system size.

Bottom line: when you increase the number of atoms, some parts of PPPM need more time, others not, when you increase the volume, some other parts of PPPM need more time and the rest not. Until you reach the point where one is dominant (e.g. for a very dilute system), the actual increase does not follow any specific function. Plus you have the “parallel transpose cost” wildcard, which only matters when you have a large number of MPI ranks relative to the size of the grid (but then quickly becomes dominant with an exponential growth, I think).

6

I did not discuss load imbalance and how LAMMPS maps the domains to the volume assuming a uniform distribution of particles which will lead to a large load imbalance of the steps depending on the number of atoms in Kspace and for Pair.

I would recommend to use the processors command and set the number of processors for the domain where you change the dimensions to 1. So if that would be the z-dimension, you could use processors * * 1.

I would also suggest to have a look at: 7.2. Measuring performance — LAMMPS documentation

7

Thank you very much for clarifying these points, this has been very helpful and informative. Apologies for my misunderstandings of your answer!

1 Like
8

Hello again,

Sorry to reopen this thread, but I have done some more detailed analysis of the kspace performance data and have read through the page 7.2. Measuring Performance in the docs as you suggested. I realized something which in hindsight should have been obvious, but I initially missed. I am not, however, sure how to interpret it nor what the solution might be and would appreciate some further guidance please.

Between the simulations, all of the kspace solver performance metrics remain almost constant (the number of grid points in z increases by 4-5 % increments with system length in that direction, same stencil order of 5) but the “3d grid and FFT values/proc” changes dramatically. The values for this line, and simulation cell dimension in z are listed below:

Sim cell Z length (nm) _____________ “3d grid and FFT values/proc”
53.146 __________________________ 550,912 308,160
55.966 __________________________ 941,824 609,120
58.786 __________________________ 591,872 331,776
61.606 __________________________ 591,360 338,928
64.426 __________________________ 1,054,976 681,480
67.246 __________________________ 1,121,536 725,760

The grid is 45 x 45 x 750 for the 53.1 nm sim, 45 x 45 x 960 for the 67.2 nm one, with basically linear increment in the z value between. I assume this has to do with how the grid is being subdivided between my processors but am not sure what exactly is happening nor how to fix it. I am using balance 1.1 rcb and comm_style tiled. I have also checked that this is not a communication issue by benchmarking short runs of these sims with different combinations of OMP threads and MPIs. For a fixed OMP x MPI, OMP = 1 is always fastest. All sims used 64 MPI processes. Thank you for any advice in advance.

9

That only balances the per-atom computation.
The 3d grid for the FFTs is has to be regular in space and what you describe is exactly the expected behavior.

I would recommend against using balance rcb and comm_style tiled, since recursive bisectioning is addressing load imbalance problems of very different geometries.

From what I know about your system, a processors * * 1 command should be all that is needed, given that you have only a moderate number of MPI ranks. The rest is what I like to call “premature optimization” and just adding overhead without any specific performance gains.

At this point it is completely unclear to me, what you are trying to “fix”.

10

Thank you, this did decrease the runtime of the kspace solver. As a final check, my simulation cell contains necessary regions of vacuum at the ends (in z) and, in some simulations, in the center as well between the two blocks of NaCl. Is not using a balance command and using processors * * 1 cause likely to cause a load imbalance?

11

You can observe any load imbalances in the performance summary. As with all simulation program features, you should not use a feature unless you fully understand how it works

12

note that if you have an even number of processors, and you have arranged your system such that the gaps are roughly symmetric on each z-end of the cell, you can also use processors * * 2 and may be able to observe a minor speedup.

the main goal is to have each processor “handle” a roughly equal amount of work. if each processor has an even amount of electrolyte, electrode, and gap, then this criterion is roughly reached.

1 Like