momentum conservation by using GPU package

LAMMPS LAMMPS Mailing List Mirror
1

Dear Lammps users,

I am running a job, using the GPU package, in the NPT statistical ensemble (T=550K and P=1atm) using the DREIDING force - field.

When I visualized the trajectory file, I mentioned that the whole system started to travel “biased”

to a specific direction after a matter of simulation time (~ 40000000 steps). I started to investigate

the reason that this is happening. From the log file I didn’t mention any shift or drop to any of the sizes

I measure (such as Energy, Temperature, Volume etc.). When I calculated the total momentum of the system

with respect to simulation time though, I mentioned that it wasn’t fluctuating around 0 (as I was expecting)

but it was linearly increasing. In order to be sure that the code I wrote for the momentum calculation was right, I

made exactly the same calculation on trajectory files of simulations I run in the past, without there using the gpu package, and the

result I took, was the one I expected (the total momentum of the system was fluctuating around zero).

I am wondering if this happens due to the GPU package which demands by default the newton command to

be off. Does this affect the momentum conservation of the system?

I also post the .in script I am using to see the simulation parameters I am using.

package gpu force/neigh 0 5 1.0

units real
atom_style full
boundary p p p

pair_style lj/charmm/coul/long/gpu 8.0 12.0
dielectric 1.0
kspace_style pppm/gpu 0.00001
special_bonds lj 0.0 0.0 1.0 coul 0.0 0.0 1.0
bond_style harmonic
angle_style harmonic
dihedral_style charmm
improper_style harmonic

read_data pol_550K.dat

group mobile union all
timestep 1
#reset_timestep 0

fix 1 mobile npt temp 550.00 550.00 10.0 iso 1.000000 1.000000 350.0

velocity mobile create 550 1284578715 mom yes rot yes dist gaussian
dump 1 all atom 50000 pol_water_550K.dump
dump_modify 1 scale no image yes append yes
dump 2 all custom 50000 pol_water_550K.veldump vx vy vz
dump_modify 2 append yes
thermo_style custom etotal ke pe ebond eangle edihed eimp evdwl ecoul elong etail enthalpy te
mp press pxx pxy pxz pyy pyz pzz vol lx ly lz
thermo_modify line multi
thermo 50000
thermo_modify flush yes
restart 50000 pol_550K.rst pol_550K.rst
run 100000000
write_restart pol_550K_final.rst

Thank you very much for your help and your support in advance.

with respect,

Emmanuel Skountzos.

2

No, newton on vs off should not affect this,
other than round-off kinds of differences.

What precision are you running the GPU package
at? Running on the CPU only (w/out GPU package)
will always be double precision. If you ran
using the GPU package with single or mixed
precision, that might affect long timescale properties.

Steve

3

Dear Steve,

i installed the GPU package with its default settings

(mixed precision). So you are right and probably this causes the problem.

I will re-install it with double precision settings and run it again.

Thank you very much for your immediate and accurate response!

Best,

Emmanuel Skountzos.

Στις 2014-09-19 17:04, Steve Plimpton έγραψε:

4

Dear Steve,

I re-installed the Lammps with the GPU package with Double Precision. I did some benchmark tests, but I took strange results. When I tested for the “small” system (~20000 atoms) the results, compared to those of mixed precision, were what I was expected. About 30% down. (For example in the case of the mixed precision the performance was 13ns/day and with the double precision 7ns/day).
When I tested the “big” system (~49000 atoms) though, the results were pretty dissapointing. From 8ns/day in the case of the mixed precision calculations to almost 0.7ns/day in the double precision.

Can this be possible? I tested this performance in two machines I have currenty access (both with NVIDIA M2070 cards) and the results were alsmost the same.

Is it something I haven’t taken in account?

Thank you in advance for your help.

Best,

Manolis.

Στις 2014-09-19 17:04, Steve Plimpton έγραψε:

5

Maybe Trung can comment on performance. I’ve never seen mixed vs double
be that much slower. Are you running on one GPU? What kind of GPU?
Are you using multiple MPI tasks per GPU?

Steve

6

Dear Steve,

I re-installed the Lammps with the GPU package with Double Precision. I did
some benchmark tests, but I took strange results. When I tested for the
"small" system (~20000 atoms) the results, compared to those of mixed
precision, were what I was expected. About 30% down. (For example in the
case of the mixed precision the performance was 13ns/day and with the double
precision 7ns/day).
When I tested the "big" system (~49000 atoms) though, the results were
pretty dissapointing. From 8ns/day in the case of the mixed precision
calculations to almost 0.7ns/day in the double precision.

Can this be possible? I tested this performance in two machines I have
currenty access (both with NVIDIA M2070 cards) and the results were alsmost
the same.

that is difficult to say without knowing the exact details of the
input. it could be something peripheral. you need to carefully study
the timing breakdown to see where the time is really spent.

axel.

7

Dear Steve,

I found my mistake. In the read_restart command (for the “big” system) the restart file I used

was a file which was produced by a Molecular Dynamics simulation on CPU nodes.

When I converted it to a data file everything worked perfect. That is the reason that

in the “small” system the performance was excellent. The restart file i used there was produced

by a previous M.D. simulation on GPU nodes.

I would appreciate it if you explain me why this is happening!

Thank you again for your excellent support!

Best,

Emmanuel Skountzos.

Στις 2014-09-23 16:15, Steve Plimpton έγραψε:

8

Dear Steve,

I found my mistake. In the read_restart command (for the "big" system) the
restart file I used

was a file which was produced by a Molecular Dynamics simulation on CPU
nodes.

When I converted it to a data file everything worked perfect. That is the
reason that

in the "small" system the performance was excellent. The restart file i used
there was produced

by a previous M.D. simulation on GPU nodes.

I would appreciate it if you explain me why this is happening!

with read_restart the original styles are recreated, using the suffix
flag does not apply.

axel.

9

-Trung

10

I.e. when you run a restarted run, you have to
use the command-line -sf switch or suffix command
again to turn on a GPU-enable run.

Steve

11

Dear LAMMPS users,

I post here, to the LAMMPS-users list, in order to share with a you an issue that I noticed

with the binary restart files of LAMMPS and more specifically when I ran an NPT simulation.

I performed a 10step NPT simulation and when the simulation finished I took the last *.rst file

and I used it as initial configuration to a new NPT simulation (with exactly the same parameters).

What I noticed is that the E_long and E_coul energies of the last configuration of the 1st simulation

are not the same to those of the first configuration of the 2nd simulation as they should have been.

While I was searching throught the Internet and reading at the mailing list of LAMMPS whether or not

this issue has already been reported, I found a similar topic:

http://lammps.sandia.gov/threads/msg11699.html

Something similar to my issue has been previously reported for the case of the box/relax fix style.

At the end of this topic, Dr. Harold Hatch mentioned that he had no such problem when he ran an NVT simulation.

Taking this into account I followed the same procedure that I described in the first paragraph just changing the simulation from NPT to NVT.

Indeed, the energies of last configuration of 1st simulation and the fist configuration of the second configuration were precisely the same.

In order you to have a closer look on this issue I also attach 4 files which are the log files after each simulation.

From these files you can also find the parameters that I employed (k_space solver, bond, angle and dihedral

styles, fixes etc) for these simulations:

  1. First NPT log.lammps (log1_NPT.lammps)

  2. Second NPT log.lammps (log2_NPT.lammps)

  3. First NVT log.lammps (log1_NVT.lammps)

  4. Second NVT log.lammps (log2_NVT.lammps)

Is there something that I might be missing?

Thank you in advance for your help,

Best regards,

Emmanuel N. Skountzos.

log1_NPT.lammps (10.7 KB)

log1_NVT.lammps (10.6 KB)

log2_NPT.lammps (10.6 KB)

log2_NVT.lammps (10.6 KB)

12

Dear LAMMPS users,

I post here, to the LAMMPS-users list, in order to share with a you an issue
that I noticed

with the binary restart files of LAMMPS and more specifically when I ran an
NPT simulation.

I performed a 10step NPT simulation and when the simulation finished I took
the last *.rst file

and I used it as initial configuration to a new NPT simulation (with exactly
the same parameters).

What I noticed is that the E_long and E_coul energies of the last
configuration of the 1st simulation

are not the same to those of the first configuration of the 2nd simulation
as they should have been.

While I was searching throught the Internet and reading at the mailing list
of LAMMPS whether or not

this issue has already been reported, I found a similar topic:

LAMMPS Molecular Dynamics Simulator

Something similar to my issue has been previously reported for the case of
the box/relax fix style.

At the end of this topic, Dr. Harold Hatch mentioned that he had no such
problem when he ran an NVT simulation.

Taking this into account I followed the same procedure that I described in
the first paragraph just changing the simulation from NPT to NVT.

Indeed, the energies of last configuration of 1st simulation and the fist
configuration of the second configuration were precisely the same.

In order you to have a closer look on this issue I also attach 4 files which
are the log files after each simulation.

From these files you can also find the parameters that I employed (k_space
solver, bond, angle and dihedral

styles, fixes etc) for these simulations:

1) First NPT log.lammps (log1_NPT.lammps)

2) Second NPT log.lammps (log2_NPT.lammps)

3) First NVT log.lammps (log1_NVT.lammps)

4) Second NVT log.lammps (log2_NVT.lammps)

Is there something that I might be missing?

elementary!

your volume is changing and thus are the automatically chosen kspace
parameters different. since your kspace convergence is rather coarse
at 1.0e-4, this will have a more visible impact. if you set all those
parameters manually in both inputs, the run will continue with the
same numbers from the restart.

axel.

13

In other words, some kspace parameters are only set once at the beginning of the simulation, and they depend on volume.

14

Dear Axel,

thank you for your immediate and very accurate response.

The parameters that you mentioned are the G vector, grid, stencil order, estimated absolute RMS force accuracy

estimated relative force accuracy, right?

I thought that these values were included into the restart file and I didn’t have to give them manually.

And if I want to give them manually I have to do this via the kspace_modify command?

Thanks,

Emmanuel.

Στις 2016-02-05 00:04, Axel Kohlmeyer έγραψε:

15

Dear Stan,

I want also to thank you for your very immediate and clear response.

So, as far as I understand, and please me correct me if I am wrong because I am now reading the theory behind the PPPM method,

is it not possible to attain EXACTLY the same energy (up to the last decimal digit) due to the Coulombic interactions when I want to continue

a simulation from a restart file at an NPT simulation and I am using PPPM as solver?

Thank you in advance,

Emmanuel.

Στις 2016-02-05 00:34, Moore, Stan έγραψε:

16

Dear Stan,

I want also to thank you for your very immediate and clear response.

So, as far as I understand, and please me correct me if I am wrong because I
am now reading the theory behind the PPPM method,

is it not possible to attain EXACTLY the same energy (up to the last
decimal digit) due to the Coulombic interactions when I want to continue

a simulation from a restart file at an NPT simulation and I am using PPPM as
solver?

you are wrong. it *is* possible, but there are some caveats that have
nothing to do with the theory or implementation of the potentials or
PPPM specifically, but the procedure and hardware restrictions.

you are doing computations in using floating point numbers. those have
a varying resolution (depending on magnitude) and floating point math
is not associative (i.e. the exact result depends on the order of
individual operations). with that in mind, you have to factor in that
restarting enforces a neighbor list rebuild and that in turn will
enforce wrapping of atoms into the periodic cell and usually causes a
reshuffle in the order in which forces and energies are accumulated.
this is all well understood and frequently discussed on this and other
mailing lists. of course the same thing will happen, if you use only
cutoffs or potentials with no coulomb at all.

axel

17

This has more to do with the way PPPM is implemented in LAMMPS than PPPM theory. If LAMMPS updated the parameters each time the box dimensions changed, then you could get very close to the exact same energy across restarts. But that is not the case in LAMMPS–it would be expensive. The idea is that if your box dimensions change too much during a simulation, you need to break up your simulation into multiple ones. One thing you could do is estimate the maximum box size and manually set your parameters based on those dimensions. Then energy would be constant over restarts.

18

It’s also a question of what LAMMPS stores in its restart files.
We never thought about the issue you are raising or that
it would be important in an NPT context to preseve
the original PPPM settings and not re-compute them for
the current box, when you restart a run. You’re
the first person to raise this question (that I remember).

I’ll note that if the box size changed by a significant
percentage, then you would want PPPM to recompute
these values to assure the requested accuracy.

One could also argue that even if the box size has barely changed,
then continuing a run with a slightly altered set of PPPM
parameters is acceptable. Both sets of params are giving
you results within the requested error tolerance.

Steve