pair_kim needs to use multiple neighbor lists

Hello,

I am working on an update to pair_kim that is compatible with the forthcoming kim-api-v2 package.

So far I have this

https://github.com/ellio167/lammps/commit/3549420f7a725a84df00d2477ecaa3b6969d8c7f

which is working with the latest development version of kim-api-v2.

However, there are two neighbor list related issues I need help with.

(1) kim-api-v2 requires full neighbor lists with all neighbors:

   local-local
   local-ghost
   ghost-local
   ghost-ghost

What should be done here:

https://github.com/ellio167/lammps/blob/3549420f7a725a84df00d2477ecaa3b6969d8c7f/src/KIM/pair_kim.cpp#L378

to make this work?

Does this have an impact on how/if pair_kim supports the "newton" option?

(2) In addition to the above, the kim-api-v2 supports models that require an arbitrary (but finite) number of neighbor lists (with different (or possibly equal) cutoff distances).

How can this best be handled in LAMMPS?

Thanks!

Ryan Elliott

Hello,

I am working on an update to pair_kim that is compatible with the forthcoming
kim-api-v2 package.

So far I have this

https://github.com/ellio167/lammps/commit/3549420f7a725a84df00d2477ecaa3b6969d8c7f

which is working with the latest development version of kim-api-v2.

However, there are two neighbor list related issues I need help with.

(1) kim-api-v2 requires full neighbor lists with all neighbors:

local-local
local-ghost
ghost-local
ghost-ghost

What should be done here:

https://github.com/ellio167/lammps/blob/3549420f7a725a84df00d2477ecaa3b6969d8c7f/src/KIM/pair_kim.cpp#L378

to make this work?

​if you need neighbors of ghosts, you need to do what is done in pair_airebo.cpp:

// need a full neighbor list, including neighbors of ghosts

int irequest = neighbor->request(this,instance_me);
neighbor->requests[irequest]->half = 0;
neighbor->requests[irequest]->full = 1;
neighbor->requests[irequest]->ghost = 1;

from that you can build sub-neighbor lists, if needed (like some manybody potentials do for implicitly bonded interactions).
you have to make certain, that your communication cutoff is large enough, as the ghost atoms furthers away from the subdomain of the local atoms will not have all their neighbors included, as they are not included in the ghost atoms. by default it is only guaranteed to have all neighbors of local atoms included within global pair cutoff plus skin.

Does this have an impact on how/if pair_kim supports the “newton” option?

​a full neighbor list has all pairs twice and thus is not affected by force->newton_pair.

(2) In addition to the above, the kim-api-v2 supports models that require an
arbitrary (but finite) number of neighbor lists (with different (or possibly
equal) cutoff distances).

How can this best be handled in LAMMPS?

​by post-processing the master neighbor list into custom sublists. see. pair_airebo.cpp and others. in some cases, this sublist can be conveniently produced while computing the pair-wise additive part of a potential​. check out pair_vashishta.cpp

axel.

Thanks Axel,

More below...

(2) In addition to the above, the kim-api-v2 supports models that require an arbitrary (but finite) number of neighbor lists (with different (or possibly equal) cutoff distances).

How can this best be handled in LAMMPS?

​by post-processing the master neighbor list into custom sublists. see. pair_airebo.cpp and others. in some cases, this sublist can be conveniently produced while computing the pair-wise additive part of a potential​. check out pair_vashishta.cpp

I looked at pair_airebo, and that seems do-able...

But, is there no way to know when the neighbor list is updated so that the sublist's don't have to be computed every time that compute() is called?

What is the

int irequest = neighbor->request(this,instance_me);

all about? Before your previous answer, I had guessed that it was a mechanism for pair styles to request multiple neighbor lists...

Thanks as always for the expert and fast responses!

Ryan

Thanks Axel,

More below…

(2) In addition to the above, the kim-api-v2 supports models that require an
arbitrary (but finite) number of neighbor lists (with different (or possibly
equal) cutoff distances).

How can this best be handled in LAMMPS?

​by post-processing the master neighbor list into custom sublists. see.
pair_airebo.cpp and others. in some cases, this sublist can be conveniently
produced while computing the pair-wise additive part of a potential​. check
out pair_vashishta.cpp

I looked at pair_airebo, and that seems do-able…

But, is there no way to know when the neighbor list is updated so that the
sublist’s don’t have to be computed every time that compute() is called?

​there is a way.

but first off, please keep in mind, that the REBO list has a very short cutoff and thus must be rebuilt every time, but the savings due to the multiple nested loops over the REBO list instead of the full list are enormous. for vashishta, the short list is build inside the non-bonded pair wise loop.

please also keep in mind, that for neighbor (sub)lists with a fixed number of entries (e.g. the N closest atoms) will have to be rebuilt after every position update as well.

the “age” of the (perpetual) neighbor lists is stored in lmp->neighbor->ago. if this counter is 0, the neighbor lists were updated. please see Neighbor::decide() and Neighbor::build() in the file neighbor.cpp

What is the

int irequest = neighbor->request(this,instance_me);

all about? Before your previous answer, I had guessed that it was a mechanism
for pair styles to request multiple neighbor lists…

​it is. but most important​ is that a neighbor request is created which includes “this” and “instance_me” to be able to identify where a request is coming from. LAMMPS is internally trying to alias multiple neighbor list requests together, if they have the same parameters, and also has mechanisms, to e.g. create a half neighbor list from a full list.
irequest is thus the index into the list of requests and required to manipulate the settings, i.e. change from half to full or include neighbors of ghosts and other settings.

you can indeed request multiple neighbor lists inside the same style. see, e.g. pair_meam_c.cpp
​but you have to set a unique id, since the neighbor list code, only looks at “this” otherwise.

// need full and half neighbor list

int irequest_full = neighbor->request(this,instance_me);
neighbor->requests[irequest_full]->id = 1;
neighbor->requests[irequest_full]->half = 0;
neighbor->requests[irequest_full]->full = 1;
int irequest_half = neighbor->request(this,instance_me);
neighbor->requests[irequest_half]->id = 2;

it is still probably cleaner, if you have the sublist creation provided inside the KIM API library, as this allows you more flexibility, keeps a cleaner interface and there is not much time to be saved there, as the major cost of the neighbor list build, is binning and then processing of the most inclusive list.

axel.​

Hi Axel,

Thanks for your response. I'll talk a look at pair_meam_c and consider the possibilities.

You say that the sublist build time is small compared to the most inclusive list build time. That seems reasonable, but I wonder how it would compare to the "typical" compute() time (of course I know compute() time varies dramatically depending on the pair style.) Since, as you suggest, the sublist build would be done for every compute() call, it seems that this would be an important comparison to understand. Let's take AIREBO as an example. Do you have a sense of what percent of total compute() time for AIREBO is associated with its sublist generation?

Now a somewhat different question; For this question, let's simplify and just assume that a KIM model only requests one neighbor list.

I know that in init_one() the return value needs to be a cutoff distance that is large enough to ensure that all necessary ghost atoms are generated. In KIM we call this the "influence distance".

A KIM model is able to request the neighbor list with a cutoff that is less than or equal to its "influence distance". So, I'm wondering how I can tell lammps what neighbor list cutoff distance the model needs (in particular for the case where the cutoff is less than the influence distance)?

Thanks,

Ryan

​to assess the cost of neighbor list builds​ and sublist/skip list builds, it is better to look at the “melt” example (with a few tweaks: longer cutoff 4 sigma, more frequent neighbor list builds). also i have 8 different atom types (but with all the same parameters). and now i compare performance against using 8 substyle hybrid with plain lj/cut. input is below.

for the plain style, i have about 31% time spent on neighborlist (regular melt has 10-12%)

Loop time of 19.6678 on 1 procs for 2500 steps with 4000 atoms

Performance: 54912.006 tau/day, 127.111 timesteps/s
99.9% CPU use with 1 MPI tasks x no OpenMP threads

MPI task timing breakdown:
Section | min time | avg time | max time |%varavg| %total

Hi Ryan - all of Axel’s advice is good. I have a separate Q/comment on these

points:

(1) kim-api-v2 requires full neighbor lists with all neighbors:
local-local
local-ghost
ghost-local
ghost-ghost

Does this mean all KIM potentials require all of this info,

or some will want it?

It seems overkill for many potentials.

Does this have an impact on how/if pair_kim supports the “newton” option?

As Axel said, a full neighbor list has all pairs twice, so you have

all the info you need to compute whatever you wish inside KIM.

However, the forces you return are handled differently by LAMMPS depending

on newton on vs off. If “on”, then LAMMPS will assume there are

forces on ghost atoms and do 2 things: reverse comm to sum ghost

forces to owned-atom forces. And do a cheaper virial computation

which uses the ghost forces (doesn’t need to worry about periodic BC).

If “off” then LAMMPS will assume each owned atom already
has its total force and do no comm.

So whatever KIM computes for forces would need to match that.

If KIM will always return total force on each owned atoms
(and 0.0 on the ghost atoms), then the extra comm for newton on

will not hurt I guess.

Steve

Hi Steve,

Thanks for your response. See below.

(1) kim-api-v2 requires full neighbor lists with all neighbors:

local-local
       local-ghost
       ghost-local
       ghost-ghost

Does this mean all KIM potentials require all of this info,
or some will want it?

It seems overkill for many potentials.

Yes, all KIM potentials will require this type of neighbor list.

For simplicity, we decided that only this sort of neighbor list would be supported in kim-api-v2. We believe that the overhead incurred by this choice is minimal, except maybe for very simple potentials. The worst case, I think, is something like LJ where this overhead consists of:

(1) Extra time to generate the full-list with all l-l, l-g, g-l, g-g neighbors; as opposed to the minimal necessary half-list with only l-l, l-g neighbors. Since the neighbor list is only updated periodically, this overhead is amortized across multiple compute() calls.

(2) Extra time in the main LJ loops, but this can be mitigated to an extent by making the loops look like (in pseudo code):

    for (i=0, i < n_local, ++i) {
      for (jj=0, jj< n_neigh_i; ++jj) {
        j = neigh_i[jj];
        if (i < j) {
          phi += energy contrib. of i,j pair
          atom->f[i][:] += force contrib. of i,j pair
          atom->f[j][:] -= force contrib. of i,j pair
        }
      }
    }

Does this have an impact on how/if pair_kim supports the "newton" option?

As Axel said, a full neighbor list has all pairs twice, so you have
all the info you need to compute whatever you wish inside KIM.

However, the forces you return are handled differently by LAMMPS depending
on newton on vs off. If "on", then LAMMPS will assume there are
forces on ghost atoms and do 2 things: reverse comm to sum ghost
forces to owned-atom forces. And do a cheaper virial computation
which uses the ghost forces (doesn't need to worry about periodic BC).

If "off" then LAMMPS will assume each owned atom already
has its total force and do no comm.

So whatever KIM computes for forces would need to match that.
If KIM will always return total force on each owned atoms
(and 0.0 on the ghost atoms), then the extra comm for newton on
will not hurt I guess.

OK, thanks for the clarification. We had already grappled with this in the existing pair_kim code and the way forces are computed has not changed in kim-api-v2. So, I think I've preserved working behavior for the forces...

Any further comments/suggestions are always welcome.

Thanks,

Ryan

Hi Steve,

I wanted to make an update to this response. As we have been working on the updated pair_kim code and converting our openkim.org content to the kim-api-v2 package, I've released that there is a nice way that we can have KIM models provide hints to the Simulator about how it will use neighbor lists. This will allow the simulator to optimize its neighbor list build, if it can.

So, for those models that provide hints that they do not use neighbors-of-ghosts and that they only use neighbors with j>i (like above), then pair_kim in lammps will be able to use standard half-lists (ghost = 0).

This will be a nice optimization!

Let me know if you have any questions!

Cheers,

Ryan