MPI communicator

Dear LAMMPS developers and users,

Greetings. I am modifying temper.cpp to add a density swap capability in addition to the already-implemented T-swap.

My design for the density-and-temperature parallel tempering works fine now, but it only enables one core per replica.

Now I would like to use more cores per replica, and my idea is to register a “global” table of domain coordinate.

Say, I have 4 densities and 6 temperatures, so there are 24 states. I would like to run each state on 4 cores, so totally I need 464=96 cores.

I would like collect all the current domain->boxlo and domain->boxhi and register them onto each cores…

(Namely, each core has a table of 96 set of domain coordinates)

So, which MPI communicator should I use in such a case? I tried communicator roots but it does not seem right…

Your help is greatly appreciated.

LC Liu

Dear LAMMPS developers and users,

Greetings. I am modifying temper.cpp to add a density swap capability in
addition to the already-implemented T-swap.

My design for the density-and-temperature parallel tempering works fine now,
but it only enables one core per replica.

Now I would like to use more cores per replica, and my idea is to register a
"global" table of domain coordinate.

huh? why is that necessary?

Say, I have 4 densities and 6 temperatures, so there are 24 states. I would
like to run each state on 4 cores, so totally I need 4*6*4=96 cores.

I would like collect all the current domain->boxlo and domain->boxhi and
register them onto each cores....

why not follow the example of the temper code and only maintain
the "global" information on the rank=0 task for each partition?
and then use the "roots" communicator to communicate the global
state between partitions and then the corresponding "world" communicators
to communicate inside the "world" (ie. the multiple tasks per replica).

like this:
  set_temp = new double[nworlds];
  if (me == 0) MPI_Allgather(&temp,1,MPI_DOUBLE,set_temp,1,MPI_DOUBLE,roots);
  MPI_Bcast(set_temp,nworlds,MPI_DOUBLE,0,world);

(Namely, each core has a table of 96 set of domain coordinates)

that seems to me like a bad idea. it is best to organize communication
"between" replica and "inside" a replica as separate entities. this way
you maintain independence and reduce synchronization overhead.

So, which MPI communicator should I use in such a case? I tried communicator
roots but it does not seem right....

i suggest you spend a little more time reading the code and thinking
about how multi-partition runs are organized in LAMMPS.

that being said, the global, across-all-cores communicator is
- of course - in universe->uworld. it is a bit surprising to me how
you can modify the code in temper.cpp, know about the "roots"
communicator and not see this?? it is, like, right in your face, is it not?

axel.

Hi, Axel,

Thank you so much for the time and effort. I followed your advice and modified the code. It runs successfully.

Please bear with me for the long discussion below since I think it might be useful to other folks in the community.

To realize a v-t tempering as used in Paschek, PRL 94, 217802 (2005) and Pashek et al., ChemPhysChem 2008, 9, 2737 – 2741, I took the following approaches (in modifying temper.cpp):

  1. prepare the 2D matrix of temperature and density. Align them into a 1D array.

For example, I have 4 T’s and 3 rho’s, and that sum to 12 states. States are numbered as 0,1,2,3,4,5,6,7,8,9,10,11, where elements in the brackets [0,1,2,3], [4,5,6,7], [8,9,10,11] are of the same density.[0,4,8], [1,5,9], [2,6,10], and [3,7,11] are of the same temperatures.

Hence, elements in [0,1,2,3] do the t-swap, and elements in [0,4,8] do the v-swap, and etc.

  1. in the original t-swap code segment: add a line to disable the t-swap if the two neighbor states are of different densities, say, state 3 and 4.

  2. add a piece of code to do the v-swap. In Pashek 2008, they do v-swap attempts after 10 t-swap attempts, so I follow their method and the pseudo-code reads as:

for (int iswap = 0; iswap < nswaps; iswap++)
{
// run for nevery timesteps
update->integrate->run(nevery);

if(iswap % 10 != 9) // T tempering-------------------------------------
{

<>
}
else
{
<< V swap>>
}

// root procs update their value if swap took place
// allgather across root procs
// bcast within my world
if (me == 0)
{
MPI_Allgather(&my_set_temp,1,MPI_INT,world2temp,1,MPI_INT,roots);
for (i = 0; i < nworlds; i++) temp2world[world2temp[i]] = i;
}
MPI_Bcast(temp2world,nworlds,MPI_INT,0,world);

  1. the T swap segment is pretty much the same, except adding the line (mentioned in 2.) to prevent from swap between different densities.

The V-swap need some deliberations.

a. the swap pairs are not neighbors but every 4 states (in my case above).

b. atoms are retained in each replica, and the densities are exchanged through expanding/changing the box sizes.

Hence, at the beginning of v-swap, I collect current box dimensions of each world (replica) and store them in all the rank0 processors.

If the v-swap attempt succeeds, atom coordinates is so modified to exchange the volumes.

(I simulate rigid model water, so only the O-O coordinates are changed. O-H bonds remain the same)

The box dimension is also set to partner’s dimension.

The following piece of code is added to reflect the box change: (taken from change_box.cpp)

// change_box routines
domain->set_initial_box();
domain->set_global_box();
domain->set_local_box();
if (domain->nonperiodic == 2) domain->reset_box();

  1. The states swap history can be seen in the output file. Post-processing is the same as the original one.

Axel and Steve, thank you so much for the help on giving advice and tolerated with my stupid questions.

I had zero experience in c++ and MPI programming before this, and it is truly a great learning process!

Happy holidays!

LC Liu