Hi all,
I would like to know how exactly the processors are mapped to the domain
in LAMMPS - Lets say the origin of the rectangular domain is (-L, -L,
-L) and the opposite corner is (L, L, L) and we have a Cartesian
processor topology (NPx, NPy, NPz). Is it always true that the (0, 0, 0)
Cartesian processor will me mapped to the origin of the domain?
Using an example to be more clear, assuming "xyz" labelling for the
processors and having a 2x2x1 grid, is the mapping always done like in
picture (A)?
(A)
--------------(L,L)
Hi all,
I would like to know how exactly the processors are mapped to the domain
in LAMMPS - Lets say the origin of the rectangular domain is (-L, -L,
-L) and the opposite corner is (L, L, L) and we have a Cartesian
processor topology (NPx, NPy, NPz). Is it always true that the (0, 0, 0)
Cartesian processor will me mapped to the origin of the domain?
do you mean a cubic domain? or a generic rectangular domain?
please see the documentation for the "processors" command to learn how
LAMMPS subdivides domains. please also note that MPI just provides the
total number of processors. any specific ordering of processors will have
to be explicitly specified. things get even more complex with load
balancing and/or comm_style rcb instead of the default (brick).
Using an example to be more clear, assuming "xyz" labelling for the
processors and having a 2x2x1 grid, is the mapping always done like in
picture (A)?
(A)
--------------(L,L)
>(0,1)|(1,1) |
> > >
>------------|
>(0,0)|(1,0) |
> > >
--------------
(-L,-L)
(B)
--------------(L,L)
>(1,0)|(0,0) |
> > >
>------------|
>(1,1)|(0,1) |
> > >
--------------
(-L,-L)
Also which variables/data structure in LAMMPS' code holds the
information of the domain boundaries of each processor (disregarding
halos).
please read the domain.h file
axel.
Thanks Axel,
I meant a generic rectangular domain. I would like to know if the
(0,0,0) processor is always mapped to the (x0,y0,z0) corner of the
domain, that is, the one with the lowest coordinates for x, y and z.
Then the processors would be mapped for a certain coordinate x € [x,y,z]
to region {[proc_x * Lx/nPx + x0] , [proc_x * Lx/nPx + x0 + Lx/nPx]}
where proc_x is the cartesian coordinate of the processor in x, Lx is
the length of the domain in x direction and nPx is the number of
processors in x direction. Is that how the decomposition occurs in
LAMMPS? (Assuming for a certain direction the mapping is homogeneous,
like in a structured grid).
Regards,
Eduardo
Thanks Axel,
I meant a generic rectangular domain. I would like to know if the
(0,0,0) processor is always mapped to the (x0,y0,z0) corner of the
domain, that is, the one with the lowest coordinates for x, y and z.
Then the processors would be mapped for a certain coordinate x € [x,y,z]
to region {[proc_x * Lx/nPx + x0] , [proc_x * Lx/nPx + x0 + Lx/nPx]}
where proc_x is the cartesian coordinate of the processor in x, Lx is
the length of the domain in x direction and nPx is the number of
processors in x direction. Is that how the decomposition occurs in
LAMMPS? (Assuming for a certain direction the mapping is homogeneous,
like in a structured grid).
always? no!
you obviously have not read the documentation of the processors command.
please do so.
axel.
Hi Axel,
I did in fact. I am assuming "xyz" order as I said in my previous
message, with a regular grid of processors (spacing is the same for a
certain direction). No load balancing, no fancy stuff. Maybe I am
reading terribly wrong but I cannot extract the piece of information I
am asking for from the documentation.
So assuming the topology I state, does the (0,0,0) processors is always
mapped to the corner of the domain with the lowest coordinates? - I am
not sure if it can happen that as you increase the Cartesian coordinate
of the processors for a certain direction, the domain coordinate can
decrease due to the mapping or it will always increase as well (picture
(B) and (A) respectively in my first email)
Sorry if I am not clear enough. I am trying hard to be!
Regards,
Eduardo
Hi Axel,
I did in fact. I am assuming "xyz" order as I said in my previous
message, with a regular grid of processors (spacing is the same for a
certain direction). No load balancing, no fancy stuff. Maybe I am
reading terribly wrong but I cannot extract the piece of information I
am asking for from the documentation.
So assuming the topology I state, does the (0,0,0) processors is always
mapped to the corner of the domain with the lowest coordinates? - I am
not sure if it can happen that as you increase the Cartesian coordinate
of the processors for a certain direction, the domain coordinate can
decrease due to the mapping or it will always increase as well (picture
(B) and (A) respectively in my first email)
Sorry if I am not clear enough. I am trying hard to be!
perhaps it is better that you explain what you need this for.
there is a general problem with what you are asking about, because code in
LAMMPS *should* not make any assumptions about a specific processor
assignment and layout outside of the Domain and the Comm classes and their
functions and member variables.
you have the information about the subdomain of a given processor and
methods to communicate with neighboring subdomains.
any code that makes assumptions beyond this, will just limit its
applicability, and there would have to be a *very* good reason to do that.
axel.
So this is all about coupling LAMMPS with a CFD software using domain
decomposition for doing concurrent MD-CFD hybrid simulations. The idea
is that part of the domain is simulated by the CFD code and part by the
MD and they overlap in a certain region. Attached there are two images
to help you visualize why this uncertainty about mapping is important to
me. In the overlap region, processors from the CFD are mapped (they are
put into a new communicator) to the ones overlapping them on the MD
side.
Asumme "xyz" ordering, no load balance (processor topology wont change
during running), regular spacing between processes in a certain
direction.
I would like to have the case depicted in the first image of the
document. I mean, assume always that (x0,y0,z0), the corner with the
lowest domain coordinates, is mapped to the (0,0,0) processor and then
as the cartesian coordinates of the processors grow the domain
coordinates do so as well. How can I enforce this?
The second image shows a mapping where the (0,0,0) processor is mapped
to the corner with the highest domain coordinates and I am worried about
if this can happen.
Regards,
Eduardo
figure4axel.pdf (24.1 KB)
I don’t understand your diagram, but see the section
of the processors command doc page on the “map”
keyword. It explains that the default is map cart
which lets the MPI_Cart_create() method map the
MPI tasks to the logical 3d grid. I think typically
that will be the same as “xyz”. Xyz will map
the proc IDs with 0 in the lower left corner, then
increase in x coord to Px-1, then Px is above 0, etc.
X varies fastest, then Y, then Z.
Does that answer your Q?
Steve
Hi Steve,
Thanks for the interest. What you describe is how processors ranks are
mapped to Cartesian coordinates in a Cartesian communicator. But I am
asking here about mapping between logical Cartesian processor
coordinates with domain physical ones and if there is a way to enforce
processor (0,0,0) in "xyz" ordering, to be mapped to the (x0,y0,z0)- the
corner with the lowest domain coordinates - and ensure that as the
Cartesian coordinate of a a processor increases the domain coordinate
will increase as well. So for example a box defined by a region [-10 10
-10 10 -10 10] in the format [xlo xhi ylo yhi zlo zhi], for a 2x2x1
decomposition of processors the mapping I would like to enforce would
be:
(0,0,0) - [-10 0 -10 0 -10 10]
(1,0,0) - [0 10 0 -10 -10 -10]
(0,1,0) - [-10 0 0 10 -10 -10]
(1,1,0) - [0 10 0 10 -10 -10]
So as you see as the processor coordinate grows, the domain coordinate
grows too. What is not stated in the documentation is that if this is
always the case or one can have (0,0,0) at the corner with the highest
coordinates and then as the the processor coordinate grows, the domain
coordinate decrease as in the case below.
(0,0,0) - [0 10 0 10 -10 -10]
(1,0,0) - [-10 0 0 10 -10 -10]
(0,1,0) - [0 10 0 -10 -10 -10]
(1,1,0) - [-10 0 -10 0 -10 10]
Is it more clear now?
Thanks for your time.
Eduardo
Hi Steve,
Thanks for the interest. What you describe is how processors ranks are
mapped to Cartesian coordinates in a Cartesian communicator. But I am
asking here about mapping between logical Cartesian processor
coordinates with domain physical ones and if there is a way to enforce
processor (0,0,0) in "xyz" ordering, to be mapped to the (x0,y0,z0)- the
corner with the lowest domain coordinates - and ensure that as the
Cartesian coordinate of a a processor increases the domain coordinate
will increase as well. So for example a box defined by a region [-10 10
-10 10 -10 10] in the format [xlo xhi ylo yhi zlo zhi], for a 2x2x1
decomposition of processors the mapping I would like to enforce would
be:
(0,0,0) - [-10 0 -10 0 -10 10]
(1,0,0) - [0 10 0 -10 -10 -10]
(0,1,0) - [-10 0 0 10 -10 -10]
(1,1,0) - [0 10 0 10 -10 -10]
So as you see as the processor coordinate grows, the domain coordinate
grows too. What is not stated in the documentation is that if this is
always the case or one can have (0,0,0) at the corner with the highest
coordinates and then as the the processor coordinate grows, the domain
coordinate decrease as in the case below.
(0,0,0) - [0 10 0 10 -10 -10]
(1,0,0) - [-10 0 0 10 -10 -10]
(0,1,0) - [0 10 0 -10 -10 -10]
(1,1,0) - [-10 0 -10 0 -10 10]
Is it more clear now?
processors in MPI (and thus in LAMMPS) are not incremented on a 3-d grid,
but as a linear rank id. as steve mentioned, a specific geometric relation
between ranks (e.g. for optimizing communication patterns to physical
layout of nodes in a parallel computer) can only be enforced using such
custom created communicators (which essentially change the ordering of the
ranks).
in any case, it seems to me, that you are addressing the issue in the wrong
way. why even worry about enforcing a particular MPI rank to geometry
relation? why not infer the section of your grid that a particular MPI rank
is responsible for by looking at its subdomain geometry (sublo[], subhi[]
in domain.h)?
this way, you can have any kind of weird and mapping and domain subdivision
(cf. comm_style rcb) and still have always correctly working code.
axel.
yes, for xyz ordering the mapping of MPI ranks to the
physical domain always goes from lo to hi, whether in x,y, or z.
Again this is for a logical onelevl 3d grid of procs, which is not always
what LAMMPS uses (load balancing, other processors command
options like twolevel or numa).
I agree with Axel, that more generally you should probably have
your code query the subdomain bounds of each MPI task.
Steve