Hi all,

I would like to know how exactly the processors are mapped to the domain

in LAMMPS - Lets say the origin of the rectangular domain is (-L, -L,

-L) and the opposite corner is (L, L, L) and we have a Cartesian

processor topology (NPx, NPy, NPz). Is it always true that the (0, 0, 0)

Cartesian processor will me mapped to the origin of the domain?

Using an example to be more clear, assuming "xyz" labelling for the

processors and having a 2x2x1 grid, is the mapping always done like in

picture (A)?

(A)

--------------(L,L)

Hi all,

I would like to know how exactly the processors are mapped to the domain

in LAMMPS - Lets say the origin of the rectangular domain is (-L, -L,

-L) and the opposite corner is (L, L, L) and we have a Cartesian

processor topology (NPx, NPy, NPz). Is it always true that the (0, 0, 0)

Cartesian processor will me mapped to the origin of the domain?

do you mean a cubic domain? or a generic rectangular domain?

please see the documentation for the "processors" command to learn how

LAMMPS subdivides domains. please also note that MPI just provides the

total number of processors. any specific ordering of processors will have

to be explicitly specified. things get even more complex with load

balancing and/or comm_style rcb instead of the default (brick).

Using an example to be more clear, assuming "xyz" labelling for the

processors and having a 2x2x1 grid, is the mapping always done like in

picture (A)?

(A)

--------------(L,L)

>(0,1)|(1,1) |

> > >

>------------|

>(0,0)|(1,0) |

> > >

--------------

(-L,-L)

(B)

--------------(L,L)

>(1,0)|(0,0) |

> > >

>------------|

>(1,1)|(0,1) |

> > >

--------------

(-L,-L)

Also which variables/data structure in LAMMPS' code holds the

information of the domain boundaries of each processor (disregarding

halos).

please read the domain.h file

axel.

Thanks Axel,

I meant a generic rectangular domain. I would like to know if the

(0,0,0) processor is always mapped to the (x0,y0,z0) corner of the

domain, that is, the one with the lowest coordinates for x, y and z.

Then the processors would be mapped for a certain coordinate x € [x,y,z]

to region {[proc_x * Lx/nPx + x0] , [proc_x * Lx/nPx + x0 + Lx/nPx]}

where proc_x is the cartesian coordinate of the processor in x, Lx is

the length of the domain in x direction and nPx is the number of

processors in x direction. Is that how the decomposition occurs in

LAMMPS? (Assuming for a certain direction the mapping is homogeneous,

like in a structured grid).

Regards,

Eduardo

Thanks Axel,

I meant a generic rectangular domain. I would like to know if the

(0,0,0) processor is always mapped to the (x0,y0,z0) corner of the

domain, that is, the one with the lowest coordinates for x, y and z.

Then the processors would be mapped for a certain coordinate x € [x,y,z]

to region {[proc_x * Lx/nPx + x0] , [proc_x * Lx/nPx + x0 + Lx/nPx]}

where proc_x is the cartesian coordinate of the processor in x, Lx is

the length of the domain in x direction and nPx is the number of

processors in x direction. Is that how the decomposition occurs in

LAMMPS? (Assuming for a certain direction the mapping is homogeneous,

like in a structured grid).

always? no!

you obviously have not read the documentation of the processors command.

please do so.

axel.

Hi Axel,

I did in fact. I am assuming "xyz" order as I said in my previous

message, with a regular grid of processors (spacing is the same for a

certain direction). No load balancing, no fancy stuff. Maybe I am

reading terribly wrong but I cannot extract the piece of information I

am asking for from the documentation.

So assuming the topology I state, does the (0,0,0) processors is always

mapped to the corner of the domain with the lowest coordinates? - I am

not sure if it can happen that as you increase the Cartesian coordinate

of the processors for a certain direction, the domain coordinate can

decrease due to the mapping or it will always increase as well (picture

(B) and (A) respectively in my first email)

Sorry if I am not clear enough. I am trying hard to be!

Regards,

Eduardo

Hi Axel,

I did in fact. I am assuming "xyz" order as I said in my previous

message, with a regular grid of processors (spacing is the same for a

certain direction). No load balancing, no fancy stuff. Maybe I am

reading terribly wrong but I cannot extract the piece of information I

am asking for from the documentation.

So assuming the topology I state, does the (0,0,0) processors is always

mapped to the corner of the domain with the lowest coordinates? - I am

not sure if it can happen that as you increase the Cartesian coordinate

of the processors for a certain direction, the domain coordinate can

decrease due to the mapping or it will always increase as well (picture

(B) and (A) respectively in my first email)

Sorry if I am not clear enough. I am trying hard to be!

perhaps it is better that you explain what you need this for.

there is a general problem with what you are asking about, because code in

LAMMPS *should* not make any assumptions about a specific processor

assignment and layout outside of the Domain and the Comm classes and their

functions and member variables.

you have the information about the subdomain of a given processor and

methods to communicate with neighboring subdomains.

any code that makes assumptions beyond this, will just limit its

applicability, and there would have to be a *very* good reason to do that.

axel.

So this is all about coupling LAMMPS with a CFD software using domain

decomposition for doing concurrent MD-CFD hybrid simulations. The idea

is that part of the domain is simulated by the CFD code and part by the

MD and they overlap in a certain region. Attached there are two images

to help you visualize why this uncertainty about mapping is important to

me. In the overlap region, processors from the CFD are mapped (they are

put into a new communicator) to the ones overlapping them on the MD

side.

Asumme "xyz" ordering, no load balance (processor topology wont change

during running), regular spacing between processes in a certain

direction.

I would like to have the case depicted in the first image of the

document. I mean, assume always that (x0,y0,z0), the corner with the

lowest domain coordinates, is mapped to the (0,0,0) processor and then

as the cartesian coordinates of the processors grow the domain

coordinates do so as well. How can I enforce this?

The second image shows a mapping where the (0,0,0) processor is mapped

to the corner with the highest domain coordinates and I am worried about

if this can happen.

Regards,

Eduardo

figure4axel.pdf (24.1 KB)

I don’t understand your diagram, but see the section

of the processors command doc page on the “map”

keyword. It explains that the default is map cart

which lets the MPI_Cart_create() method map the

MPI tasks to the logical 3d grid. I think typically

that will be the same as “xyz”. Xyz will map

the proc IDs with 0 in the lower left corner, then

increase in x coord to Px-1, then Px is above 0, etc.

X varies fastest, then Y, then Z.

Does that answer your Q?

Steve

Hi Steve,

Thanks for the interest. What you describe is how processors ranks are

mapped to Cartesian coordinates in a Cartesian communicator. But I am

asking here about mapping between logical Cartesian processor

coordinates with domain physical ones and if there is a way to enforce

processor (0,0,0) in "xyz" ordering, to be mapped to the (x0,y0,z0)- the

corner with the lowest domain coordinates - and ensure that as the

Cartesian coordinate of a a processor increases the domain coordinate

will increase as well. So for example a box defined by a region [-10 10

-10 10 -10 10] in the format [xlo xhi ylo yhi zlo zhi], for a 2x2x1

decomposition of processors the mapping I would like to enforce would

be:

(0,0,0) - [-10 0 -10 0 -10 10]

(1,0,0) - [0 10 0 -10 -10 -10]

(0,1,0) - [-10 0 0 10 -10 -10]

(1,1,0) - [0 10 0 10 -10 -10]

So as you see as the processor coordinate grows, the domain coordinate

grows too. What is not stated in the documentation is that if this is

always the case or one can have (0,0,0) at the corner with the highest

coordinates and then as the the processor coordinate grows, the domain

coordinate decrease as in the case below.

(0,0,0) - [0 10 0 10 -10 -10]

(1,0,0) - [-10 0 0 10 -10 -10]

(0,1,0) - [0 10 0 -10 -10 -10]

(1,1,0) - [-10 0 -10 0 -10 10]

Is it more clear now?

Thanks for your time.

Eduardo

Hi Steve,

Thanks for the interest. What you describe is how processors ranks are

mapped to Cartesian coordinates in a Cartesian communicator. But I am

asking here about mapping between logical Cartesian processor

coordinates with domain physical ones and if there is a way to enforce

processor (0,0,0) in "xyz" ordering, to be mapped to the (x0,y0,z0)- the

corner with the lowest domain coordinates - and ensure that as the

Cartesian coordinate of a a processor increases the domain coordinate

will increase as well. So for example a box defined by a region [-10 10

-10 10 -10 10] in the format [xlo xhi ylo yhi zlo zhi], for a 2x2x1

decomposition of processors the mapping I would like to enforce would

be:

(0,0,0) - [-10 0 -10 0 -10 10]

(1,0,0) - [0 10 0 -10 -10 -10]

(0,1,0) - [-10 0 0 10 -10 -10]

(1,1,0) - [0 10 0 10 -10 -10]

So as you see as the processor coordinate grows, the domain coordinate

grows too. What is not stated in the documentation is that if this is

always the case or one can have (0,0,0) at the corner with the highest

coordinates and then as the the processor coordinate grows, the domain

coordinate decrease as in the case below.

(0,0,0) - [0 10 0 10 -10 -10]

(1,0,0) - [-10 0 0 10 -10 -10]

(0,1,0) - [0 10 0 -10 -10 -10]

(1,1,0) - [-10 0 -10 0 -10 10]

Is it more clear now?

processors in MPI (and thus in LAMMPS) are not incremented on a 3-d grid,

but as a linear rank id. as steve mentioned, a specific geometric relation

between ranks (e.g. for optimizing communication patterns to physical

layout of nodes in a parallel computer) can only be enforced using such

custom created communicators (which essentially change the ordering of the

ranks).

in any case, it seems to me, that you are addressing the issue in the wrong

way. why even worry about enforcing a particular MPI rank to geometry

relation? why not infer the section of your grid that a particular MPI rank

is responsible for by looking at its subdomain geometry (sublo[], subhi[]

in domain.h)?

this way, you can have any kind of weird and mapping and domain subdivision

(cf. comm_style rcb) and still have always correctly working code.

axel.

yes, for xyz ordering the mapping of MPI ranks to the

physical domain always goes from lo to hi, whether in x,y, or z.

Again this is for a logical onelevl 3d grid of procs, which is not always

what LAMMPS uses (load balancing, other processors command

options like twolevel or numa).

I agree with Axel, that more generally you should probably have

your code query the subdomain bounds of each MPI task.

Steve