LAMMPS python interface and MPI library symbol resolution order

Having (for the moment) worked around my lammps/numpy array issues, I have a perhaps subtle question about symbol resolution.

I have a python script that uses mpi4py, and each MPI process is supposed to start a _serial_ lammps process. I’ve compiled the lammps shared library in serial (using the default Makefile.serial), with mpi-stubs, and indeed when I do “nm” on the .so I see the MPI routines with ’T’. When I run this python script truly in serial, without mpi4py imported, it seems to do what I expect - the lammps runs in serial, and indeed invokes the lammps MPI stubs. However, when I import mpi4py (before or after I import lammps), I get an MPI error when I try to start the LAMMPS process:

[tin:19579] *** An error occurred in MPI_Comm_rank
[tin:19579] *** reported by process [3967680513,0]
[tin:19579] *** on communicator MPI_COMM_WORLD
[tin:19579] *** MPI_ERR_COMM: invalid communicator
[tin:19579] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[tin:19579] *** and potentially your MPI job)

and from what I can tell, the mpi-stubs routine isn’t being called (although I’m not 100% sure of this because I just checked it with a printf, and output may not be getting flushed when the program dies).

Here’s a trivial example that reproduces this problem:

  from lammps import lammps
  from mpi4py import MPI

  print "creating lammps object with mpi4py imported"
  lmp = lammps()
  print “done”

It gives the error above. If I comment out the line “from mpi4py import MPI”, it works fine.

I can think of a couple of possible ways to fix this, but don’t know how to do either of them:

1. Is there any way to ensure that in this somewhat complex, dynamically linked construct, the lammps routines end up invoking the mpi stubs routines rather than the real MPI routines that mpi4py loads (without me having to patch the LAMMPS source)?
2. Even more general, given that there’s clearly interest in having lammps accessible a a python module, is there a way to pass an MPI communicator to the lammps python startup process, so it’ll operate on that instead of mpi_comm_world? This way an MPI python script can split MPI_COMM_WORLD and have each LAMMPS process operate in its own subset of the MPI tasks.

Does anyone have any suggestions in the short term? Is there any interest by any LAMMPS developers in implementing #2, which is (I think) the more general and long-term useful solution? I’d be happy to contribute to a discussion of that possibility, and in principle I’d be happy to contribute a patch, but unfortunately for our internal bureaucratic reasons it would be quite a slow process if I had to actually write any code.
  
                    thanks,
                    Noam

Having (for the moment) worked around my lammps/numpy array issues, I have a perhaps subtle question about symbol resolution.

I have a python script that uses mpi4py, and each MPI process is supposed to start a _serial_ lammps process. I’ve compiled the lammps shared library in serial (using the default Makefile.serial), with mpi-stubs, and indeed when I do “nm” on the .so I see the MPI routines with ’T’. When I run this python script truly in serial, without mpi4py imported, it seems to do what I expect - the lammps runs in serial, and indeed invokes the lammps MPI stubs. However, when I import mpi4py (before or after I import lammps), I get an MPI error when I try to start the LAMMPS process:

[tin:19579] *** An error occurred in MPI_Comm_rank
[tin:19579] *** reported by process [3967680513,0]
[tin:19579] *** on communicator MPI_COMM_WORLD
[tin:19579] *** MPI_ERR_COMM: invalid communicator
[tin:19579] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[tin:19579] *** and potentially your MPI job)

and from what I can tell, the mpi-stubs routine isn’t being called (although I’m not 100% sure of this because I just checked it with a printf, and output may not be getting flushed when the program dies).

Here’s a trivial example that reproduces this problem:

        from lammps import lammps
        from mpi4py import MPI

        print "creating lammps object with mpi4py imported"
        lmp = lammps()
        print “done”

It gives the error above. If I comment out the line “from mpi4py import MPI”, it works fine.

I can think of a couple of possible ways to fix this, but don’t know how to do either of them:

1. Is there any way to ensure that in this somewhat complex, dynamically linked construct, the lammps routines end up invoking the mpi stubs routines rather than the real MPI routines that mpi4py loads (without me having to patch the LAMMPS source)?

this would require some experimentation, but i believe it might be
possible by performing what is called an "incremental link" or a
"partial link" with the stubs library before building the final shared
object and then building that shared object without including the
stubs library.

a simpler approach might be to compile LAMMPS with the same MPI
library as used by mpi4py but then change the library.cpp file to use
MPI_COMM_SELF instead of MPI_COMM_WORLD.

2. Even more general, given that there’s clearly interest in having lammps accessible a a python module, is there a way to pass an MPI communicator to the lammps python startup process, so it’ll operate on that instead of mpi_comm_world? This way an MPI python script can split MPI_COMM_WORLD and have each LAMMPS process operate in its own subset of the MPI tasks.

this is a yet unresolved issue with the python interface in LAMMPS.
after reading through some of mpi4py's docs, it appears that it would
be possible to pass communicators with mpi4py to objects loaded via
ctypes. this would also make the python interface specific not only to
the MPI library, but also to the corresponding python wrapper. not
really a big deal, but it may require some additional preprocessor
directives and some extract script magic inside the python code to
identify which python-mpi wrapper is used to then do the corresponding
setup and calling sequence.

Does anyone have any suggestions in the short term? Is there any interest by any LAMMPS developers in implementing #2, which is (I think) the more general and long-term useful solution? I’d be happy to contribute to a discussion of that possibility, and in principle I’d be happy to contribute a patch, but unfortunately for our internal bureaucratic reasons it would be quite a slow process if I had to actually write any code.

let us see what steve has to say about this, but this is something
that i am willing to spend some time on. this is a capability that is
worth having in LAMMPS, yet it had been postponed for lack of a
particular use case where this was actually needed.

axel.

Thanks. I’ll try to experiment with this, although your later suggestions sound cleaner to me.

That sounds like something I could easily patch, so maybe I’ll try it before the linking manipulation approach, but would be even better if it could be “officially” supported by lammps.

Do you think one could rely on there being some way to extract the underlying value (via ctypes) associated with a communicator from any particular python MPI wrapper, which could then be passed to lammps, under the assumption that the python MPI wrapper is providing the MPI shared object to the process, and the same value could then be passed by lammps routines to MPI routines and just work? If so, the only challenge would be for someone writing a python code that wants to do this to figure out how to extract the underlying MPI library’s communicator value from the python communicator object (assuming that such an object exists).

That’d be great. Let me know if there’s anything I can do to help.

Noam

this would require some experimentation, but i believe it might be
possible by performing what is called an "incremental link" or a
"partial link" with the stubs library before building the final shared
object and then building that shared object without including the
stubs library.

Thanks. I’ll try to experiment with this, although your later suggestions
sound cleaner to me.

a simpler approach might be to compile LAMMPS with the same MPI
library as used by mpi4py but then change the library.cpp file to use
MPI_COMM_SELF instead of MPI_COMM_WORLD.

That sounds like something I could easily patch, so maybe I’ll try it before
the linking manipulation approach, but would be even better if it could be
“officially” supported by lammps.

i just made a test for that, and it worked fine. that is probably as
"official" as you can get it.

i don't think it is a good option to support in general. it will
confuse less experienced people, and they are already struggling with
the compilation even though there has been quite significant efforts
to simplify and streamline the compilation process.

2. Even more general, given that there’s clearly interest in having lammps
accessible a a python module, is there a way to pass an MPI communicator to
the lammps python startup process, so it’ll operate on that instead of
mpi_comm_world? This way an MPI python script can split MPI_COMM_WORLD and
have each LAMMPS process operate in its own subset of the MPI tasks.

this is a yet unresolved issue with the python interface in LAMMPS.
after reading through some of mpi4py's docs, it appears that it would
be possible to pass communicators with mpi4py to objects loaded via
ctypes. this would also make the python interface specific not only to
the MPI library, but also to the corresponding python wrapper. not
really a big deal, but it may require some additional preprocessor
directives and some extract script magic inside the python code to
identify which python-mpi wrapper is used to then do the corresponding
setup and calling sequence.

Do you think one could rely on there being some way to extract the
underlying value (via ctypes) associated with a communicator from any
particular python MPI wrapper, which could then be passed to lammps, under
the assumption that the python MPI wrapper is providing the MPI shared
object to the process, and the same value could then be passed by lammps
routines to MPI routines and just work? If so, the only challenge would be
for someone writing a python code that wants to do this to figure out how to
extract the underlying MPI library’s communicator value from the python
communicator object (assuming that such an object exists).

let us see what steve has to say about this, but this is something
that i am willing to spend some time on. this is a capability that is
worth having in LAMMPS, yet it had been postponed for lack of a
particular use case where this was actually needed.

ok, after some digging around and some testing, here is what i could figure out.

there are two options to address this issue.

1) MPI communicators in fortran are integers and MPI-2 has a converter
function to reconstruct a C/C++ communicator from this integer. this
requires that the python MPI wrapper supports these converters and one
can get the numerical representation of the communicator, pass it as
an argument and then convert it back in the C/C++ representation
(which is an opaque typedef).

2) the python MPI wrapper has to have an API to give access to the
internal communicator handle.

neither 1) nor 2) is currently supported by mpi4py, but i found some
e-mails from steve inquiring about how to solve this problem in the
mpi4py forum and option 2) has been implemented in a development
version of mpi4py.

this would still require the lammps.py code to detect whether the MPI
wrapper is mpi4py and a communicator is passed as an argument, but
that is not very difficult to do. i have some prototype code for it
already. yet, there is no point to do this before a release of such an
mpi4py version is available.

for the time being, the only way to have parallelism within the LAMMPS
library would be to also install the USER-OMP package, recompile with
-fopenmp added and then instantiate the LAMMPS object with additional
flags that turn multi-threading on. e.g. with:

lmp = lmp.lammps(cmdargs=['-package','omp','4','-suffix','omp'])

this will use four threads per individual LAMMPS instance. for EAM and
a not too large number of threads (2-6) it should give a decent
speedup. if you are using OpenMPI, you should set processor affinity
to "socket", i.e. using --bind-to socket (or --bind-to-socket).

HTH,
    axel.

I agree that it’s not a good long term solution - just a fallback if you can’t get passing in a communicator from python to work.

I’m not sure what you mean by test, but I also got it working, but it required a change to the source (replace MPI_COMM_WORLD with a symbol that can be overridden by the Makefile, e.g. USE_MPI_COMM_WORLD, in library.cpp), but in the long term I’d prefer that users of the python code I’m writing can use stock lammps without having to patch it (although I think I can require a higher level of sophistication than lammps users in general), but again, I can wait.

This sounds very promising to me, and I therefore completely agree that it’s not worth investing a lot of effort until mpi4py implements some access to the underlying communicators. I’m perfectly happy to live with my minimal library.cpp patch which uses MPI_COMM_SELF until the new mpi4py version is ready.

That’s exactly what I was planning in the medium term, in fact, although for the moment I’m happy with each lammps process being purely serial.

thanks,
Noam

  1. Even more general, given that there’s clearly interest in having lammps accessible a a >python module, is there a way to pass an MPI communicator to the lammps python startup >process, so it’ll operate on that instead of mpi_comm_world? This way an MPI python script >can split MPI_COMM_WORLD and have each LAMMPS process operate in its own subset of >the MPI tasks.

Does anyone have any suggestions in the short term? Is there any interest by any LAMMPS >developers in implementing #2, which is (I think) the more general and long-term useful >solution?

When I last looked at this (a year or so ago), the issue was really an mpi4py issue,
not LAMMPS. Or more generally, its b/c each MPI has its own definition
of the MPI communicator. Mpi4py (and other Python MPI wrappers) thus
do not have a simple way to portably wrap any version of MPI.

I had an exchange with the mpi4py author and he had a reasonable solution
for passing a pointer to an MPI communicator as I recall,
(don’t remember the details at the moment), and was planning to release it as
a new mpi4py helper function, in the main version of mpi4py at some point.
Don’t know if that has happened.

Steve

2. Even more general, given that there’s clearly interest in having lammps
accessible a a >python module, is there a way to pass an MPI communicator to
the lammps python startup >process, so it’ll operate on that instead of
mpi_comm_world? This way an MPI python script >can split MPI_COMM_WORLD and
have each LAMMPS process operate in its own subset of >the MPI tasks.

Does anyone have any suggestions in the short term? Is there any interest
by any LAMMPS >developers in implementing #2, which is (I think) the more
general and long-term useful >solution?

When I last looked at this (a year or so ago), the issue was really an
mpi4py issue,
not LAMMPS. Or more generally, its b/c each MPI has its own definition
of the MPI communicator. Mpi4py (and other Python MPI wrappers) thus
do not have a simple way to portably wrap any version of MPI.

I had an exchange with the mpi4py author and he had a reasonable solution
for passing a pointer to an MPI communicator as I recall,
(don't remember the details at the moment), and was planning to release it
as
a new mpi4py helper function, in the main version of mpi4py at some point.
Don't know if that has happened.

no. still at 1.3.1.

axel.