patch for python module

LAMMPS LAMMPS Mailing List Mirror
1

Hiya,

This email yesterday was a bit premature - I had forgotten that the CDLL() call was not run immediately on import, so my ‘test’ wasn’t actually testing anything - the problem was still there. Luckily, I’ve solved it!

I have two patches for the code. First, CDLL() doesn’t look in the python modules directory for C libraries, as it doesn’t think they live there, it looks on LD_LIBRARY_PATH, see these Stack Overflow questions about how that works:

http://stackoverflow.com/questions/9775425/ctypes-doesnt-find-fink-installed-libraries-in-sw-lib
http://stackoverflow.com/questions/856116/changing-ld-library-path-at-runtime-for-ctypes

I’m not sure how this ever worked, but I suspect that you might have tested it in the install directory (which doesn’t import the installed version, but the local one) or something, when weird things happen and things seem to work when they don’t.

Anyway, in order to get this to work, you either need to move _lammps.so and/or _lammps_serial.so into LD_LIBRARY_PATH somewhere like /usr/lib, or you can hack the python module to know where it lives and give the absolute path to the C library, like this:

(python/lammps.py)

LMPINT = 0
LMPDOUBLE = 1
LMPIPTR = 2
LMPDPTR = 3
LMPDPTRPTR = 4

LOCATION = os.path.dirname(file)

class lammps:
def init(self,args=None):

attempt to load parallel library first, serial library next

could provide caller a flag to choose which library to load

try:
self.lib = CDLL(os.path.join(LOCATION, “_lammps.so”))
except:
#try:
self.lib = CDLL(os.path.join(LOCATION, “_lammps_serial.so”))
#except:
#raise OSError,“Could not load LAMMPS dynamic library”

(note Steve that this also includes my previous patch for the invalid try/except block.) I think this is the more ‘pythonic’ way of doing it, as you don’t need to mess around assuming anything about the system configuration for this to work, and you python libraries all live under site-packages where they belong.

Now, when you call

l = lammps.lammps()

you are actually loading the correct library - which brings me to my second patch.

When I run this I get a symbol error like this (illustrated in the interactive interpreter):

import lammps
l = lammps.lammps()
Traceback (most recent call last):
File “”, line 1, in
File “/Library/Python/2.6/site-packages/lammps.py”, line 38, in init
self.lib = CDLL(os.path.join(LOCATION, “_lammps_serial.so”))
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/init.py”, line 345, in init
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Library/Python/2.6/site-packages/_lammps_serial.so, 6): Symbol not found: _MPI_Abort
Referenced from: /Library/Python/2.6/site-packages/_lammps_serial.so
Expected in: flat namespace
in /Library/Python/2.6/site-packages/_lammps_serial.so

The important part, “Symbol not found: _MPI_Abort” seems to indicate that the MPI stub library has not been included in the build correctly.

This is because there’s a subtle error in the setup_serial.py install script which means that the MPI library isn’t included! It’s here:

libfiles = glob.glob("s/src/*.cpp" path) +
glob.glob("s/src/STUBS/*.cpp" path)

Now, if you actually look in the STUBS folder, you’ll see that we have a simple mpi.c file, and no *.cpp matching ones! Rather than renaming it to mpi.cpp and wrapping all the function declarations in extern ‘C’ { … } blocks in the header, the smallest code change to fix this is just to change the above to:

libfiles = glob.glob("s/src/*.cpp" path) +
glob.glob("s/src/STUBS/*.c" path)

which gives us exactly what we wanted. We can now do everything with a simple sudo python setup_serial.py install and then run the following in the interactive interpreter as follows:

import lammps
l = lammps.lammps()
LAMMPS (8 Feb 2012)

and finally, we have a working python module. If anyone has a system other than OS X (10.6.8, python 2.6) to test this on, I’d be grateful, as the main platform for lammps seems to be linux, and I don’t have a linux box handy. I don’t expect there to be any platform issues with these changes though, as they’re all in python land, and I’ve managed to avoid any hard coded path issues.

Thanks, apologies about the millions of very long emails to the list, and please let me know if anyone would like these changes in diff format or anything.

Joe

2

Thanks for looking into these Python issue carefully - I'll take
a look at your code changes next week. A couple of comments:

a) are you sure you are using the latest version of LAMMPS?
    I recall fixing the MPI *.cpp vs *.c issue previously
    which occurred when we changed the MPI stubs to
    a fully C-compatible code and forgot to twiddle the Python
    build. Ditto for leaving some debug try/expect code in lammps.py

b) You are correct that a Python wrapper on LAMMPS isn't
    fully "working" until you have instantiated a LAMMPS object
    from Python, which is more than the import. You have to
    invoke the CDLL command at instantiation to see if it all
    works. I believe the lammps.py module does this in a way
    that is fairly standard for ctypes-wrapped C libraries
    where you want an object-oriented Python-style interface to
    the lib, which is what lammps.py provides. The doc pages
    doc/Section_python.html describe all the steps to make
    sure you can do this, including common errors that arise
    when CDLL doesn't work (e.g. can't find the *.so files).
    I consider that something for the user to fix by making sure
    you have set the correct LD_LIBRARY_PATH in your shell;
    it's not a lammps.py issue.

c) your earlier comment about LAMMPS dependencies needed
    to do the Python wrap is correct. LAMMPS has several
    possible dependencies on other libs: MPI, FFTW, any of
    the auxiliary packages you can build LAMMPS with that
    use an external lib (see the lammps/lib directory). If you
    want to Python-wrap LAMMPS when built with any of those, then
    I think you have no choice but to insure that those libs
    have been built as *.so files and installed somewhere that
    the Python build script can find them. We provide a means
    to do that for the MPI case (since that's the most common)
   and the doc/Section_python.html page describes how to
   do this and insure you have a libmpi.so file. If you or others
   want to provide some more powerful variants of python/setup.py
   and documentation that work with other LAMMPS build options
   (FFTW, MEAM, ReaxFF, etc) we'd be happy to include them.
   It's just that until we or someone else needs it, we haven't
   worked out the details.

d) your earlier comments about the lib interface being
    under-documented are also corrrect. Note that the
    Python wrapper is just a wrapper on the lib interface,
    so this is really an issue with the C interface functions.
    My view of the src/library.cpp file is that it's a dummy file
    which we've provided simply to illustrate what can be done.
    We're happy to add functions or documentation to it
    if other people write new interface functions. I myself
    just add something when I need it for a new project. Ideally
    more doc info might become a new chapter in the Developers.pdf
    file. We just haven't had time to do more on this,
    so feel free to contribute code/docs or suggestions
    as you make progress. You can always post Qs if
    you want to know how to do something thru the lib
    interface. Your coments/questions indicate you
    know what you are doing, so your contributions
    would be valuable!

Thanks,
Steve

3

Steve,

Indeed, upon trying my patch on a fresh checkout, I saw that the
*.c/*.cpp issue had been resolved. I was using a checkout from
February - apologies for double-reporting the bug.

However, I need to disagree with you on the LD_LIBRARY_PATH issue. I
believe you that instantiating the lammps class is the standard way of
doing things for ctypes wrapped libraries (I've not met one before,)
but having to set environment variables yourself for a python library,
as you suggest in the documentation, is definitely non standard, and I
would say unnecessarily complex for the user. My patch resolves the
issue by hiding that detail away. It is much more pythonic to make
everything work magically than to require people understand the detail
of how CDLL imports C libraries!

It is up to you how to configure this in future, but I would like to
write a python library/libraries that depend on lammps as a python
package, and I'd like installation of this dependency to be as
pythonic (and therefore as painless) as possible!

On point c), if I discover that I need one of these other dependencies
then I may contribute some easier ways for users to include them, but
at the moment I don't need MPI or JPEG or anything, so I'm happy to
leave these as they are.

My next job is to extend the C interface to suit my needs - I will
definitely contribute back my work, and make sure it is documented, so
that if you'll have it people can use it with my dependent libraries.

Thanks for your thoughts, and I look forward to hearing your decision
about my patch to the CDLL calls.

Joe

4

I'll look at your CDLL suggestions next week.
My recollection is that the CDLL load command
does not go thru the normal Python paths to find
libs, but thru LD_LIBRARY_PATH. But maybe you
have a more clever way to do this, or there is some
upgraded version of ctypes to get around this.
I agree the mechanism is a bit non-Pythonic.

Thanks,
Steve

5

I'll look at your CDLL suggestions next week.
My recollection is that the CDLL load command
does not go thru the normal Python paths to find
libs, but thru LD_LIBRARY_PATH. But maybe you

it'll end up as a call to dlopen(3), which in turn
will not need to use LD_LIBRARY_PATH, if
an absolute pathname is given, which is what
joe's patch does.

i also would suggest to change the default
installation method to be

python setup.py install --user

as that can install the package to be found
automatically without having to set or amend
PYTHONPATH or using superuser privilege.

cheers,
    axel.

6

Hi - I just changed the Python wrapping of LAMMPS
in the 17Aug patch. It is now simpler and hopefully
addresses your points.

More specifically, I got rid of the setup.py step, and just
let Python load the LAMMPS wrapper and shared lib directly.
This seems much simpler, particularly if you are rebuilding
LAMMPS or the editing the Python interface methods to
add new ones.

The steps are now:

1) build LAMMPS as a shared lib (added new targets for this):

make makeshlib
make -f Makefile.shlib g++

2) add 2 env variables settings

one for PYTHONPATH, to point to lammps/python/lammps.py
the other to LD_LIBRARY_PATH to point to lammps/src/liblmp.so
  so that CDLL can find it

Then you can just do "import lammps" from a Python script

I think you mention that step 2 seems painful, but it's a one-time
thing and I don't see anyway around it. You could copy the
lammps.py and liblmp.so into the Python site-packages, but
you generally need root privileges to do that, and you need
to redo it every time you rebuild LAMMPS (e.g. add a package).

Or you could put the 2 files in some local space where you
keep all your Python add-ons, assuming you have set PYTHONPATH
and LD_LIBRARY_PATH to point to that space.

It is much more pythonic to make
everything work magically than to require people understand the detail
of how CDLL imports C libraries!

Try it out and see if you have additional suggestions or if
this is still not Pythonic? Every ctypes wrapper I've seen uses
CDLL which then requires LD_LIBRARY_PATH to be correct
(although you can specify an absolute path to CDLL I believe),
so I don't see how/why you would use custom code for the load
operation.

Steve