Pymongo Connection Error? Worker Config issue?

Atomate
1

Hello!

I’ve installed atomate and the necessary prereqs on a worker and server where I’ve setup and configured a MongoDB instance. I have the atomate config files for the db and workers set up and have confirmed that my MongoDB port is open and reachable via a telnet query and Pymongo connection via MongoClient.

However, I am getting a Pymongo connection error to localhost (I’ve set the correct host and port for my MongoDB server via the db.json file and my_launchpad.yaml files) when I try to submit a workflow via the following script:

#!/usr/bin/env python3
import numpy as np
from pymatgen import MPRester
from fireworks import LaunchPad
from atomate.vasp.workflows.presets.core import wf_elastic_constant
from atomate.vasp.powerups import add_modify_incar

Get the structure from the Materials Project

with MPRester() as mpr:
struct = mpr.get_structure_by_material_id(‘mp-5229’)

    # Create the workflow
    # Create the workflow
    workflow = wf_elastic_constant(struct)

    # Create the launchpad and add our workflow
    launchpad = LaunchPad.auto_load()
    launchpad.add_wf(workflow)

The error is as follows:

$ python3 elastic_constant.py
/apps2/python/3.6.8/lib/python3.6/site-packages/pandas/compat/init.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
warnings.warn(msg)
/apps2/python/3.6.8/lib/python3.6/site-packages/pandas/compat/init.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
warnings.warn(msg)
Traceback (most recent call last):
File “/home/det16103/.local/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1537, in get_new_fw_id
‘$inc’: {‘next_fw_id’: quantity}})[‘next_fw_id’]
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/collection.py”, line 3136, in find_one_and_update
array_filters, session=session, **kwargs)
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/collection.py”, line 2885, in __find_and_modify
write_concern.acknowledged, _find_and_modify, session)
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1490, in _retryable_write
with self._tmp_session(session) as s:
File “/apps2/python/3.6.8/lib/python3.6/contextlib.py”, line 81, in enter
return next(self.gen)
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1823, in _tmp_session
s = self._ensure_session(session)
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1810, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1763, in __start_session
server_session = self._get_server_session()
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1796, in _get_server_session
return self._topology.get_server_session()
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/topology.py”, line 485, in get_server_session
None)
File “/home/det16103/.local/lib/python3.6/site-packages/pymongo/topology.py”, line 209, in _select_servers_loop
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “elastic_constant.py”, line 19, in
launchpad.add_wf(workflow)
File “/home/det16103/.local/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 402, in add_wf
reassign_all=reassign_all)
File “/home/det16103/.local/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1575, in _upsert_fws
first_new_id = self.get_new_fw_id(quantity=len(fws))
File “/home/det16103/.local/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1540, in get_new_fw_id
“Could not get next FW id! If you have not yet initialized the database,”
ValueError: Could not get next FW id! If you have not yet initialized the database, please do so by performing a database reset (e.g., lpad reset)

Any help in regards to resolving this error is appreciated.
Thanks!
-Dennis

2

Basically I have no idea why it’s trying to reach out to ‘localhost: 27017’ when that is not anywhere in my config files.

3

Update: I can get workflows to submit and run if I copy my ‘my_launchpad.yaml’, ‘my_fworker.yaml’, and ‘my_qadapter.yaml’ into my workdir where I’m submitting the job from. I’m guessing I have an unset environment variable which tells atomate/python where to find my config files?

this is some of the output from my run which suggests I need to set some environment vars:

Found many potential paths for LAUNCHPAD_LOC: ['/home/det16103/atomate/config/my_launchpad.yaml', '/home/det16103/atomate/test/cubic_sto/my_launchpad.yaml']

Choosing as default: /home/det16103/atomate/config/my_launchpad.yaml

Found many potential paths for FWORKER_LOC: ['/home/det16103/atomate/config/my_fworker.yaml', '/home/det16103/atomate/test/cubic_sto/my_fworker.yaml']

Choosing as default: /home/det16103/atomate/config/my_fworker.yaml

Found many potential paths for QUEUEADAPTER_LOC: ['/home/det16103/atomate/config/my_qadapter.yaml', '/home/det16103/atomate/test/cubic_sto/my_qadapter.yaml']

Choosing as default: /home/det16103/atomate/config/my_qadapter.yaml
4

Those are configuration options for Fireworks, the workflow software used to manage atomate calculations.

Read more on your Fireworks config, FWConfig, here: https://materialsproject.github.io/fireworks/config_tutorial.html?highlight=launchpad_loc

Your problem is that you are using LaunchPad.auto_load to instantiate your lpad. If you’re running the auto_load from a directory containing FW config files, the CWD is a higher priority for auto-loading, so it defaults to that location. If not, it will default to another lpad, in your case the localhost one included with the fireworks installation.

If you want to set your FW config globally, follow the instructions at that link.

If you just want to specify the launchpad in your script, create your lpad object with LaunchPad.from_file(file), or by specifying host, port, name, etc. manually with LaunchPad(*args, **kwargs).

If you’re working from the command line and you just want to specify a launchpad for a single command, use the -l option: lpad -l path/to/launchpad/file [COMMAND] ...

Hope this helps.

1 Like
5

Perfect! I corrected it by creating a FW_config.yaml in my ‘/home//.fireworks’ dir.

Thanks for the help!