Using Fireworks with an open IP address over an HPC cluster

FireWorks
1

Dear all,

I have tried to connect to a MongoDB database from an HPC cluster (M100 at Cineca). I have contacted the HPC help desk and we managed to whitelist the IP address of the database. I have managed to connect successfully to it once I was on the login node. However, when I try to launch a simulation on the compute node, I received the following error message

  File "/m100/home/userexternal/ochehaim/miniconda3/envs/tribchem/lib/python3.7/site-packages/pymongo/topology.py", line 220, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: No primary available for writes, Timeout: 30s, Topology Description: <TopologyDescription id: 63089acbe8c856497cd7b78c, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('cluster0-shard-00-00.pj6wi.mongodb.net', 27017) server_type: RSSecondary, rtt: 0.021975754328072073>, <ServerDescription ('cluster0-shard-00-01.pj6wi.mongodb.net', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('cluster0-shard-00-01.pj6wi.mongodb.net:27017: [Errno 101] Network is unreachable')>, <ServerDescription ('cluster0-shard-00-02.pj6wi.mongodb.net', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('cluster0-shard-00-02.pj6wi.mongodb.net:27017: [Errno 101] Network is unreachable')>]>

which seems like the database is unreachable (which is not the case, since the connection on the login node works fine). Is it possible that I need to tweak Fireworks in order to make it work on the compute node? If so, is there anything I can do?

Thanks.

1 Like
2

If it’s possible to run interactive jobs on the compute node, my first suggestion would be to do that and see whether you can connect to MongoDB from the interactive job (try a direct connection using pymongo / MongoDB tools to avoid any issues that may be speific to FireWorks). If you can successfully get a connection to the MongoDB server from the compute node, then FireWorks should also work.

If you can’t get a connection to the MongoDB server from the compute node, you may need to try some hacky solution. These include:

  • setting up MongoDB on the compute cluster itself, or installing the database anywhere that the compute nodes can actually connect to
  • ssh tunneling between login node and compute node - but you’ll need to keep the tunnel alive
  • FireWorks in “offline” mode (see docs), but honestly I don’t recommend this. It hasn’t been tested in awhile, is not a great experience when it works, and certain codes like atomate don’t support it
3

Dear Anubhav,

Thanks for the kind reply. I will try your suggestions and come back here to let you know what happens.

Thanks again for the help.