Connecting to MongoDB atlas

Hi,

I have been trying to set up the MongoDB atlas database for fireworks.

My launchpad looks like the following:

host: mongodb+srv://cluster0-b05rv.mongodb.net/test

port: 27017

name: test

username: admin

password: <>

logdir: null

ssl: false

strm_lvl: INFO

user_indices: []

wf_user_indices: []

However, I have been getting the following error when I run lpad reset:

ValueError: FireWorks was not able to connect to MongoDB. Is the server running? The database file specified was /n/home02/zzhu/atomate/config/my_launchpad.yaml.

It only connects if I change the fire /fireworks/core/.launchpad.py around line 149 to the following:

get connection

    #self.connection = MongoClient(host, port, ssl=self.ssl,

    #    username=username,password=password,

    #    ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,

    #    ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,

    #    socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False)

    #self.db = self.connection[name]

    #if username:

    #     self.db.authenticate(username, password, source=self.authsource)

    self.connection = MongoClient(host, port, username=username, password=password)

    self.db = self.connection[name]

The db.authenticate function always gives me trouble.

I wonder if this is a known issue and if there’s any solution to this?

Thanks,

Zoe Zhu

Hi Zoe,

Based on talking to some colleagues, it seems the db.authenticate() function in pymongo is deprecated. In FWS v1.8.5 (just released now) we changed the authentication code. Let us know if it works for you.

···

Best,
Anubhav

Hi Anubhav,

Thank you so much for your help.

I am still getting an error (but a different one):

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

···

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz…@g.harvard.edu, [email protected]

(310)-210-0580

And I got this error when I did lpad reset.

···

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz…@g.harvard.edu, [email protected]

(310)-210-0580

Hi Zoe

Do you happen to know what part of the connection code is causing the error? I’m happy to push a fix, just not sure which part of the code is causing problems on your end.

Best,

Anubhav

···

Best,
Anubhav

Hi Anubhav,

Sorry I didn’t include all the error messages! See below:

Traceback (most recent call last):

File “/n/home02/zzhu/.conda/envs/atomate_env/bin/lpad”, line 11, in

load_entry_point('FireWorks==1.8.5', 'console_scripts', 'lpad')()

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py”, line 1147, in lpad

args.func(args)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py”, line 148, in reset

lp.workflows.count()))[0].upper() == 'Y':

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py”, line 1766, in count

return self._count(cmd, collation, session)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py”, line 1563, in _count

with self._socket_for_reads(session) as (sock_info, slave_ok):

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/contextlib.py”, line 81, in enter

return next(self.gen)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1099, in _socket_for_reads

server = topology.select_server(read_preference)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py”, line 224, in select_server

address))

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py”, line 183, in select_servers

selector, server_timeout, address)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py”, line 199, in _select_servers_loop

self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

···

Thanks,

Zoe


Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz…@g.harvard.edu, [email protected]

(310)-210-0580

Hi Zoe,

Actually, we found an issue in the previous release and pushed a fix in v1.8.6. Let me know if that one works or if you still have problems.

···

Best,
Anubhav

Hi Anubhav,

Thanks for your response. However, I am still getting the same error message when I do lpad reset:

Traceback (most recent call last):

File “/n/home02/zzhu/.conda/envs/atomate_env/bin/lpad”, line 11, in

load_entry_point('FireWorks==1.8.6', 'console_scripts', 'lpad')()

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py”, line 1147, in lpad

args.func(args)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py”, line 148, in reset

lp.workflows.count()))[0].upper() == 'Y':

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py”, line 1766, in count

return self._count(cmd, collation, session)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/collection.py”, line 1563, in _count

with self._socket_for_reads(session) as (sock_info, slave_ok):

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/contextlib.py”, line 81, in enter

return next(self.gen)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1099, in _socket_for_reads

server = topology.select_server(read_preference)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py”, line 224, in select_server

address))

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py”, line 183, in select_servers

selector, server_timeout, address)

File “/n/home02/zzhu/.conda/envs/atomate_env/lib/python3.6/site-packages/pymongo/topology.py”, line 199, in _select_servers_loop

self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

···

Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz…@g.harvard.edu, [email protected]

(310)-210-0580

On Wed, Feb 13, 2019 at 5:14 PM Anubhav Jain [email protected] wrote:

Hi Zoe,

Actually, we found an issue in the previous release and pushed a fix in v1.8.6. Let me know if that one works or if you still have problems.

On Tue, Feb 12, 2019 at 6:49 PM Anubhav Jain [email protected] wrote:

Hi Zoe

Do you happen to know what part of the connection code is causing the error? I’m happy to push a fix, just not sure which part of the code is causing problems on your end.

Best,

Anubhav

On Tue, Feb 12, 2019 at 3:13 PM Zoe Zhu [email protected] wrote:

Hi Anubhav,

Thank you so much for your help.

I am still getting an error (but a different one):

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed


Ziyan (Zoe) Zhu

Ph.D. Candidate

Department of Physics, Harvard University

zz…@g.harvard.edu, [email protected]

(310)-210-0580

On Tue, Feb 12, 2019 at 4:37 PM Anubhav Jain [email protected] wrote:

Hi Zoe,

Based on talking to some colleagues, it seems the db.authenticate() function in pymongo is deprecated. In FWS v1.8.5 (just released now) we changed the authentication code. Let us know if it works for you.

On Tue, Feb 12, 2019 at 12:38 PM Zoe Zhu [email protected] wrote:

Hi,

I have been trying to set up the MongoDB atlas database for fireworks.

My launchpad looks like the following:

host: mongodb+srv://cluster0-b05rv.mongodb.net/test

port: 27017

name: test

username: admin

password: <>

logdir: null

ssl: false

strm_lvl: INFO

user_indices: []

wf_user_indices: []

However, I have been getting the following error when I run lpad reset:

ValueError: FireWorks was not able to connect to MongoDB. Is the server running? The database file specified was /n/home02/zzhu/atomate/config/my_launchpad.yaml.

It only connects if I change the fire /fireworks/core/.launchpad.py around line 149 to the following:

get connection

    #self.connection = MongoClient(host, port, ssl=self.ssl,
    #    username=username,password=password,
    #    ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,
    #    ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,
    #    socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False)
    #self.db = self.connection[name]
    #if username:
    #     self.db.authenticate(username, password, source=self.authsource)
    self.connection = MongoClient(host, port, username=username, password=password)
    self.db = self.connection[name]

The db.authenticate function always gives me trouble.

I wonder if this is a known issue and if there’s any solution to this?

Thanks,

Zoe Zhu

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav


Best,
Anubhav

Hi all,

Due to Google groups withholding these messages, I can’t tell if these problems are with FWS v1.8.5 or v1.8.6.

Can you clarify if the problem is still present in FWS v1.8.6?

···

Best,
Anubhav

I realized I had v 1.8.4 installed
Upgrading did not work nonetheless, and I still have a pymongo.error with version 1.8.6 (see below).

I am actually behind a firewall, how can I make sure this is not the problem?

Thank you

$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)

import fireworks
print(fireworks.version)
1.8.6

import pymongo
print(pymongo.version)
3.7.2

$ which lpad
/home/mdi0316/anaconda3/bin/lpad

$ lpad -l my_launchpad_testing.yaml get_wflows

Traceback (most recent call last):
File “/home/mdi0316/anaconda3/bin/lpad”, line 10, in
sys.exit(lpad())
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py”, line 1147, in lpad
args.func(args)
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/fireworks/scripts/lpad_run.py”, line 297, in get_wfs
ids = lp.get_wf_ids(query, sort, args.max, count_only=args.display_format == ‘count’)
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 667, in get_wf_ids
for fw in self.workflows.find(criteria, {“nodes”: True}, sort=sort).limit(limit):
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/cursor.py”, line 1189, in next
if len(self.__data) or self._refresh():
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/cursor.py”, line 1087, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1563, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1516, in __start_session
server_session = self._get_server_session()
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 1549, in _get_server_session
return self._topology.get_server_session()
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/topology.py”, line 427, in get_server_session
None)
File “/home/mdi0316/anaconda3/lib/python3.6/site-packages/pymongo/topology.py”, line 199, in _select_servers_loop
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: ds049170.mongolab.com:49170: timed out

Hi Marco

To make the firewall is not the problem, I would first try connecting to your database using the standard mongo shell (from MongoDB) rather than FireWorks. If you can connect with the mongo shell, then you should (in theory) be able to connect with FWS - so let us know if you can connect with mongo shell but not FWS. If you cannot connect with the mongo shell, FireWorks will also not work. In that case, you will need to work with a system administrator to get that aspect worked out before moving on to FWS…

Best,

Anubhav

···

Best,
Anubhav

Dear Anubhav,

as I stated in my original question, yes, I can connect to port 27017

Nevertheless I get two wanings and I am not sure I am really connected

$ mongo
MongoDB shell version v4.0.3
connecting to: mongodb://127.0.0.1:27017
WARNING: No implicit session: Logical Sessions are only supported on server versions 3.6 and greater.
Implicit session: dummy session
MongoDB server version: 2.6.10
WARNING: shell and server versions do not match

Please let me know if you have any suggestion on this

Kind regards

Hi Marco,

I can connect to the testing ds049170.mongolab.com:49170 server using the my_launchpad_testing.yaml document specified in the documentation using the latest FireWorks. So, my guess is that there is a firewall issue that is preventing you from accessing that test database hosted online.

In terms of testing using the “mongo” shell, I meant testing that you can connect to the remote server using the mongo shell. That is, testing to see if you can connect to the ds049170.mongolab.com:49170 server using the mongo shell.

There are some details here:

https://docs.mongodb.com/manual/mongo/

But you can just type the following shell commands:

mongo --username test_user --password testing123 --authenticationDatabase fireworks --host ds049170.mongolab.com --port 49170

MongoDB shell version v3.4.4

connecting to: mongodb://ds049170.mongolab.com:49170/

MongoDB server version: 3.6.6

WARNING: shell and server versions do not match

use fireworks

switched to db fireworks

db.fireworks.count()

2

If those commands do not work on your system (they do on mine), it is almost certainly a firewall problem as those commands just use the mongo shell and not FWS

···

Best,
Anubhav

Note that another way to test the firewall issue is to try the same commands on home network vs a company network.

···

Best,
Anubhav

Hello Anubhav,

I am trying to connect to MongoDB atlas through fireworks. The error is below.

My Fireworks is version 1.8.7.

Please let me know if there is any other information that I can provide that would help.

C:\Users\athre\Miniconda3\Lib\site-packages\fw_tutorials\introduction>lpad reset

Traceback (most recent call last):

File “c:\users\athre\miniconda3\lib\runpy.py”, line 193, in _run_module_as_main

main”, mod_spec)

File “c:\users\athre\miniconda3\lib\runpy.py”, line 85, in _run_code

exec(code, run_globals)

File “C:\Users\athre\Miniconda3\Scripts\lpad.exe_main_.py”, line 9, in

File “c:\users\athre\miniconda3\lib\site-packages\fireworks\scripts\lpad_run.py”, line 1146, in lpad

args.func(args)

File “c:\users\athre\miniconda3\lib\site-packages\fireworks\scripts\lpad_run.py”, line 148, in reset

lp.workflows.count()))[0].upper() == ‘Y’:

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\collection.py”, line 1766, in count

return self._count(cmd, collation, session)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\collection.py”, line 1563, in _count

with self._socket_for_reads(session) as (sock_info, slave_ok):

File “c:\users\athre\miniconda3\lib\contextlib.py”, line 112, in enter

return next(self.gen)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\mongo_client.py”, line 1099, in _socket_for_reads

server = topology.select_server(read_preference)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py”, line 224, in select_server

address))

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py”, line 183, in select_servers

selector, server_timeout, address)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py”, line 199, in _select_servers_loop

self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

Thanks,

Athreya Ramesh

···

On Tuesday, February 12, 2019 at 3:38:15 PM UTC-5, Zoe Zhu wrote:

Hi,

I have been trying to set up the MongoDB atlas database for fireworks.

My launchpad looks like the following:

host: mongodb+srv://cluster0-b05rv.mongodb.net/test

port: 27017

name: test

username: admin

password: <>

logdir: null

ssl: false

strm_lvl: INFO

user_indices: []

wf_user_indices: []

However, I have been getting the following error when I run lpad reset:

ValueError: FireWorks was not able to connect to MongoDB. Is the server running? The database file specified was /n/home02/zzhu/atomate/config/my_launchpad.yaml.

It only connects if I change the fire /fireworks/core/.launchpad.py around line 149 to the following:

get connection

    #self.connection = MongoClient(host, port, ssl=self.ssl,
    #    username=username,password=password,
    #    ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,
    #    ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,
    #    socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False)
    #self.db = self.connection[name]
    #if username:
    #     self.db.authenticate(username, password, source=self.authsource)
    self.connection = MongoClient(host, port, username=username, password=password)
    self.db = self.connection[name]

The db.authenticate function always gives me trouble.

I wonder if this is a known issue and if there’s any solution to this?

Thanks,

Zoe Zhu

Hi Athreya

Sorry but this is not enough information to debug.

Can you first try connecting to Atlas using the pymongo tool directly (and not using FireWorks)? Once you have that working, you can let us know what settings you used to get it to work with direct pymongo connection that are not working when you try it with FireWorks.

···

On Tuesday, March 26, 2019 at 3:59:21 PM UTC-7, Athreya Ramesh wrote:

Hello Anubhav,

I am trying to connect to MongoDB atlas through fireworks. The error is below.

My Fireworks is version 1.8.7.

Please let me know if there is any other information that I can provide that would help.

C:\Users\athre\Miniconda3\Lib\site-packages\fw_tutorials\introduction>lpad reset

Traceback (most recent call last):

File “c:\users\athre\miniconda3\lib\runpy.py”, line 193, in _run_module_as_main

main”, mod_spec)

File “c:\users\athre\miniconda3\lib\runpy.py”, line 85, in _run_code

exec(code, run_globals)

File “C:\Users\athre\Miniconda3\Scripts\lpad.exe_main_.py”, line 9, in

File “c:\users\athre\miniconda3\lib\site-packages\fireworks\scripts\lpad_run.py”, line 1146, in lpad

args.func(args)

File “c:\users\athre\miniconda3\lib\site-packages\fireworks\scripts\lpad_run.py”, line 148, in reset

lp.workflows.count()))[0].upper() == ‘Y’:

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\collection.py”, line 1766, in count

return self._count(cmd, collation, session)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\collection.py”, line 1563, in _count

with self._socket_for_reads(session) as (sock_info, slave_ok):

File “c:\users\athre\miniconda3\lib\contextlib.py”, line 112, in enter

return next(self.gen)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\mongo_client.py”, line 1099, in _socket_for_reads

server = topology.select_server(read_preference)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py”, line 224, in select_server

address))

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py”, line 183, in select_servers

selector, server_timeout, address)

File “c:\users\athre\miniconda3\lib\site-packages\pymongo\topology.py”, line 199, in _select_servers_loop

self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: connection closed,connection closed,connection closed

Thanks,

Athreya Ramesh

On Tuesday, February 12, 2019 at 3:38:15 PM UTC-5, Zoe Zhu wrote:

Hi,

I have been trying to set up the MongoDB atlas database for fireworks.

My launchpad looks like the following:

host: mongodb+srv://cluster0-b05rv.mongodb.net/test

port: 27017

name: test

username: admin

password: <>

logdir: null

ssl: false

strm_lvl: INFO

user_indices: []

wf_user_indices: []

However, I have been getting the following error when I run lpad reset:

ValueError: FireWorks was not able to connect to MongoDB. Is the server running? The database file specified was /n/home02/zzhu/atomate/config/my_launchpad.yaml.

It only connects if I change the fire /fireworks/core/.launchpad.py around line 149 to the following:

get connection

    #self.connection = MongoClient(host, port, ssl=self.ssl,
    #    username=username,password=password,
    #    ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,
    #    ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,
    #    socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False)
    #self.db = self.connection[name]
    #if username:
    #     self.db.authenticate(username, password, source=self.authsource)
    self.connection = MongoClient(host, port, username=username, password=password)
    self.db = self.connection[name]

The db.authenticate function always gives me trouble.

I wonder if this is a known issue and if there’s any solution to this?

Thanks,

Zoe Zhu