Question about remote host

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.

    How can I use ssh key by fireworks?

  2. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd

(in scripts/qlaunch.py)

is failed because fabric.api no longer exists.

   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error

    File "........./lib/python3.5/site-packages/fabric/context_managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.

Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.

   How should I configure my environment?

I would appreciate your help.

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo
  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.
···

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like

···

for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

···

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel

  2. Type “mongo --port 27018 mydb” to open a tunnel

  3. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

···

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp”

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            

    r = os.path.expanduser(r)                                                                         

    with conn.cd(r):                                                                                  

        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

···

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

···

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

A quick note - there is an extraneous space in my last message, the revision should read:

                    connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:
···

On Friday, September 7, 2018 at 4:32:12 PM UTC-7, Anubhav Jain wrote:

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': ' /home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Hi, Prof. Anubhav Jain,

Thanks for replying, and your modified code works well for us.
We would appreciate if you could add the command line argument that takes in the key filename, because all supercomputers we use require key files.

Thanks,

Akira Takahashi

2018年9月8日土曜日 8時32分53秒 UTC+9 Anubhav Jain:

···

A quick note - there is an extraneous space in my last message, the revision should read:

                    connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:

On Friday, September 7, 2018 at 4:32:12 PM UTC-7, Anubhav Jain wrote:

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': ' /home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

I have added a new command line option --remote_keyfile, to qlaunch and released it in FWS v1.7.9. Unfortunately, I was not able to test it thoroughly.

Could you please give it a try and let me know if the new version works?

···

On Sunday, September 23, 2018 at 1:44:01 AM UTC-7, Akira Takahashi wrote:

Hi, Prof. Anubhav Jain,

Thanks for replying, and your modified code works well for us.
We would appreciate if you could add the command line argument that takes in the key filename, because all supercomputers we use require key files.

Thanks,

Akira Takahashi

2018年9月8日土曜日 8時32分53秒 UTC+9 Anubhav Jain:

A quick note - there is an extraneous space in my last message, the revision should read:

                    connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:

On Friday, September 7, 2018 at 4:32:12 PM UTC-7, Anubhav Jain wrote:

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': ' /home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

Thanks for upgrade, but actually, the connection failed for me.

It seems that 215th line in scripts/qlaunch.py should be written that

connect_kwargs=connect_kwargs) as conn:

instead of

connect_kwargs={‘password’: args.remote_password}) as conn:

, like 183rd line.

Akira Takahashi

2018年9月25日火曜日 6時08分53秒 UTC+9 Anubhav Jain:

···

I have added a new command line option --remote_keyfile, to qlaunch and released it in FWS v1.7.9. Unfortunately, I was not able to test it thoroughly.

Could you please give it a try and let me know if the new version works?

On Sunday, September 23, 2018 at 1:44:01 AM UTC-7, Akira Takahashi wrote:

Hi, Prof. Anubhav Jain,

Thanks for replying, and your modified code works well for us.
We would appreciate if you could add the command line argument that takes in the key filename, because all supercomputers we use require key files.

Thanks,

Akira Takahashi

2018年9月8日土曜日 8時32分53秒 UTC+9 Anubhav Jain:

A quick note - there is an extraneous space in my last message, the revision should read:

                    connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:

On Friday, September 7, 2018 at 4:32:12 PM UTC-7, Anubhav Jain wrote:

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': ' /home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

You are correct! I think I have fixed it in v1.8.0 according to your suggestion; please give it a try and let me know if you still have issues.

···

Best,
Anubhav

Thank you very much!

We successfully qlaunched to our remote supercomputer!

···

Our command:

qlaunch -rh XXX.XXX.XXX.ac.jp -rc /XXX/XXX/XXX/ -rkf ~/.ssh/id_rsa -ru XXX -rsh ‘/usr/bin/zsh’ rapidfire -m 2

Environment (qlauncher):

cat /proc/version

-> Linux version 4.4.0-127-generic ([email protected]) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018

python -V

-> Python 3.6.6

Environment (Remote supercomputer):

cat /proc/version

-> Linux version 3.10.0-693.21.1.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Feb 23 18:54:16 UTC 2018

python -V

-> Python 3.6.4 :: Anaconda custom (64-bit)


Best regards,

Akira Takahashi

2018年9月28日金曜日 1時42分08秒 UTC+9 ajain:

You are correct! I think I have fixed it in v1.8.0 according to your suggestion; please give it a try and let me know if you still have issues.

On Tue, Sep 25, 2018 at 6:45 PM Akira Takahashi [email protected] wrote:

Thanks for upgrade, but actually, the connection failed for me.

It seems that 215th line in scripts/qlaunch.py should be written that

connect_kwargs=connect_kwargs) as conn:

instead of

connect_kwargs={‘password’: args.remote_password}) as conn:

, like 183rd line.

Akira Takahashi

2018年9月25日火曜日 6時08分53秒 UTC+9 Anubhav Jain:

I have added a new command line option --remote_keyfile, to qlaunch and released it in FWS v1.7.9. Unfortunately, I was not able to test it thoroughly.

Could you please give it a try and let me know if the new version works?

On Sunday, September 23, 2018 at 1:44:01 AM UTC-7, Akira Takahashi wrote:

Hi, Prof. Anubhav Jain,

Thanks for replying, and your modified code works well for us.
We would appreciate if you could add the command line argument that takes in the key filename, because all supercomputers we use require key files.

Thanks,

Akira Takahashi

2018年9月8日土曜日 8時32分53秒 UTC+9 Anubhav Jain:

A quick note - there is an extraneous space in my last message, the revision should read:

                    connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:

On Friday, September 7, 2018 at 4:32:12 PM UTC-7, Anubhav Jain wrote:

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': ' /home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_    managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

Great to hear! Thanks for your help and let us know if you have any further questions.

···

On Mon, Oct 1, 2018 at 11:53 PM Akira Takahashi [email protected] wrote:

Thank you very much!

We successfully qlaunched to our remote supercomputer!


Our command:

qlaunch -rh XXX.XXX.XXX.ac.jp -rc /XXX/XXX/XXX/ -rkf ~/.ssh/id_rsa -ru XXX -rsh ‘/usr/bin/zsh’ rapidfire -m 2

Environment (qlauncher):

cat /proc/version

-> Linux version 4.4.0-127-generic ([email protected]) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018

python -V

-> Python 3.6.6

Environment (Remote supercomputer):

cat /proc/version

-> Linux version 3.10.0-693.21.1.el7.x86_64 (moc[email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Feb 23 18:54:16 UTC 2018

python -V

-> Python 3.6.4 :: Anaconda custom (64-bit)


Best regards,

Akira Takahashi

2018年9月28日金曜日 1時42分08秒 UTC+9 ajain:

You are correct! I think I have fixed it in v1.8.0 according to your suggestion; please give it a try and let me know if you still have issues.

On Tue, Sep 25, 2018 at 6:45 PM Akira Takahashi [email protected] wrote:

Thanks for upgrade, but actually, the connection failed for me.

It seems that 215th line in scripts/qlaunch.py should be written that

connect_kwargs=connect_kwargs) as conn:

instead of

connect_kwargs={‘password’: args.remote_password}) as conn:

, like 183rd line.

Akira Takahashi

2018年9月25日火曜日 6時08分53秒 UTC+9 Anubhav Jain:

I have added a new command line option --remote_keyfile, to qlaunch and released it in FWS v1.7.9. Unfortunately, I was not able to test it thoroughly.

Could you please give it a try and let me know if the new version works?

On Sunday, September 23, 2018 at 1:44:01 AM UTC-7, Akira Takahashi wrote:

Hi, Prof. Anubhav Jain,

Thanks for replying, and your modified code works well for us.
We would appreciate if you could add the command line argument that takes in the key filename, because all supercomputers we use require key files.

Thanks,

Akira Takahashi

2018年9月8日土曜日 8時32分53秒 UTC+9 Anubhav Jain:

A quick note - there is an extraneous space in my last message, the revision should read:

                    connect_kwargs={'password': args.remote_password, 'key_filename': '/home/taka/.ssh/id_rsa'}) as conn:

On Friday, September 7, 2018 at 4:32:12 PM UTC-7, Anubhav Jain wrote:

Hi Akira,

Does the code work if you modify the following line:

fireworks/scripts/qlaunch_run.py:205

which currently reads:

                        connect_kwargs={'password': args.remote_password}) as conn:

to instead read:

                        connect_kwargs={'password': args.remote_password, 'key_filename': ' /home/taka/.ssh/id_rsa'}) as conn:

If this works, we can simply add another command line argument that takes in the key filename and pass it into fabric as a connect_kwarg.

Best

Anubhav

On Tuesday, August 21, 2018 at 11:38:46 PM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

(#1):

As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.

However, actually, I doubt the problem is about connection to mongo.

I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.

I’m sorry I was not clear, or I’m still misunderstanding.

My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error

File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in

sys.exit(qlaunch())

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch

pre_non_default, args.command, non_default))

File “”, line 2, in run

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens

self.open()

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open

self.client.connect(**kwargs)

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect

passphrase,

File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth

raise SSHException('No authentication methods available')

paramiko.ssh_exception.SSHException: No authentication methods available

I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.

Actually, following small script works well for me.

import fabric

import os

host = “my.remote.supercomputer.ac.jp

key_filename="/home/taka/.ssh/id_rsa"

with fabric.Connection(

host=host,

user=“my_user”,

config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),

connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:

connect_kwargs={‘password’: None}) as conn: # fails with the same error

for r in ["/path/to/dir"]:                                            
    r = os.path.expanduser(r)                                                                         
    with [conn.cd](http://conn.cd)(r):                                                                                  
        conn.run("ls")

For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.

Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.

I’m sorry for prolonging my question and thanks for your patience.

(#2):

The updated script works well for us!

That will be very helpful for us.

We really appreciate the work.

Thanks,

2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:

Also regarding issue #1, did you try using the native Mongo tools to connect?

e.g.,

  1. Start the SSH tunnel
  1. Type “mongo --port 27018 mydb” to open a tunnel
  1. Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).

If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.

On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:

Hi Akira

Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.

On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:

Hi Prof. Anubhav

Thanks for fast reply.

However, I’m afraid that, I could not solve my problems.

As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute

ssh -i /path/to/ssh_public_key [email protected]_host_address -Nf -L 27018:localhost:27017

and execute

mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config

and on another terminal, I did

qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50

, however, it failed to connect.

( Fireworks require my password with the message:

[remote_host_address] Login password for ‘user’

, however, it failed and repeated infinitely. )

Actually, I added key_filename option to qlaunch_run.py like


for h in args.remote_host:

with settings(host_string=h, user=args.remote_user,

password=args.remote_password,

key_filename="/path/to/ssh_public_key", # added

shell=args.remote_shell):


, then, connection succeeded.

Is my command wrong, or does anything else should be configured?

I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.

Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.

Therefore, I’m afraid that, I think that I should NOT update FireWorks code.

Thanks,

Akira Takahashi

2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:

Hi Akira

  1. FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
    https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo

https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69

  1. Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.

On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:

Hi,

I am tackling execution to qlaunch to remote host.

I have two question about that.

  1. To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
  1. In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message

“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and

from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
   However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
    File "........./lib/python3.5/site-packages/fabric/context_managers.py", line 535

def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):

, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
   How should I configure my environment?

I would appreciate your help.

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.


Best,
Anubhav