MPRester through proxy server

Hello,
I’m trying to use the MPRester in Pymatgen from a national lab computing resource (the login nodes), which proxies all HTTPS connections. However, attempts to retrieve Materials Project data produce the errors:

requests.exceptions.ProxyError: HTTPSConnectionPool(host=‘materialsproject.org’, port=443): Max retries exceeded with url: /rest/v2/query (Caused by ProxyError(‘Cannot connect to proxy.’, OSError(‘Tunnel connection failed: 403 Forbidden’,)))

pymatgen.ext.matproj.MPRestError: HTTPSConnectionPool(host=‘materialsproject.org’, port=443): Max retries exceeded with url: /rest/v2/query (Caused by ProxyError(‘Cannot connect to proxy.’, OSError(‘Tunnel connection failed: 403 Forbidden’,)))

I have the URL of the lab’s HTTPS proxy server; is there a way I can connect to Materials Project though this proxy?

Thanks

1 Like

Hi,
I always do this. You should be able to connect through proxy if your Unix variable https_proxy and http_proxy is set properly. Check https://www.golinuxcloud.com/set-up-proxy-http-proxy-environment-variable/

From python you can do this also by

In [1]: import os
        os.environ['http_proxy'] = "http://user:[email protected]:port" 
        os.environ['https_proxy'] = "https://user:[email protected]:port" 

Hope it helps.
Regards
Sandip

Thanks for the advice, however, I have already done this by setting the proxy in my .bash_profile. I can use utilities such as pip through the proxy server, it’s only Materials Project which is not accepting the connection.

Which national lab computing resource? I haven’t seen this before, imagine it might be difficult to investigate without interactive access but I can ask a colleague to investigate.

The Badger cluster at Los Alamos National Laboratory. I’m currently working with LANL tech support to try to figure it out, but so far we haven’t made any progress.

I’d start with using the requests library and trying to ping https://materialsproject.org/rest/v2/api_check. If you can find a way to get that to work via a proxy in Python then we’d have a route forward. I’m not sure if I know anyone with access to Badger right now but I’ll ask around.

I tried:
import requests
requests.get(‘https://materialsproject.org/rest/v2/api_check’, proxies= [LANL proxy info])
and got the same 403 error message. Trying to use the UNIX curl utility with https://www.materialsproject.org (or any other website, for that matter) also gives a 403 error. I can still do some things through the proxy, like use pip to install packages, but it might be a LANL-specific issue if it’s blocking such a wide variety of sites and functions. I’ll update this thread if I can figure things out with LANL tech support.

Yes, perhaps a LANL specific issue, it could be that https is causing the issue(?) unfortunately I don’t think we support requests over http alone.

1 Like

It’s actually likely HTTPS. You can’t have a https connection out from the client, the proxy has to initiate that. Basically your request would be to http://materialsproject.org/ and the proxy server has to then convert that to a HTTPS request when it initiates. HTTPS basically guarantees that no one is in the middle of the client and the server, but you’re putting a proxy server in between so it has to act as the HTTPS client.