MPRester was working last Thursday, now not working

Hi all,

I am teaching an Integrated Computational Materials Science summer school where we are doing a Jupyter notebook on Pymatgen. This is being hosted at the University of Michigan.

Last Thursday, we ran our code as a class and it worked well. Today we are running the same code, without changing anything, and about 70% of the class cannot run the MP Query. We are getting a 403 error. This is using the Legacy MP API. Can someone help us investigate?

Here is the example code:

from pymatgen.ext.matproj import MPRester

MPR = MPRester(your_API_key)

criteria = {‘elements’:{“$in”:[“Li”, “Na”, “K”], “$all”: [“O”]}, # All compounds contain O, and must have Li or Na or K
‘nelements’:3, # Each compound should be ternary (3 elements in the composition)
‘anonymous_formula’: {“A”: 1, “B”: 1, “C”: 3}, # Each compound should have stoichiometry matching ABC3 stoichiometry (Perovskite li)
‘icsd_ids’: {‘$gte’: 0}, # Must have a non-zero ICSD ID (no hypothetical compounds)
‘e_above_hull’: {‘$lte’: 0.01}, # Energy of metastability below 10 meV/atom.
“band_gap”: {“$gt”: 1} # DFT-calculated Band Gap is greater than 1
}

    # The properties and the criteria use MaterialsProject features 
    # You can see what is queryable from the MP API documentation: 
    # https://github.com/materialsproject/mapidoc/tree/master/materials
    
    # The criteria uses mongodb query language. See here 
    # for more details: https://docs.mongodb.com/manual/reference/operator/query/

props = [“material_id”,‘pretty_formula’,‘e_above_hull’,“band_gap”,‘structure’, ‘band_structure’]

entries = MPR.query(criteria=criteria, properties=props)

print(len(entries))

for e in entries:
print(e[‘pretty_formula’],e[‘band_gap’])
#print(e)

Here is our error output:


MPRestError Traceback (most recent call last)
File ~\anaconda3\lib\site-packages\pymatgen\ext\matproj.py:253, in MPRester._make_request(self, sub_url, payload, method, mp_decode)
251 raise MPRestError(data[“error”])
→ 253 raise MPRestError(f"REST query returned with error status code {response.status_code}")
255 except Exception as ex:

MPRestError: REST query returned with error status code 403

During handling of the above exception, another exception occurred:

MPRestError Traceback (most recent call last)
Input In [3], in <cell line: 44>()
40 json.dump(new_entries, f)
41 return entries
—> 44 entries=get_bs_entries()
46 print(len(entries))

Input In [3], in get_bs_entries()
24 props = [‘structure’, “material_id”,‘pretty_formula’,‘e_above_hull’,“warnings”,“band_gap”,“band_structure”]
25 #The properties and the criteria use MaterialsProject features
26 #You can see what is queryable from the MP API documentation: mapidoc/materials at master · materialsproject/mapidoc · GitHub
—> 28 entries = MPR.query(criteria=criteria, properties=props)
29 print(len(entries))
31 #Save files are prepared in a ‘JSON’ file.
32 #Some MP objects are not JSONable, and so they must be turned into a dictionary before they can be saved.

File ~\anaconda3\lib\site-packages\pymatgen\ext\matproj.py:994, in MPRester.query(self, criteria, properties, chunk_size, max_tries_per_chunk, mp_decode)
992 count_payload = payload.copy()
993 count_payload[“options”] = json.dumps({“count_only”: True})
→ 994 num_results = self._make_request(“/query”, payload=count_payload, method=“POST”)
995 if num_results <= chunk_size:
996 return self._make_request(“/query”, payload=payload, method=“POST”, mp_decode=mp_decode)

File ~\anaconda3\lib\site-packages\pymatgen\ext\matproj.py:257, in MPRester._make_request(self, sub_url, payload, method, mp_decode)
255 except Exception as ex:
256 msg = f"{str(ex)}. Content: {response.content}" if hasattr(response, “content”) else str(ex)
→ 257 raise MPRestError(msg)

MPRestError: REST query returned with error status code 403. Content: b’{“valid_response”: false, “error”: “Invalid request method.”, “version”: {“db”: “2020_09_08”, “pymatgen”: “2022.0.8”, “rest”: “2.0”}, “created_at”: “2023-06-19T11:59:37.467667”}’

There is something else extremely unusual:

Many of us in the class have been able to get the Jupyter notebook to run when downloading Anaconda and running Jupyter Notebook locally. However, not all of us have been able to.

Notably, I have two computers here where we are running the exact same code, and the exact same API key. One computer works while the other does not. These are both computers running Anaconda and Jupyter.

Importantly, when we run it in the Jupyter environment on Nanohub, the notebook does not work.

I cannot figure out the origin of the differences that makes it such that it does not work uniformly.

The Python notebook is attached for your reference.
ICMEd Materials Informatics and Data Science Module FIXED.ipynb (821.5 KB)

Hi @wenhaosun, I’m unfortunately not able to reproduce the error in your first message. It works fine for me in a notebook on my local machine and using my legacy API key. Do you still have the issue with the invalid request method error? There might be a different reason for the request failing in Nanohub. But to help debug it, I’ll have to see the exact error message you’re getting on Nanohub.

Hi Patrick, thanks for your response. I’m still getting the error. The peculiar thing is that it works when I run the code locally from my personal laptop, but it doesn’t work when I run the exact same code on Nanohub.

I noticed someone else is having the same error as me: MPRestError status code 403

Maybe you can try making a free Nanohub account and running it from there? To do so:

→ Jupyter Notebook
→ Launch app (but make an account first)
→ Upload my notebook and try to execute it …?

Thanks!

@Wenhao_Sun good to hear that the code works on your local machine. Before we dig deeper, can you share which versions of pymatgen and mp-api you’re using? And confirm that they’re the same on your local machine and Nanohub? @Matthew_Greenberg would you mind sharing the code versions that are working for you?

I have a suspicion why Nanohub might be getting blocked but I’d want to confirm the above basics first before exploring an exception for Nanohub.

Hi Patrick, the pymatgen version is: 2022.2.10. Python is 3.8.

We are actually using the legacy query. (Nanohub doesn’t have mp-api installed in their environment).

I have seen some threads here where excessive queries get blocked. I am wondering if it’s possible that we had 30 students performing a large-ish query (7500 entries) all within 2 minutes of each other, which got the Nanohub Jupyter notebook blocked?

Oh sorry, the Pymatgen 2022.2.10 is for Nanohub. On my local machine the Pymatgen is 2023.5.10.

Best,
Wenhao Sun

Does the error still occur on Nanohub after updating pymatgen to the latest version? Also, double-check that you’re using your legacy API key. That’ll avoid taking a detour through our next-gen servers and directly connect to legacy.materialsproject.org.

Legacy doesn’t work for me (tried to create a new account too).

import requests

data = {
    'criteria': {
        'elements': {'$in': ['Li', 'Na', 'K'], '$all': ['O']},
        'nelements': 2,
    },
    'properties': [
        'formula',
        'formation_energy_per_atom',
    ]
}
r = requests.post('https://materialsproject.org/rest/v2/query',
                 headers={'X-API-KEY': 'XXX'},
                 data={k: json.dumps(v) for k,v in data.items()})
response_content = r.json() # a dict
print(response_content)
{'valid_response': False, 'error': 'Invalid request method.', 'version': {'db': '2020_09_08', 'pymatgen': '2022.0.8', 'rest': '2.0'}, 'created_at': '2023-06-28T08:30:43.994478'}```

@sasha does going to legacy.materialsproject.org directly work for you? Did you try using MPRester in the latest pymatgen instead of the requests library?

Hi @tschaume , yes legacy website works for me.

Did you try using MPRester in the latest pymatgen instead of the requests library?

I can’t install packages in this env.

1 Like

Just to clarify, I meant setting https://legacy.materialsproject.org/rest/v2/query in your post request explicitly.

Yes, that works, thanks a lot.

Dear @tschaume
I am a graduate student studying Solid Mechanics at Wuhan University. Currently, my research focuses on affine deformation of crystals. I have come across Materials Project as a valuable platform that provides extensive material properties data, including elastic data, which would greatly benefit my research.

I am particularly interested in the existing elastic data available in Materials Project and would like to inquire if it is possible to obtain the relevant files for computing elastic data, such as the OUTCAR files. I am curious to know if the website offers access to these files and where I can find them.

Thank you very much for your time and consideration. I am looking forward to your response.

Best regards,

Zhihe Chen

@c1chenzhihe This is a good question but it would have probably been better posted as a new topic :slight_smile: We are working on providing more of our raw output data and intermediate build products to our users. Please look out for announcements in the near future through our newsletter or the notifications system on our website.

Thread closed due to inactivity, please open a new thread to address related issues.