COD response does not include response_fields in 'next' link

I am using the OptimadeRester to retrieve data from the COD. But I am seeing that the implementation gets the first 10 entries from the COD API response, and then it raises an error with ‘failed to parse returned data ‘species_at_sites’’. Which is something that does not happen when I query the MaterialsProject for example.

Checking a bit more in detail I see that when querying the materials project the response is the following:

"links": {"next": ""}

While the same query in the COD returns:


You can see that the ‘next’ key does not include the queried response_fields. So it fails to find them when OptimadeRester changes pages.

I just saw that COD actually includes this information in:


But the pymatgen class does not code it as such. But in any case the ‘next’ link should include the original query parameters right? Maybe I am not correct in this.

Thanks for reporting this @Oier_Arcelus, I think there is ambiguity in the specification here that should be resolved. I have raised an issue on GitHub to track this - Should pagination links include all URL query parameters (e.g. response fields)? · Issue #391 · Materials-Consortia/OPTIMADE · GitHub

Just a small, comment, I will not post it in the Github page because it is a quite workaround solution that Saulius Gražulis suggested to me.

It is possible to manually attach the ‘response_fields’ to every ‘next’ URL; this should also work in the future, even if the ‘next’ URL will get a copy of its own ‘response_fields’, since, duplicated QS parameter is not an error. Or you can check of the ‘response_fields’ is already present.

The consensus on that GitHub issue is that the next link should include response fields, and it looks like the COD have now implemented this.

I just queried the COD database and the response fields were not yet in the next field, so I think they have not implemented it yet.