COD response does not include response_fields in 'next' link

OPTIMADE
1

I am using the OptimadeRester to retrieve data from the COD. But I am seeing that the implementation gets the first 10 entries from the COD API response, and then it raises an error with ‘failed to parse returned data ‘species_at_sites’’. Which is something that does not happen when I query the MaterialsProject for example.

Checking a bit more in detail I see that when querying the materials project the response is the following:

"links": {"next": "http://optimade.materialsproject.org/v1/structures?filter=%28elements+HAS+ALL+%22Co%22%2C+%22Li%22%2C+%22O%22%29&response_fields=lattice_vectors%2Ccartesian_site_positions%2Cspecies%2Cspecies_at_sites&page_offset=20"}

While the same query in the COD returns:

"links":{"base_url":"https://www.crystallography.net/cod/optimade/v1.0.0/","first":{"href":"https://www.crystallography.net/cod/optimade/v1.0.0/structures?page_limit=10&filter=%28elements%20HAS%20ALL%20%22Co%22%2C%20%22Li%22%2C%20%22O%22%29"},"next":{"href":"https://www.crystallography.net/cod/optimade/v1.0.0/structures?page_limit=10&page_offset=10&filter=%28elements%20HAS%20ALL%20%22Co%22%2C%20%22Li%22%2C%20%22O%22%29"}

You can see that the ‘next’ key does not include the queried response_fields. So it fails to find them when OptimadeRester changes pages.

2

I just saw that COD actually includes this information in:

"query":{"representation":"/structures?filter=%28elements+HAS+ALL+%22Co%22%2C+%22Li%22%2C+%22O%22%29&response_fields=lattice_vectors%2Ccartesian_site_positions%2Cspecies%2Cspecies_at_sites"}

But the pymatgen class does not code it as such. But in any case the ‘next’ link should include the original query parameters right? Maybe I am not correct in this.

3

Thanks for reporting this @Oier_Arcelus, I think there is ambiguity in the specification here that should be resolved. I have raised an issue on GitHub to track this - Should pagination links include all URL query parameters (e.g. response fields)? · Issue #391 · Materials-Consortia/OPTIMADE · GitHub

4

Just a small, comment, I will not post it in the Github page because it is a quite workaround solution that Saulius Gražulis suggested to me.

It is possible to manually attach the ‘response_fields’ to every ‘next’ URL; this should also work in the future, even if the ‘next’ URL will get a copy of its own ‘response_fields’, since, duplicated QS parameter is not an error. Or you can check of the ‘response_fields’ is already present.

5

The consensus on that GitHub issue is that the next link should include response fields, and it looks like the COD have now implemented this.

6

I just queried the COD database and the response fields were not yet in the next field, so I think they have not implemented it yet.