Pulling Entire Databases in Optimade

OPTIMADE
1

I was trying to pull whole database such as oqmd or mp to look at the data. I used this code

from optimade.client import OptimadeClient
client = OptimadeClient(
include_providers={“oqmd”}, max_results_per_provider=-1
)
results = client.get()

I keep getting this error:
Error: Provider ‘https://oqmd.org/optimade/’ returned: (‘Connection aborted.’, ConnectionResetError(10054, ‘An
existing connection was forcibly closed by the remote host’, None, 10054, None))

Does anyone have a good suggestion in pulling full databases as chunks or something

2

If you are using the OptimadeClient then it is already pulling from the database in chunks, one page at a time. If the database connection is flaky then you can write a custom callback that caches the results of each request, see e.g., the example at Using the OPTIMADE client - OPTIMADE Python tools. Whether or not a particular database’s OPTIMADE API is set up to be used in this way (i.e., their usage terms, whether they will throttle your requests etc.) is up to the individual database and not OPTIMADE itself. This error is probably related to OQMD’s policy for that.