Hi @Minju ,
was away for a few days so I couldn’t reply right away, hopefully this still helps
As long you know roughly when you began your research you should be able to find the correct version of each collection you need from AWS: MP build bucket explorer (collections sub-directory)
To get more accurate than just the YYYY-MM-DD time stamps in the explorer you can also cross-reference the database change log for the dates + approximate times each version went live: MP Database Version change log
Since you mention that your workflow originated about a year ago, the following will apply to the <version>/<collection_name> versions you see in the explorer. Using the April 2025 summary collection as a concrete example:
For all of the below you’ll likely want to additionally filter on deprecated == False since the mp-api client by default excludes deprecated materials.
Step 0: Check if the manifest file has the filter fields you need:
>>> import pandas as pd
>>> df = pd.read_json("s3://materialsproject-build/collections/2025-04-10/summary/manifest.jsonl.gz", lines=True, compression="gzip", storage_options={"anon": True})
>>> df.columns
Index(['theoretical', 'formation_energy_per_atom', 'deprecated', 'material_id',
'formula_pretty', 'nelements', 'last_updated', 'band_gap',
'energy_above_hull', 'e_total', 'density', 'e_electronic', 'task_ids',
'total_magnetization', 'symmetry_number', 'sourced_from_path'],
dtype='object')
If you just need band_gap and energy_above_hull that might be enough and you can skip the remaining steps. You can repeat this with the other collections as needed (electronic_structure, thermo, etc.)
Step 1: Just pull the whole collection via the AWS cli: aws s3 --no-sign-request sync s3://materialsproject-build/collections/2025-04-10/summary/ summary/
Step 2: Filter:
if you have a decent amount of memory and don’t want to think about it, materialize everything in one go:
import gzip
import json
import os
from itertools import chain
import pandas as pd
docs = []
for root, dirs, files in os.walk("summary"):
for file in files:
if "manifest" not in file:
with gzip.open(os.path.join(root, file), "rb") as f:
docs.append(
[json.loads(line) for line in f.read().splitlines()],
)
docs = list(chain.from_iterable(docs))
df = pd.DataFrame(docs)
# filter via pandas ...
Or if you’re resource constrained just pipeline your way through:
import gzip
import json
import operator
import os
from functools import reduce
def read_docs(dir_path):
for root, dirs, files in os.walk(dir_path):
for file in files:
if "manifest" not in file:
with gzip.open(os.path.join(root, file), "rb") as f:
for line in f.read().splitlines():
yield json.loads(line)
# some vals are nullable
def safe_filter(docs, field, operator, target_val):
for doc in docs:
if doc[field] is not None and operator(doc[field], target_val):
yield 1
# or do something useful and yield the actual entry, ;shrug;
# yield doc
docs = read_docs("summary")
e_above_hull_gt_5_meV = reduce(
operator.add,
safe_filter(docs, "energy_above_hull", operator.gt, 0.005),
0,
)
print(f"Entries above hull by 5 meV/atom: {e_above_hull_gt_5_meV}")
# -> "Entries above hull by 5 meV/atom: 114914"
^All this only applies to the jsonl datasets
edit: Additionally, in your paper it would be good to mention which db version your data is from for reproducibility for readers 
This is mostly FYI/general info: And as you said, with the latest version (v2026.04.13) the storage format (and by necessity some of the schemas) have changed. Prior to v2026.04.13 all the collections were stored in the JSON Lines (jsonl) format, going forward everything will be available as parquet datasets as the parquet format is more performant in the context in which MP delivers/uses large datasets.
In the course of the parquet change, the partitioning of the collections have been inverted as well. So the jsonl datasets were <version>/<collection_name>, the parquet datasets are now <collection_name>/.
Eventually all the old db versions will be migrated to the parquet format and be backfilled into the existing parquet datasets.
Which will enable stuff like the following (disclaimer: only works for version 2026-04-13):
import pyarrow.compute as pc
import pyarrow.dataset as ds
import pyarrow.fs as fs
anon_s3_fs = fs.S3FileSystem(anonymous=True, region="us-east-1")
summary_dataset = ds.dataset(
"materialsproject-build/collections/summary/",
partitioning="hive",
filesystem=anon_s3_fs,
)
# sub in "2026-04-13" for version for this to use the latest version
# and actually work
expr = (pc.field("version") == "2025-04-10") & (pc.field("energy_above_hull") > 0.005)
e_above_hull_gt_5_meV = summary_dataset.count_rows(filter=expr)
Which is much more ergonomic (and performant) than the jsonl stuff. Passing the version to the mp-api client is also on the roadmap for integration once we backfill the parquet versions of the previous versions.
