Possible rework for object storage in addtional stores

jshen · November 28, 2022, 8:30pm

So I’m noticing that there are lots of null entries in the addtional_store database of the JOB_STORE.

Having looked at it super carefully, but it looks like its triggering a db write here regardless of the presence of actual data to write:

github.com

materialsproject/jobflow/blob/b1628dcacaa4d592d7d4baa0e69ae8f456101582/src/jobflow/core/store.py#L296-L328


      
          for doc in docs:
              doc = jsanitize(doc, strict=True, allow_bson=True)
              dict_docs.append(doc)
          
          
    if save_keys:
                  locations = []
                  for store_name, store_save in save_keys.items():
                      for save_key in store_save:
                          locations.extend(find_key(doc, save_key, include_end=True))
          
          
            objects = [get(doc, list(loc)) for loc in locations]
                      object_map = {
                          tuple(k): o for k, o in zip(locations, objects) if o is not None
                      }
                      object_info = {
                          k: _get_blob_info(o, store_name) for k, o in object_map.items()
                      }
                      update_in_dictionary(doc, object_info)
          
          
            # Now format blob data for saving in the data_store

This file has been truncated. show original

Proposal:
Since the data_store in atomate2 only serializes python objects, we should:

Check if there is an object
~~Store the class (??headers??, the @ClassName stuff) in the index store while we are at it.~~ NVM just realized it’s already doing that.

Let me know what you think.