Dear NOMAD developers,
while processing the “Atom Probe Microscopy” sample dataset bundled with NOMAD, the worker container throws the following error:
nomad_oasis_worker | 2024-01-30T10:38:29.455508608Z ERROR nomad.processing 2024-01-30T10:38:29 could not convert to nxs
nomad_oasis_worker | 2024-01-30T10:38:29.455551049Z - exception: Traceback (most recent call last):
nomad_oasis_worker | 2024-01-30T10:38:29.455554023Z File "/usr/local/lib/python3.9/site-packages/nomad/datamodel/metainfo/eln/nexus_data_converter.py", line 215, in normalize
nomad_oasis_worker | 2024-01-30T10:38:29.455555947Z convert(**converter_params)
nomad_oasis_worker | 2024-01-30T10:38:29.455557548Z File "/usr/local/lib/python3.9/site-packages/pynxtools/dataconverter/convert.py", line 243, in convert
nomad_oasis_worker | 2024-01-30T10:38:29.455559312Z helpers.add_default_root_attributes(data=data, filename=os.path.basename(output))
nomad_oasis_worker | 2024-01-30T10:38:29.455561121Z File "/usr/local/lib/python3.9/site-packages/pynxtools/dataconverter/helpers.py", line 613, in add_default_root_attributes
nomad_oasis_worker | 2024-01-30T10:38:29.455562807Z f"blob/{get_nexus_version_hash()}"
nomad_oasis_worker | 2024-01-30T10:38:29.455564376Z File "/usr/local/lib/python3.9/site-packages/pynxtools/__init__.py", line 83, in get_nexus_version_hash
nomad_oasis_worker | 2024-01-30T10:38:29.455566012Z version = re.search(r"g([a-z0-9]+)", get_nexus_version())
nomad_oasis_worker | 2024-01-30T10:38:29.455567631Z File "/usr/local/lib/python3.9/site-packages/pynxtools/__init__.py", line 73, in get_nexus_version
nomad_oasis_worker | 2024-01-30T10:38:29.455569273Z return format_version(get_nxdl_version())
nomad_oasis_worker | 2024-01-30T10:38:29.455570893Z File "/usr/local/lib/python3.9/site-packages/pynxtools/definitions/dev_tools/globals/nxdl.py", line 20, in get_nxdl_version
nomad_oasis_worker | 2024-01-30T10:38:29.455572587Z with open(get_nxdl_version_file(), "r") as fh:
nomad_oasis_worker | 2024-01-30T10:38:29.455574137Z FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/site-packages/pynxtools/definitions/NXDL_VERSION'
nomad_oasis_worker | 2024-01-30T10:38:29.455575740Z - exception_hash: ZCpFcqcmc1Gp4jwM9sg2GJ-PMlYV
nomad_oasis_worker | 2024-01-30T10:38:29.455577261Z - nomad.commit:
nomad_oasis_worker | 2024-01-30T10:38:29.455578713Z - nomad.deployment: oasis
nomad_oasis_worker | 2024-01-30T10:38:29.455580213Z - nomad.entry_id: 6irmz_m4tJDavABlsVowZ-kRrqc4
nomad_oasis_worker | 2024-01-30T10:38:29.455581694Z - nomad.mainfile: apm.nxs
nomad_oasis_worker | 2024-01-30T10:38:29.455583232Z - nomad.processing.logger: nomad.processing
nomad_oasis_worker | 2024-01-30T10:38:29.455584709Z - nomad.processing.normalizer: MetainfoNormalizer
nomad_oasis_worker | 2024-01-30T10:38:29.455586129Z - nomad.processing.parser: parsers/archive
nomad_oasis_worker | 2024-01-30T10:38:29.455587518Z - nomad.processing.proc: Entry
nomad_oasis_worker | 2024-01-30T10:38:29.455588935Z - nomad.processing.process: process_entry
nomad_oasis_worker | 2024-01-30T10:38:29.455596841Z - nomad.processing.process_status: RUNNING
nomad_oasis_worker | 2024-01-30T10:38:29.455598442Z - nomad.processing.process_worker_id: THYmO_-WRKK8Ycds2V8soA
nomad_oasis_worker | 2024-01-30T10:38:29.455599976Z - nomad.processing.step: MetainfoNormalizer
nomad_oasis_worker | 2024-01-30T10:38:29.455601456Z - nomad.service: unknown nomad service
nomad_oasis_worker | 2024-01-30T10:38:29.455602908Z - nomad.upload_id: SKikUN6QTeGSVNUIOq-_kQ
nomad_oasis_worker | 2024-01-30T10:38:29.455604359Z - nomad.version: 1.2.2.dev295+g2e611aff1
And indeed the file “NXDL_VERSION” seems to be missing in the container:
nomad@a377583d0b83:/app$ ls -la /usr/local/lib/python3.9/site-packages/pynxtools/definitions/
total 96
drwxr-xr-x. 9 root nomad 181 Dec 25 22:15 .
drwxr-xr-x. 7 root nomad 140 Dec 25 22:15 ..
drwxr-xr-x. 2 root nomad 4096 Dec 25 22:15 applications
drwxr-xr-x. 2 root nomad 4096 Dec 25 22:15 base_classes
drwxr-xr-x. 2 root nomad 12288 Dec 25 22:15 contributed_definitions
drwxr-xr-x. 9 root nomad 138 Dec 25 22:15 dev_tools
drwxr-xr-x. 3 root nomad 40 Dec 25 22:15 impatient-guide
drwxr-xr-x. 3 root nomad 20 Dec 25 22:15 manual
-rwxr-xr-x. 1 root nomad 51563 Dec 25 22:15 nxdl.xsd
-rw-r--r--. 1 root nomad 17659 Dec 25 22:15 nxdlTypes.xsd
drwxr-xr-x. 3 root nomad 123 Dec 25 22:15 utils
Is this file supposed to be created during startup or should it be present right from the beginning?
I checked the gitlab issues (Issues · nomad-lab / nomad-FAIR · GitLab) but did not find any related issue. A hot fix would be to create this file outside and mount it into the container.
Thank you for pointing out this issue on the development branch! We will fix it soon. Please note that in the meantime, one can check out the previous version of the APM example in our production system: https://nomad-lab.eu/prod/v1/gui/
with /hotfix/nexus-version.txt being the output of git describe applied on the nexus_definitions repo, NXcanSAS-1.1-1681-g60500ddc
Now, processing of the APM example succeeds but throws warnings:
Don’t know whether this is related to my “forged” version designator (i.e. processing tried to extract the name and version from the version string) or whether this is unrelated.
If someone could maybe give me a hint what to write into nexus-version.txt or NXDL_VERSION as a short-time fix…
A short feedback from my side, Markus Kühbach the developer of the atom probe functionalities, in NOMAD OASIS: It is great that you are testing these functionalities and show interest in this work. The comments that the nomad worker emits are not critical, the NeXus data artifact should have been generated and be explorable in your upload. We have been working on a complete refactoring of the atom probe capabilities to support even more data formats and support richer metadata, not only for Cameca instruments, these feature branches are already implemented and tested but currently in review, after which they will be merged into the NOMAD development branch, planned mid / end Q1 this year.
For questions about atom probe and electron microscopy functionalities of NOMAD OASIS you are, like everybody else is, very much invited to get in contact with me directly.
Dear Markus Kühbach,
thank you very much for the clarification. Frankly speaking, i am not especially interested in the APM functionality. My main task is to set up and operate an OASIS and hence i just tested a couple of the sample datasets after deploying the container set. I am not even sure whether the APM functionality will be used by our users. On the other hand, if the NeXus parser is inoperable, all kinds of functionality will be broken.