More parsers for Quantum ESPRESSO-based packages

Dear Nomad developers,

I have some questions about some more potential parsers for Quantum ESPRESSO (QE). I did recently quite a lot of phonons calculation with the PHonon package (User's Guide for the PHonon package) from Quantum ESPRESSO and I would like to upload my results to NOMAD. I mostly need just for the calculation mainfiles to be recognized and few simple information parsed so I can get a DOI pointing to at least a bit searchable/understandable archive. Similarly to the LOBSTER and OpenMX parsers before (that are now integrated thanks to @ladinesa), I’m willing to contribute some code of needed, but most likely I will need some help as the calculations are a bit beyond the simple DFT scheme so I might need some advice how to properly integrate this into NOMAD.

The PHonon package differs a bit from the current phonopy package since it uses Density functional perturbation theory (DFPT) instead of the standard phonopy finite differences approach, the output is basically a list of vibrational frequencies for a bunch of q-points. So that I guess should go into vibrational_frequencies subsection, but that again is not q-point dependent? Here is some example output: ph.out (773.9 KB) Its also not 100% clear to me if the method here should still be DFT and this will be just a workflow parser or if this needs its own new method…

Another program I would like to have a simple parser for (as I have also few cases to upload) is the EPW https://docs.epw-code.org/ (also part of the QE package) it calculates the quantities related to electron phonon interaction. It uses the data from QE PHonon and PW (the main DFT package) and Wannier90 and calculates properties related to the electron-phonon interaction (electron-phonon coupling strenght, anisotropic Eliashberg spectral function, phonon or electrons linewidths and lifetimes from the interaction, etc…). This doesn’t also fit into the current metainfo definitions at all… So yeah suggestions how to proceed welcomed.

@ladinesa Please have a look at this.

As of now quantum espresso in nomad refers to the PWSCF module. I actually need to discuss how to manage this with our “domain experts”. For now, we put the Phonon module under workflowparsers. We do not parse the full method now as the metainfo is not yet defined. We can simply put under method.electronic.method = ‘DFPT’. We do something similar to EPW. We put it under workflowparsers. I surmise this is a single_point workflow. We simply extend the SinglePoint metainfo, with the code-specific names for the properties for now. Should I create an initial parser for you @Pavel_Ondracka to extend?

Great, if you can create a stub parsers that get the general metainfo scheme at least semi-correctly that would be awesome. Than I’ll try to extend it to parse some actual result quantities. EPW should be single_point I assume, here is some example output file just so you have something for the parser matcher:
epw.out (777.2 KB)

@ladinesa Do you have any time estimate on the stub parsers?

@mscheidgen On a related note. My most common NOMAD use case is to publish calculations for a manuscripts as an open data. I really like NOMAD, unfortunately every now and than (like previously with the OpenMX and LOBSTER and now with the QE-based calculations) I would like to upload something that doesn’t have a parser. In that regards it is slightly unfortunate, that the datasets are composed of parsed entries and not of files… Now if I upload just results that have no parser I have nothing to create the dataset from.

You could already upload *.archive.json files that use our schema. Basically provide the parser results your self. We work on this more for experiment data and there are not great examples for this. There might be a few complications left and right, but if you have a good case, we could try something.

The upcoming nomad v1.1.0 will also support custom schemas to add data that require a novel data structure.

Hi Pavel,
Sorry for the rather late reply. I already added an initial version of the parser in

Please have a look at the branch qe-parsers and please tell me if changes have to be made.

1 Like

Hi Alvin,

thanks for the work, it is much appreciated. I think the parser is doing quite fine job. It was able to handle most of the calculations I did try so far.

It fails to parse the cell vectors, atomic positions don’t have the proper units, smearing also isn’t in SI units and few other nitpicks. It also needs more checks for partial runs (QE supports saving checkpoints to drive and continuation runs, which I use extensively because the HPC centers I use have quite short job max time quotas…). But I’ll send a pull request to fix those during the week. BTW there should be quite a lot of QE phonon mainfiles in NOMAD if you want more testcases (currently undetected ofc).

If you find some time to create a similar parser for the QE EPW code that would be awesome (or at least stub it and than I’ll try to take care of the rest).

I made a PR with some small fixes for the EPW parser, and with that I think both the QE EPW and phonon parsers are ready for master.

Thanks a lot for the PR. I see that it is already merged.

@ladinesa Did all the hard work, I just made few tweaks here and there.

When it all hits production could you please trigger reprocessing for NOMAD ?

But just the uploads with quantum espresso calculations?

Actually the link was just to a single upload_id: apfw2NtURz2BKNzlNpib_A

I am finally getting to reprocessing this. The upload has new mainfiles matches. Is this indented by the fixes or a potential problem? Do you want me to add the new entries or keep the published set of entries as they are?

That is expected, the new entries should be from the new quantum-espresso-phonon parser that was not supported before. The rest was processed fine previously (the phonopy parsing crashes early but that is Phonopy parsing fails with ModuleNotFoundError("You need to install python-h5py.") · Issue #34 · nomad-coe/nomad · GitHub). I’ll check later when its done and get a new DOI including the new entries. Its actually just on time, we are submitting in few days.

Thanks.

I now reprocessed apfw2NtURz2BKNzlNpib_A. There are new entries now; it was 200 something before, now its 347. Please have a look.

@ladinesa Some of the phonopy.yaml processing are failing with an exception in the workflow parser: NOMAD

Actually, I tried to add the new entries into a dataset and have this issue:
Process edit_upload_metadata failed: KeyError: ‘parsers/quantumespressophonon’

I haven’t forgot about this issue and I hope I will have time to look at this soon!

There is something seriously wrong with the new QE parsers, or at least with the version that is deployed at the production server.

I have some cases like this
ph-27381627.out (178.5 KB)
that are not detected when I upload them to production, but work fine when I do nomad parse manually using a develop branch locally.

The strange thing is that some cases which were matched previously are not matched anymore, for example NOMAD was parsed by parsers/quantumespressophonon (uploaded 7/15/2022, 6:24:55 PM and last processing time 11/11/2022, 3:28:56 PM) however if I download the mainfile and reupload, it won’t be matched.

Also looking at that old entry, for some reason while it was parsed with the correct parser, the program name says “Quantum Espresso EPW”. If I download it and parse locally, I get correct program name “Quantum Espresso Phonon”.

@ladinesa any ideas why everything works locally and fails so badly in production?

prod does not yet include the qe workflow parsers but staging does.