Running the parser tests locally

Pavel_Ondracka · February 21, 2023, 10:23am

I’ve been working on some minor updates to the LOBSTER parser (with tests and I wanted to test it locally). It however fails with lot of errors.

I probably do it in some wrong way, so looking for some advice how to make it work, maybe from @ladinesa ?

My setup:

create a new clean environment (python3.7) with virtualenv, activate it
clone nomad, checkout latest develop and init all submodules
install nomad with pip install -e nomad
clone workflow-parsers
install with pip install -e workflow-parsers
go to workflow-parsers/tests and run pytest

pytest
================================================ test session starts =================================================
platform linux -- Python 3.7.16, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/ondracka/software/workflow-parsers/tests
plugins: anyio-3.6.2
collected 0 items / 10 errors                                                                                        

======================================================= ERRORS =======================================================
________________________________________ ERROR collecting test_aflowparser.py ________________________________________
test_aflowparser.py:21: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:334: in <module>
    north = North()
pydantic/main.py:339: in pydantic.main.BaseModel.__init__
    ???
pydantic/main.py:1038: in pydantic.main.validate_model
    ???
pydantic/fields.py:840: in pydantic.fields.ModelField.validate
    ???
pydantic/fields.py:1121: in pydantic.fields.ModelField._apply_validators
    ???
pydantic/class_validators.py:280: in pydantic.class_validators._generic_validator_cls.lambda3
    ???
../../nomad/nomad/config.py:328: in load_tools
    with open(v, 'rt') as f:
E   FileNotFoundError: [Errno 2] No such file or directory: 'dependencies/nomad-remote-tools-hub/tools.json'
_________________________________________ ERROR collecting test_asrparser.py _________________________________________
test_asrparser.py:25: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
_______________________________________ ERROR collecting test_atomateparser.py _______________________________________
test_atomateparser.py:22: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
_______________________________________ ERROR collecting test_elasticparser.py _______________________________________
test_elasticparser.py:22: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
______________________________________ ERROR collecting test_fhivibesparser.py _______________________________________
test_fhivibesparser.py:22: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
_______________________________________ ERROR collecting test_lobsterparser.py _______________________________________
test_lobsterparser.py:23: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
____________________________________ ERROR collecting test_mofstructuresparser.py ____________________________________
test_mofstructuresparser.py:22: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
_______________________________________ ERROR collecting test_phonopyparser.py _______________________________________
test_phonopyparser.py:22: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
___________________________________ ERROR collecting test_quantum_espresso_epw.py ____________________________________
test_quantum_espresso_epw.py:21: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
__________________________________ ERROR collecting test_quantum_espresso_phonon.py __________________________________
test_quantum_espresso_phonon.py:21: in <module>
    from nomad.datamodel import EntryArchive
../../nomad/nomad/datamodel/__init__.py:74: in <module>
    from nomad.metainfo import Environment
../../nomad/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
../../nomad/nomad/metainfo/metainfo.py:37: in <module>
    from nomad.config import process
../../nomad/nomad/config.py:282: in <module>
    class North(NomadSettings):
../../nomad/nomad/config.py:324: in North
    @validator('tools', pre=True, always=True)
pydantic/class_validators.py:88: in pydantic.class_validators.validator.dec
    ???
pydantic/class_validators.py:147: in pydantic.class_validators._prepare_validator
    ???
E   pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, set `allow_reuse=True`
============================================== short test summary info ===============================================
ERROR test_aflowparser.py - FileNotFoundError: [Errno 2] No such file or directory: 'dependencies/nomad-remote-tools-hub/tools.json'
ERROR test_asrparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_atomateparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_elasticparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_fhivibesparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_lobsterparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_mofstructuresparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_phonopyparser.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_quantum_espresso_epw.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
ERROR test_quantum_espresso_phonon.py - pydantic.errors.ConfigError: duplicate validator function "nomad.config.North.load_tools"; if this is intended, s...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 10 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================= 10 errors in 2.37s =================================================

BTW I have the dependencies/nomad-remote-tools-hub/tools.json file in the nomad directory. So what am I missing?

mscheidgen · March 2, 2023, 6:51am

There currently is a known bug that you have to be in the project root (the nomad project, not the parser project) to run anything that reads the config.

There is a fix in the making: Fixed missing tools.json problem. (!1128) · Merge requests · nomad-lab / nomad-FAIR · GitLab

Pavel_Ondracka · March 2, 2023, 7:31am

Thanks @mscheidgen I can confirm cherry-picking a49de4ff on top of the latest develop allows the pytest to run from the parsers project root.