Great, thanks for the hard work. Adopting the entry points will pay off later down the road.
So I think you are missing installing the runschema and nomad_simulations. For that, you will need to add them. For adding them, it is as simple as adding these lines to the pyproject.toml in your plugin:
dependencies = [
"nomad-lab>=1.3.6.dev1", # import using `from nomad...`
"nomad-schema-plugin-run>=1.0.1", # import using `from runschema...`
"nomad-simulations>0.0.7", # import using `from nomad_simulations...`
]
And instead of pip installing nomad-lab in your environment using the first command, you can do:
This command will read the pyproject.toml and install the packages defined in dependencies and the optional ones in otherflags that you can define in the pyproject as well (see e.g. dev in this pyproject.toml).
Can you try this and let us know?
And in short as an explanation: you basically need to add as many dependencies as packages you are going to use. Like NOMAD uses heavily plugins, that includes nomad-lab, runschema and nomad-simulations.
Thanks @JosePizarro , importing from runschema works now. But how do I create custom quantities/subsections? For example I’m trying to create a quantity called current_density in the custom subsection run.calculation.currents. Since there’s no metainfo folder in the parser repo anymore, I changed schema_package.py into something like this:
from nomad.metainfo import Quantity, SchemaPackage, Section, MSection, SubSection
from runschema.calculation import Calculation
configuration = config.get_plugin_entry_point(
'lightforge_v2.schema_packages:schema_package_entry_point'
)
m_package = SchemaPackage()
class Currents(MSection):
m_def = Section(Validate=False)
current_density = Quantity(type=np.float64)
class LightforgeCalculation(runschema.calculation.Calculation):
m_def = Section(validate=False)
currents = SubSection(sub_section=Currents.m_def, repeats=False)
m_package.__init_metainfo__()
And then in parser.py:
from runschema.run import Run
from runschema.calculation import Calculation
from lightforge_v2.schema_packages.schema_package import Currents
sec_run = archive.m_create(Run)
sec_calc = sec_run.m_create(Calculation)
sec_currents = sec_calc.m_create(Currents) # error: There's no subsection to hold a schema_package.Currents section in runschema.calculation.Calculation section
But this throws the error 1 line above. How do I fix this? Thank you in advance.
Best
Fabian
So the error is trying to create a Currents section under Calculation. You should do it under LightforgeCalculation instead:
from runschema.run import Run
from lightforge_v2.schema_packages.schema_package import (
Currents,
LightforgeCalculation,
)
sec_run = archive.m_create(Run)
sec_calc = sec_run.m_create(LightforgeCalculation)
sec_currents = sec_calc.m_create(Currents)
This is because the class which has the definition of currents is LightforgeCalculation (runschema.Calculation does not have this attribute defined).
I also have some suggestions for you (you can consider them or just keep doing what you feel more comfortable, this is entirely up to you ):
No need to add m_def = Section(validate=False). In the m_def you can add certain “definitions” (in the NOMAD sense, “quantities that do not change at parsing time”), but turning off validation should not be one of them. Actually, I am not 100% sure that definition is working as intended (i.e., I think it just validates it, and it does not matter if you put validate=False or True). So it might be a deprecated feature already.
You can directly instantiate classes and assigns them to the attribute of a parent class, instead of using m_create. This is more appealing if you are used to work with Python and let’s you avoid using a criptic method very NOMAD-specific (so if someone takes this later, they will probably understand better if you don’t use such methods), so it could look like:
from runschema.run import Run
from lightforge_v2.schema_packages.schema_package import (
Currents,
LightforgeCalculation,
)
run = Run()
archive.run.append(run)
currents = Currents()
# populate currents here
calculation = LightforgeCalculation(currents=currents)
Note you can assign directly (calculation.currents = currents) or append (archive.run.append(Run())) depending whether the sub-section repeats or not (you can also assign lists directly, like archive.run = [Run()]).
And also some suggestions that I found out after working a lot with schemas, so from a personal experience (and I do not want to impose my style into yours, so feel free to accept them or not at will):
3. I would keep singular/plurals naming a bit more consistent. Using singular attribute names for sub-sections that repeats=False while plural when repeats=True.
4. A similar thing applies for attributes of a sub-section. E.g., if a class is called Currents and has an attribute for the density, you can simply call that attribute density, instead of current_density. You can imagine that at the end you will access data like ...current.density instead of ...current.current_density. Just a small detail, but it might also help you identifying common patterns (sections or quantities) across your sections/classes.
This gives me the error: "no subsection to hold schema_package.LightforgeCalculation section in runschema.run.Run section. So intuitively to solve this I introduce a LightforgeRun class in schema_package.py like this:
from runschema.run import Run
from runschema.calculation import Calculation
class Currents(MSection):
current_density = Quantity(type=np.float64)
class LightforgeCalculation(Calculation):
currents = SubSection(sub_section=Currents, repeats=False)
class LightforgeRun(Run):
lightforge_calculation = SubSection(sub_section=LightforgeCalculation, repeats=False)
and then import the new class LightforgeRun into parser.py and do this:
sec_run = archive.m_create(LightforgeRun)
sec_calc = sec_run.m_create(LightforgeCalculation)
# etc.
then I just get the similiar error: no subsection to hold schema_package.LightforgeRun section in nomad.datamodel.datamodel.EntryArchive section. How do I solve this error (or the first error)?
Best
Fabian
Weird… Might be an issue of m_create. Can you try with instantiating and appending classes as I suggested?
The underlying concept here is that we can use polymorphism to assign classes to attributes when they inherit from a more abstract class. I.e., if a class has a sub_section(sub_section=AbstractClass), you can directly assing other classes which inherit from that AbstractClass.
however the archive doesn’t show current_density = 9 anywhere
Ok, maybe, can you try doing nomad parse <path-to-the-file> --show-archive > test.json in a virtual environment where you installed your parser? This will save the archive in a test.json, and we can see whether the section Currents(current_density=9) was indeed created.
If you are trying it out in an OASIS, it might be that the GUI artifacts are not being generated. You can try reloading the page pressing F5. Otherwise, I defer to @laurih to help you with the OASIS in case the issue is there.
and also there are 2 warning messages:
These 2 warnings are completely normal, I’d say: they relate with the fact that you don’t have the System information under archive.run[0].system. So you can safely forget about them, or try to parse the System information if you have access to it. The warnings are coming from normalizers, i.e., a layer applied after parsing and responsible to apply logic to a bunch of different parsers.
I order to end up in the final archive, your instance of LightForgeCalculation must be assigned as a subsection somewhere.
I’m not familiar with your schema, but I would guess that it should be a part of the Run section instance, so something like:
run = Run()
archive.run.append(run)
currents = Currents()
calculation = LightforgeCalculation(currents=currents)
currents.current_density = 9
# Here we add the calculation to 'run'
run.calculations.append(calculation)
Indeed, you need to append LightforgrCalculation to run.
It is the line Lauri shared, but in singular (this is what I said before as a recommendation, a repeated subsection should be plural, but in this case is not and it feels a bit unnatural):
Thank you @JosePizarro and @laurih , the parser works locally now. We have some problems in getting the parser into our oasis. From the docs Install plugins - Documentation it seems that Option 2 is the right one for us, so my parser needs to be pip installed (instead of directly mounted as we’ve done in the past). For pip installation my admin said that my parser needs at least a setup.py file, which it currently doenst have. The following lines are from the setup.py from the “old” parser structure:
The current Python recommendation is actually to replace setup.py with a pyproject.toml file (see here), which you already have. So you should already be able to install this plugin with pip. If you would need setup.py for some legacy reasons, you can still include it: it will simply load the information from the pyproject.toml file.
Hi @laurih thanks for quick answer. I wanted to try pip-installing my parser locally to test the installation, created new venv with nomad-lab==1.3.3 and python 3.9, but pip installing leads to dependency problems:
ERROR: Cannot install Lightforge_v2 and lightforge-v2==0.1.0 because these package versions have conflicting dependencies.
The conflict is caused by:
lightforge-v2 0.1.0 depends on nomad-lab>=1.3.0
nomad-schema-plugin-run 1.0.2 depends on nomad-lab>=1.3.6.dev1
lightforge-v2 0.1.0 depends on nomad-lab>=1.3.0
nomad-schema-plugin-run 1.0.1 depends on nomad-lab>=1.3.6.dev1
I assume that my admin will receive the same error when he tries installing it on our Oasis, therefore how do we solve these dependecy issues? My .toml is:
dependencies = [
"nomad-lab>=1.3.0", # I use 1.3.0 instead of Jose's 1.3.6.dev1 because our Oasis is 1.3.3, is this the mistake?
"python-magic-bin; sys_platform == 'win32'",
"nomad-schema-plugin-run>=1.0.1",
"nomad-simulations>0.0.7",
]
And what I also dont understand: why are there 2 different versions of nomad-schema-plugin-run (1.0.2 and 1.0.1) mentioned in error message, if my pyproject.toml only mentions 1.0.1?
(Fyi locally I installed nomad using pip install nomad-lab==1.3.3 --index-url <url>
instead of pip install -e '.[otherflags]' --index-url)
Best
Fabian
to install nomad-lab1.3.6.dev1 from .toml. Then I could pip-install my Lightforge parser and it worked, so updating our Oasis to 1.3.6.dev1 should also allow pip-installation of the parser and end the issue.
But when I try the same procedure with nomad-lab1.3.7, installing it and then pip-installing the parser, testing the parser gives me:
File "/home/fabian/Documents/nomad/.nomad137/lib/python3.9/site-packages/nomad/infrastructure.py", line 30, in <module>
from mongoengine import connect, disconnect
ModuleNotFoundError: No module named 'mongoengine'
I dont know how to deal with this error, and we have to upgrade to 1.3.7.+ sooner or later anyway, so do you have an idea how to fix this? My admin thinks it’s because nomad-schema-plugin-run doesnt recognize 1.3.7. as >= 1.3.6.dev1.
Thanks in advance
Fabian
If you install an oasis based on 1.3.6/1.3.7, you will already have these additional dependencies (parsing + infrastructure), but in your local machine you may not have them, and hopefully this will fix it.
Ultimately the reason for this error lies on our end: one of the plugins (possibly nomad-simulations is importing mongoengine without declaring it as a dependency. We are aware of the problem and will try to address it on our end soon.
):