NOMAD Oasis status?

Dear NOMAD developers,

I would like to inquire about the current status of the NOMAD oasis. The FAQ states this is still work in progress, but I also found some docs that seems like it should already work somehow: https://nomad-lab.eu/prod/rae/docs/oasis.html

What is the current status? I’d like to deploy it locally at my institute (Materials Chemistry at RWTH Aachen) for research data management of data which is unsuitable for the public repository. Is this doable at the moment?

I’m also a bit uncertain how the setup described in the howto works. It claims that the user management is done by the central server. What does this actually mean for the workflow? So lets say I get it working at localdomain, how I understand it ATM is:

  • new user goes to localdomain
  • creates an account there? (or is redirected to central Nomad website?)
  • than the Oasis admin has to confirm that this user has indeed access to the Oasis
  • the user can use the Oasis at the localhostname in a same way as with the public repository?
    Is this correct?

Another slightly problematic part is that the docs claim the local server needs to be accessible on internet. We can get a server from our IT center easily but I’m not sure their rules will allow for it to be accessible from outside the university network. Could this work somehow, or alternatively how hard would it be to set up the local user management as well?

Thanks for your questions @Pavel_Ondracka. I apologise for the late reply.

Yes the OASIS can be used already and we have a few early adaptors. A proper web-page on the OASIS is still work in progress.

All OASIS and the central NOMAD share the same users and all users can use all OASIS installations (as far as they are part of the public internet, see below). Upon login/registration, users are redirected to our user-management and then back again, usually without even noticing. Currently, there is no support to restrict an OASIS use to specific users (besides running the OASIS in a private network). In principle, you can run also run your own user-management and we can help you with the setup there. But with separate users they will be no convenient way to publish data from the OASIS to central NOMAD.

If you would like to restrict your OASIS to certain users, we will consider this as a feature request.

The OASIS does not need to be accessible from the public internet. The servers need to access our user-management system (not the other way); the browsers of your users need to access your OASIS and our user-management system (not the other way). This works is typical VPN/campus network scenarios.

I hope this answers your questions. If you want to proceed in installing/test the OASIS write me an email to register an OASIS maintainer and host.

Thank you Markus,

your explanation makes most of the points clear. I think the common setup should work for us with the central user management. I just need to think a bit more how to solve some security issues (we might have some data which we really need to be careful about).

So if I understand it correctly, right now anyone who can access the address of an oasis installation could see all the data in there (similar to how the central archive works)? And one only needs an account for uploading (which in the default setup can be created with no restrictions)? I need to consider (and check with my boss) if the private network level access restrictions are enough for us. The only other easy option right now I can see would be the HTTP Basic Authentication access restrictions on the nginx level, but I have no idea how secure is that.

Yes, exactly. Of course you can have additional mechanisms on top, like a basic authentication or running everything in a local network+VPN, etc.

Otherwise, you have the same mechanisms that you have on the central NOMAD. As long as uploads are not published or if published with embargo, they are only visible to uploader and shared-with users. Of course this might be inconvenient and you have to explicitly share with the group all the time.

This is partly why we they that we still develop the OASIS. At the moment it is just a way to install NOMAD locally. But obviously the OASIS needs are different from the central NOMAD’s and we very much appreciate any discussion that helps us to figure out what additional features the OASIS needs.

Find the feature request for a more elaborate Oasis access rights management here: https://github.com/nomad-coe/nomad/issues/2

Thank you Markus for the update. I’m looking forward to this better user management.

In the meantime I was looking at making it work with the HTTP basic authentication for now. I managed to make it correctly ask for the password and after that it goes to the main page, but when I go to search or upload or pretty much anything else, I got an error “Unexpected error: “Unauthorized (401)”. Please try again and let us know, if this error keeps happening.” in the red box.

This feels like the http authentication breaks the communication between the containers somehow, but I don’t understand the scheme well enough to know if this is indeed the case… Or maybe something completely different?

Any ideas would be appreciated.

First an explanation. I guess you applied basic authentication to all paths: GUI and API. So it works for the GUI. But, once logged in the GUI (from you browser) starts to do API calls. Unfortunately, the javascript doing the API calls is unaware of the basic authentication and runs into these errors, because it does API calls without authentication.

Now ideas. You could apply the basic authentication only to the GUI path. But, outsiders could still use your API this way.

Even if we would make the GUI smarter and forward the authentication to the API, it would still not work, because we need the HTTP authentication header in our API calls for NOMAD’s own authentication mechanisms. I am sorry, but I did not think this through, when I suggested basic auth before.

There are only inconvenient options left: put the Oasis behind a VPN; do not publicly publish anything; wait for our new user management (at least 3 months).

We could introduce a simpler solution in an earlier release. Like having a fixed list of users in the nomad.yaml, require login by default, and allow those users? I have to think about this some more; no promises yet.

OK, I see.

I haven’t really looked at the GUI and API code so I have one last question. What kind of skillset and what time do you assume it would take to make the user/rights management work? We are considering hiring someone (likely some student) to write additional parsers for us, so there is a possibility we could help with this as well.

We implemented a very simple white-list based Oasis restriction mechanism that you can use, if you cannot wait for this. We will add this to the upcoming NOMAD v0.10.0 release. You can test with the pre-release build:
gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:oasis-with-auth (i.e. the regular nomad docker image with oasis-with-auth tag).

This will allow you to add a list of NOMAD account email addresses to your nomad.yaml and only those accounts can access your Oasis:

oasis:
    allowed_users:
        - [email protected]
        - [email protected]

See also comment here: https://github.com/nomad-coe/nomad/issues/2.

I hope this solves your most pressing issues.

Implementing a proper new version of NOMAD’s user rights management requires a lot of insight into NOMAD and a full-stack developer skillset. For new people (especially Physics students), I think parser work is much more fruitful. For this good Python skills, regular expressions, knowing what a numpy array is, etc. could be enough. Writing/fixing a parser is also a good entry point for more advanced NOMAD development. Of course, we offer assistance (zoom meetings, slack, etc.) when someone wants to get started.