Dear NOMAD developers,
I would like to inquire about the current status of the NOMAD oasis. The FAQ states this is still work in progress, but I also found some docs that seems like it should already work somehow: https://nomad-lab.eu/prod/rae/docs/oasis.html
What is the current status? I’d like to deploy it locally at my institute (Materials Chemistry at RWTH Aachen) for research data management of data which is unsuitable for the public repository. Is this doable at the moment?
I’m also a bit uncertain how the setup described in the howto works. It claims that the user management is done by the central server. What does this actually mean for the workflow? So lets say I get it working at localdomain, how I understand it ATM is:
- new user goes to localdomain
- creates an account there? (or is redirected to central Nomad website?)
- than the Oasis admin has to confirm that this user has indeed access to the Oasis
- the user can use the Oasis at the localhostname in a same way as with the public repository?
Is this correct?
Another slightly problematic part is that the docs claim the local server needs to be accessible on internet. We can get a server from our IT center easily but I’m not sure their rules will allow for it to be accessible from outside the university network. Could this work somehow, or alternatively how hard would it be to set up the local user management as well?
Thanks for your questions @Pavel_Ondracka. I apologise for the late reply.
Yes the OASIS can be used already and we have a few early adaptors. A proper web-page on the OASIS is still work in progress.
All OASIS and the central NOMAD share the same users and all users can use all OASIS installations (as far as they are part of the public internet, see below). Upon login/registration, users are redirected to our user-management and then back again, usually without even noticing. Currently, there is no support to restrict an OASIS use to specific users (besides running the OASIS in a private network). In principle, you can run also run your own user-management and we can help you with the setup there. But with separate users they will be no convenient way to publish data from the OASIS to central NOMAD.
If you would like to restrict your OASIS to certain users, we will consider this as a feature request.
The OASIS does not need to be accessible from the public internet. The servers need to access our user-management system (not the other way); the browsers of your users need to access your OASIS and our user-management system (not the other way). This works is typical VPN/campus network scenarios.
I hope this answers your questions. If you want to proceed in installing/test the OASIS write me an email to register an OASIS maintainer and host.
Thank you Markus,
your explanation makes most of the points clear. I think the common setup should work for us with the central user management. I just need to think a bit more how to solve some security issues (we might have some data which we really need to be careful about).
So if I understand it correctly, right now anyone who can access the address of an oasis installation could see all the data in there (similar to how the central archive works)? And one only needs an account for uploading (which in the default setup can be created with no restrictions)? I need to consider (and check with my boss) if the private network level access restrictions are enough for us. The only other easy option right now I can see would be the HTTP Basic Authentication access restrictions on the nginx level, but I have no idea how secure is that.
Yes, exactly. Of course you can have additional mechanisms on top, like a basic authentication or running everything in a local network+VPN, etc.
Otherwise, you have the same mechanisms that you have on the central NOMAD. As long as uploads are not published or if published with embargo, they are only visible to uploader and shared-with users. Of course this might be inconvenient and you have to explicitly share with the group all the time.
This is partly why we they that we still develop the OASIS. At the moment it is just a way to install NOMAD locally. But obviously the OASIS needs are different from the central NOMAD’s and we very much appreciate any discussion that helps us to figure out what additional features the OASIS needs.
Find the feature request for a more elaborate Oasis access rights management here: https://github.com/nomad-coe/nomad/issues/2
Thank you Markus for the update. I’m looking forward to this better user management.
In the meantime I was looking at making it work with the HTTP basic authentication for now. I managed to make it correctly ask for the password and after that it goes to the main page, but when I go to search or upload or pretty much anything else, I got an error “Unexpected error: “Unauthorized (401)”. Please try again and let us know, if this error keeps happening.” in the red box.
This feels like the http authentication breaks the communication between the containers somehow, but I don’t understand the scheme well enough to know if this is indeed the case… Or maybe something completely different?
Any ideas would be appreciated.
First an explanation. I guess you applied basic authentication to all paths: GUI and API. So it works for the GUI. But, once logged in the GUI (from you browser) starts to do API calls. Unfortunately, the javascript doing the API calls is unaware of the basic authentication and runs into these errors, because it does API calls without authentication.
Now ideas. You could apply the basic authentication only to the GUI path. But, outsiders could still use your API this way.
Even if we would make the GUI smarter and forward the authentication to the API, it would still not work, because we need the HTTP authentication header in our API calls for NOMAD’s own authentication mechanisms. I am sorry, but I did not think this through, when I suggested basic auth before.
There are only inconvenient options left: put the Oasis behind a VPN; do not publicly publish anything; wait for our new user management (at least 3 months).
We could introduce a simpler solution in an earlier release. Like having a fixed list of users in the nomad.yaml, require login by default, and allow those users? I have to think about this some more; no promises yet.
OK, I see.
I haven’t really looked at the GUI and API code so I have one last question. What kind of skillset and what time do you assume it would take to make the user/rights management work? We are considering hiring someone (likely some student) to write additional parsers for us, so there is a possibility we could help with this as well.
We implemented a very simple white-list based Oasis restriction mechanism that you can use, if you cannot wait for this. We will add this to the upcoming NOMAD v0.10.0 release. You can test with the pre-release build:
gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:oasis-with-auth (i.e. the regular nomad docker image with oasis-with-auth tag).
This will allow you to add a list of NOMAD account email addresses to your nomad.yaml and only those accounts can access your Oasis:
oasis:
allowed_users:
- [email protected]
- [email protected]
See also comment here: https://github.com/nomad-coe/nomad/issues/2.
I hope this solves your most pressing issues.
Implementing a proper new version of NOMAD’s user rights management requires a lot of insight into NOMAD and a full-stack developer skillset. For new people (especially Physics students), I think parser work is much more fruitful. For this good Python skills, regular expressions, knowing what a numpy array is, etc. could be enough. Writing/fixing a parser is also a good entry point for more advanced NOMAD development. Of course, we offer assistance (zoom meetings, slack, etc.) when someone wants to get started.
I’m having some issues getting this oasis-with-auth image working. It looks like the worker is crashing and the image keeps restarting. I got some logs, but I’m not sure if this is helpful:
[2021-03-01 15:14:43 +0000] [8] [INFO] Starting gunicorn 20.0.4
[2021-03-01 15:14:43 +0000] [8] [INFO] Listening at: http://0.0.0.0:8000 (8)
[2021-03-01 15:14:43 +0000] [8] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-01 15:14:43 +0000] [11] [INFO] Booting worker with pid: 11
[2021-03-01 15:14:43 +0000] [11] [ERROR] Exception in worker process
Traceback (most recent call last):
File “/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py”, line 583, in spawn_worker
worker.init_process()
File “/usr/local/lib/python3.7/site-packages/uvicorn/workers.py”, line 63, in init_process
super(UvicornWorker, self).init_process()
File “/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py”, line 119, in init_process
self.load_wsgi()
File “/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py”, line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File “/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py”, line 67, in wsgi
self.callable = self.load()
File “/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py”, line 49, in load
return self.load_wsgiapp()
File “/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py”, line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File “/usr/local/lib/python3.7/site-packages/gunicorn/util.py”, line 358, in import_app
mod = importlib.import_module(module)
File “/usr/local/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1006, in _gcd_import
File “”, line 983, in _find_and_load
File “”, line 967, in _find_and_load_unlocked
File “”, line 677, in _load_unlocked
File “”, line 728, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/app/nomad/app/main.py”, line 23, in
from nomad import config, infrastructure
File “/app/nomad/config.py”, line 419, in
load_config()
File “/app/nomad/config.py”, line 415, in load_config
_apply_env_variables()
File “/app/nomad/config.py”, line 385, in _apply_env_variables
_apply(key, value, raise_error=False)
File “/app/nomad/config.py”, line 345, in apply
group_key, config_key = full_key.split(’’, 1)
ValueError: not enough values to unpack (expected 2, got 1)
[2021-03-01 15:14:43 +0000] [11] [INFO] Worker exiting (pid: 11)
[2021-03-01 15:14:43 +0000] [8] [INFO] Shutting down: Master
[2021-03-01 15:14:43 +0000] [8] [INFO] Reason: Worker failed to boot.
Thanks for trying this. Could you post your redacted (ids, passwords, etc.) nomad.yaml? This happens while reading this config file.
Actually my redacted nomad.yaml file is just the exact one from Oasis example:
Operating an OASIS — NOMAD Repository and Archive documentation + the new
oasis:
allowed_users:
- [email protected]
I was suspecting some issues with nomad.yaml as well, but it works OK with the stable image.
I’ve tried removing the new part but it doesn’t matter, the stable image still work and the oasis-with-auth doesn’t. But I can provide the exact file by email anyway if needed, just let me know.
Sorry, I should look more and assume less. It is not the nomad.yaml, but an environment variable its stumbling over. I cannot reproduce so far. Have you added any additional variables beginning with NOMAD_?
I updated the docker image with a fix. It should now raise a log error for this problematic env variable instead of failing.
Check if docker-compose is really downloading the new image. I am not sure if it does this automatically. Otherwise, docker-compose pull should do that for you.
The new images just work, I’ll test the authentication and report later.
Thanks @mscheidgen
Update on the authentication: so far this is working fine and I believe it is sufficient for our current needs. Thank you @mscheidgen again for the quick response to this request.