Hi Nomad,
I am trying to get the Nomad remote tools hub (North) running on our oasis. Therefore I followed the steps described by:
successfully:
but the tools in the oasis just give me an error when I try to launch them:
I also tried restarting the oasis and pulling it again but it still doesnt work. Did I miss some important steps? How can I check if I have NOMAD’s Jupyter HUB service running or not?
Best Alex
Hello Alex,
The documentation that you we refering to is just the developer docs for the tool images. It will not really help you to get the installation going. Also the images already exist in our registry and you do not need to create them.
Please understand that we did not officially released the NORTH functionality yet, not even as a beta. There are still things we fix to fix the docker-compose installation for NORTH. Anyhow, here is the best current information to get it run. This only applies to linux systems and linux docker.
Use the quick start installation documented here (it might be worthwhile to read the rest as well).
Make sure that you have a user nomad with user-id 1000 and that this user is allowed to use docker. NOMAD will run everything with this user and it must be able to create and start containers. Typically you have to create the user and add it to the docker group, e.g:
useradd --uid 1000 nomad
usermod -aG docker nomad
.volumes directories exist and make sure they belong to the user-id 1000 user:mkdir -p .volumes/fs/north/shared
mkdir -p .volumes/fs/north/users
mkdir -p .volumes/fs/public
mkdir -p .volumes/fs/staging
chown -R 1000 .volumes
location ~ /hub/ { to location ~ /north/ {
These steps should allow you to start containers. If something does not work, please include the logs of the nomad_oasis_hub container (docker logs nomad_oasis_hub) when you report this here.
It might be good to manually prefetch images with docker pull. I recommend to use the jupyter image for your tests:
docker pull jupyter/datascience-notebook
For trouble shooting it might be sensible to start containers directly from jupyterhub and not from the NOMAD gui for testing. Use your browser to go to .../nomad-oasis/north/hub/home, you can create a named server called jupyter. This one should work. Unnamed servers (“start my server”) are not supposed to work. The name has to match a pre-configured tool.
jupyterhub_config.py to all the other files and mount it into the hub. In your docker-compose add the following volume to the hub container:- ./jupyterhub_config.py:/app/nomad/jupyterhub_config.py
jupyterhub_config.py (9.8 KB)
We will include these “patches” in a next release.
There might be a simpler fix to step 4. Instead of the patch, you simply might need to set some additional variables in the nomad.yaml:
north:
jupyterhub_crypt_key: '978bfb2e13a8448a253c629d8dd84ff89587f30e635b753153960930cad9d36d'
hub_ip_connect: '172.17.0.1'
users_fs: /abs/path/to/.volumes/fs/north/users
shared_fs: /abs/path/to/.volumes/fs/north/shared
fs:
public_external: /abs/path/to/.volumes/fs/public
staging_external: /abs/path/to/.volumes/fs/staging
Sorry, I didn’t know that you haven’t released NORTH yet. I saw it in the oasis, so I thought it was ready to be used. Anyways, thanks for your guide. One question: After adding the new used with the id 1000, do I have to switch to this user and start the oasis with it, or is it ok if I run the oasis with the original user?
I also I seem to have troubles getting the jupyterhub running on the oasis. If I go to .../nomad-oasis/north/hub/home on the browser it spits out a “404: Not Found” error. I checked the logs of the nomad_oasis_hub and they seem fine at first (at least I dont see an error), but as soon as I try to connect to the server …/nomad-oasis/gui/analyze/north/hub/home I get the error attached at the end of the log file:
$ docker logs nomad_oasis_hub
[D 2022-06-15 13:59:59.191 JupyterHub application:831] Looking for nomad/jupyterhub_config in /app
[D 2022-06-15 14:00:00.119 JupyterHub application:852] Loaded config file: /app/nomad/jupyterhub_config.py
[W 2022-06-15 14:00:00.125 JupyterHub configurable:200] Config option `hub_ip_connect` not recognized by `JupyterHub`. Did you mean one of: `hub_connect_ip, hub_connect_port, hub_connect_url`?
[I 2022-06-15 14:00:00.126 JupyterHub app:2459] Running JupyterHub version 1.4.2
[I 2022-06-15 14:00:00.126 JupyterHub app:2490] Using Authenticator: builtins.NomadAuthenticator
[I 2022-06-15 14:00:00.126 JupyterHub app:2490] Using Spawner: builtins.NomadDockerSpawner
[I 2022-06-15 14:00:00.126 JupyterHub app:2490] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.2
[D 2022-06-15 14:00:00.127 JupyterHub app:2422] Could not load pycurl: No module named 'pycurl'
pycurl is recommended if you have a large number of users.
[D 2022-06-15 14:00:00.129 JupyterHub app:1574] Generating new cookie_secret
[I 2022-06-15 14:00:00.129 JupyterHub app:1579] Writing cookie_secret to /app/jupyterhub_cookie_secret
[D 2022-06-15 14:00:00.129 JupyterHub app:1701] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-06-15 14:00:00.137 JupyterHub orm:784] Stamping empty database with alembic revision 4dc2d5a8c53c
[I 2022-06-15 14:00:00.138 alembic.runtime.migration migration:201] Context impl SQLiteImpl.
[I 2022-06-15 14:00:00.138 alembic.runtime.migration migration:208] Will assume non-transactional DDL.
[I 2022-06-15 14:00:00.162 alembic.runtime.migration migration:615] Running stamp_revision -> 4dc2d5a8c53c
[D 2022-06-15 14:00:00.162 alembic.runtime.migration migration:815] new branch insert 4dc2d5a8c53c
[I 2022-06-15 14:00:00.344 JupyterHub proxy:497] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2022-06-15 14:00:00.345 JupyterHub configurable:200] Config option `userdata_method` not recognized by `NomadAuthenticator`. Did you mean one of: `userdata_params, userdata_token_method, userdata_url`?
[W 2022-06-15 14:00:00.346 JupyterHub app:1808] No admin users, admin interface will be unavailable.
[W 2022-06-15 14:00:00.346 JupyterHub app:1810] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2022-06-15 14:00:00.346 JupyterHub app:1839] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-06-15 14:00:00.380 JupyterHub app:1990] Purging expired APITokens
[D 2022-06-15 14:00:00.383 JupyterHub app:1990] Purging expired OAuthAccessTokens
[D 2022-06-15 14:00:00.385 JupyterHub app:1990] Purging expired OAuthCodes
[D 2022-06-15 14:00:00.392 JupyterHub app:2113] Initializing spawners
[D 2022-06-15 14:00:00.393 JupyterHub app:2246] Loaded users:
[I 2022-06-15 14:00:00.394 JupyterHub app:2529] Initialized 0 spawners in 0.002 seconds
[W 2022-06-15 14:00:00.396 JupyterHub proxy:700] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2022-06-15 14:00:00.396 JupyterHub proxy:703] Starting proxy @ http://:9000/nomad-oasis/north/
[D 2022-06-15 14:00:00.396 JupyterHub proxy:704] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '9000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://9275ef9d45d1:8081/nomad-oasis/north/hub/error']
[D 2022-06-15 14:00:00.402 JupyterHub proxy:611] Writing proxy pid file: jupyterhub-proxy.pid
14:00:00.473 [ConfigProxy] info: Proxying http://*:9000 to (no default)
14:00:00.474 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2022-06-15 14:00:00.552 JupyterHub proxy:739] Proxy started and appears to be up
[D 2022-06-15 14:00:00.554 JupyterHub proxy:832] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:00:00.558 [ConfigProxy] info: 200 GET /api/routes
[I 2022-06-15 14:00:00.558 JupyterHub app:2774] Hub API listening on http://0.0.0.0:8081/nomad-oasis/north/hub/
[I 2022-06-15 14:00:00.558 JupyterHub app:2776] Private Hub API connect url http://9275ef9d45d1:8081/nomad-oasis/north/hub/
[D 2022-06-15 14:00:00.558 JupyterHub proxy:342] Fetching routes to check
[D 2022-06-15 14:00:00.558 JupyterHub proxy:832] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:00:00.559 [ConfigProxy] info: 200 GET /api/routes
[I 2022-06-15 14:00:00.559 JupyterHub proxy:347] Checking routes
[I 2022-06-15 14:00:00.559 JupyterHub proxy:432] Adding route for Hub: /nomad-oasis/north/ => http://9275ef9d45d1:8081
[D 2022-06-15 14:00:00.560 JupyterHub proxy:832] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/nomad-oasis/north
14:00:00.561 [ConfigProxy] info: Adding route /nomad-oasis/north -> http://9275ef9d45d1:8081
14:00:00.561 [ConfigProxy] info: Route added /nomad-oasis/north -> http://9275ef9d45d1:8081
14:00:00.561 [ConfigProxy] info: 201 POST /api/routes/nomad-oasis/north
[I 2022-06-15 14:00:00.562 JupyterHub app:2849] JupyterHub is now running at http://:9000/nomad-oasis/north/
[D 2022-06-15 14:00:00.562 JupyterHub app:2452] It took 1.374 seconds for the Hub to start
14:00:10.328 [ConfigProxy] error: 404 GET /nomad-oasis/gui/analyze/north/hub/home
[I 2022-06-15 14:00:10.351 JupyterHub log:189] 200 GET /nomad-oasis/north/hub/error/404?url=%2Fnomad-oasis%2Fgui%2Fanalyze%2Fnorth%2Fhub%2Fhome (@172.24.0.7) 18.52ms
The 404 probably means that your requests never reach the hub. Have you checked the nginx.conf and changed location ~ /hub/ { to location ~ /north/ { (step 3 in the previous message).
We are working on a new release that should fix most of the issues somewhere next week.
We now have a new beta release with the results of this discussed included as fixes. If you follow the most recent documentation: Operating NOMAD (Oasis) - NOMAD Documentation you should be able to run NORTH with the NOMAD docker-compose based installation.