Hi NOMAD-Team,
I try to setup a NOMAD Oasis with local keycloak (without SSL) as written here.
I managed to setup the local keycloak configuration and the Oasis seems to run smoothly.
Now I am trying to access the NORTH tools to launch a simple Jupyter Notebook, but all buttons are “NOT AVAILABLE” and there’s the red banner telling me “Unexpected error: “[object Object] (500)”. Please try again and let us know, if this error keeps happening.” - similiar to this post.
I followed your instructions and changed the docker-id in the north-section in docker-compose.yaml:
$ id
uid=1002(service) gid=100(users) groups=100(users),27(sudo),988(docker)
and docker-compose.yaml
# nomad remote tools hub (JupyterHUB, e.g. for AI Toolkit)
north:
...
volumes:
- ./configs/nomad.yaml:/app/nomad.yaml
- ./.volumes/fs:/app/.volumes/fs
- /var/run/docker.sock:/var/run/docker.sock
user: '1000:988'
command: python -m nomad.cli admin run hub
...
All docker containers are healthy, but the nomad_oasis_app creates an error:
- nomad.commit:
- nomad.deployment: oasis
- nomad.service: app
- nomad.version: 1.3.10
ERROR nomad.app 2025-01-10T15:39:50 unexpected exception in API
- exception: Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 716, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.11/http/client.py", line 1303, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1349, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
self.send(msg)
File "/usr/local/lib/python3.11/http/client.py", line 996, in send
self.connect()
File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x74c6379e7850>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 802, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /nomad-oasis/north/hub/api/users/test/servers/vesta/progress (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x74c6379e7850>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
…
Despite this thread is presumably outdated, I tried to access the page: http://172.26.63.90/nomad-oasis/north/hub/home and can start as ordinary user a jupyter hub with button “Start my Server”. This spawn the jupyter hub (with a docker container “nomad_oasis_north-test” and image “jupyter/datascience-notebook”) and create a jupyter notebook.
Here’s the log of worker_oasis_north:
[I 2025-01-10 15:25:48.883 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2025-01-10 15:25:48.883 JupyterHub app:2889] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-15.1.0
[I 2025-01-10 15:25:48.883 JupyterHub app:2889] Using Spawner: builtins.DockerSpawnerWithWindowsFixes
[I 2025-01-10 15:25:48.883 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2025-01-10 15:25:48.900 JupyterHub app:1709] Writing cookie_secret to /app/jupyterhub_cookie_secret
[I 2025-01-10 15:25:48.940 alembic.runtime.migration migration:215] Context impl SQLiteImpl.
[I 2025-01-10 15:25:48.940 alembic.runtime.migration migration:218] Will assume non-transactional DDL.
[I 2025-01-10 15:25:49.615 alembic.runtime.migration migration:623] Running stamp_revision -> 0eee8c825d24
[I 2025-01-10 15:25:50.735 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2025-01-10 15:25:51.084 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[W 2025-01-10 15:25:51.089 JupyterHub app:2382] Service nomad-service sets `admin: True`, which is deprecated in JupyterHub 2.0. You can assign now assign roles via `JupyterHub.load_roles` configuration. If you specify services in the admin role configuration, the Service admin flag will be ignored.
[I 2025-01-10 15:25:51.094 JupyterHub roles:238] Adding role admin for Service: nomad-service
[I 2025-01-10 15:25:51.218 JupyterHub app:2313] Adding API token for service: nomad-service
[I 2025-01-10 15:25:51.313 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2025-01-10 15:25:51.319 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-01-10 15:25:51.320 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-01-10 15:25:51.321 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2025-01-10 15:25:51.322 JupyterHub proxy:746] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2025-01-10 15:25:51.322 JupyterHub proxy:750] Starting proxy @ http://:9000/nomad-oasis/north
15:25:51.675 [ConfigProxy] e[32minfoe[39m: Proxying http://*:9000 to (no default)
15:25:51.677 [ConfigProxy] e[32minfoe[39m: Proxy API at http://127.0.0.1:8001/api/routes
15:25:51.834 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes
[I 2025-01-10 15:25:51.839 JupyterHub app:3178] Hub API listening on http://0.0.0.0:8081/nomad-oasis/north/hub/
[I 2025-01-10 15:25:51.839 JupyterHub app:3180] Private Hub API connect url http://north:8081/nomad-oasis/north/hub/
[I 2025-01-10 15:25:51.839 JupyterHub app:3198] Adding external service nomad-service
15:25:51.841 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes
[I 2025-01-10 15:25:51.841 JupyterHub proxy:477] Adding route for Hub: /nomad-oasis/north/ => http://north:8081
15:25:51.845 [ConfigProxy] e[32minfoe[39m: Adding route /nomad-oasis/north -> http://north:8081
15:25:51.846 [ConfigProxy] e[32minfoe[39m: Route added /nomad-oasis/north -> http://north:8081
15:25:51.847 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/nomad-oasis/north
[I 2025-01-10 15:25:51.849 JupyterHub app:3245] JupyterHub is now running at http://:9000/nomad-oasis/north
15:30:51.853 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes
15:35:51.853 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes
Attached also the docker-compose.yaml of the keycloak section, because I suppose something is not correct with the credentials to launch the jupyter hub with NORTH:
# keycloak user management
keycloak:
restart: unless-stopped
image: quay.io/keycloak/keycloak:16.1.1
container_name: nomad_oasis_keycloak
environment:
- PROXY_ADDRESS_FORWARDING=true
- KEYCLOAK_USER=admin
- KEYCLOAK_PASSWORD=password
- KEYCLOAK_FRONTEND_URL=http://172.26.63.90/keycloak/auth
- KEYCLOAK_IMPORT="/tmp/nomad-realm.json"
command:
- "-Dkeycloak.import=/tmp/nomad-realm.json -Dkeycloak.migration.strategy=IGNORE_EXISTING"
volumes:
- keycloak:/opt/jboss/keycloak/standalone/data
- ./configs/nomad-realm.json:/tmp/nomad-realm.json
healthcheck:
test: [ "CMD", "curl", "--fail", "--silent", "http://127.0.0.1:9990/health/live" ]
interval: 10s
timeout: 10s
retries: 30
start_period: 30s
ports:
- 8080:8080
- 8443:8443
and the part of nginx.conf:
location /keycloak {
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
rewrite /keycloak/(.*) /$1 break;
proxy_pass http://keycloak:8080;
}
...
location /nomad-oasis/north/ {
client_max_body_size 500m;
proxy_pass http://north:9000;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# websocket headers
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Scheme $scheme;
proxy_buffering off;
}
and nomad.yaml
services:
api_host: '172.26.63.90'
api_base_path: '/nomad-oasis'
oasis:
is_oasis: true
uses_central_user_management: false
north:
jupyterhub_crypt_key: '978bfb2e13a8448a253c629d8dd84ff89587f30e635b753153960930cad9d36d'
keycloak:
server_url: 'http://keycloak:8080/auth/'
public_server_url: 'http://172.26.63.90/keycloak/auth/'
realm_name: nomad
username: 'admin'
password: 'password'
...
Any suggestions are highly appreciate, how to spawn and run the jupyter notebooks. Thanks in advance.