NOMAD NORTH - Jupyter Notebook launch failed - Unexpected Error (500) - Oasis (local KeyCloack)

NOMAD and FAIRmat
1

Hi NOMAD-Team,

I try to setup a NOMAD Oasis with local keycloak (without SSL) as written here.
I managed to setup the local keycloak configuration and the Oasis seems to run smoothly.
Now I am trying to access the NORTH tools to launch a simple Jupyter Notebook, but all buttons are “NOT AVAILABLE” and there’s the red banner telling me “Unexpected error: “[object Object] (500)”. Please try again and let us know, if this error keeps happening.” - similiar to this post.

I followed your instructions and changed the docker-id in the north-section in docker-compose.yaml:

$ id
uid=1002(service) gid=100(users) groups=100(users),27(sudo),988(docker)

and docker-compose.yaml

# nomad remote tools hub (JupyterHUB, e.g. for AI Toolkit)
  north:
    ...
    volumes:
      - ./configs/nomad.yaml:/app/nomad.yaml
      - ./.volumes/fs:/app/.volumes/fs
      - /var/run/docker.sock:/var/run/docker.sock
    user: '1000:988'
    command: python -m nomad.cli admin run hub
    ...

All docker containers are healthy, but the nomad_oasis_app creates an error:
  - nomad.commit: 
  - nomad.deployment: oasis
  - nomad.service: app
  - nomad.version: 1.3.10

ERROR    nomad.app            2025-01-10T15:39:50 unexpected exception in API
  - exception: Traceback (most recent call last):
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
        conn = connection.create_connection(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
        raise err
      File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 716, in urlopen
        httplib_response = self._make_request(
                           ^^^^^^^^^^^^^^^^^^^
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
        super(HTTPConnection, self).request(method, url, body=body, headers=headers)
      File "/usr/local/lib/python3.11/http/client.py", line 1303, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.11/http/client.py", line 1349, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.11/http/client.py", line 996, in send
        self.connect()
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
        conn = self._new_conn()
               ^^^^^^^^^^^^^^^^
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
        raise NewConnectionError(
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x74c6379e7850>: Failed to establish a new connection: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
        resp = conn.urlopen(
               ^^^^^^^^^^^^^
      File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 802, in urlopen
        retries = retries.increment(
                  ^^^^^^^^^^^^^^^^^^
      File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /nomad-oasis/north/hub/api/users/test/servers/vesta/progress (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x74c6379e7850>: Failed to establish a new connection: [Errno 111] Connection refused'))
    
    During handling of the above exception, another exception occurred:

…

Despite this thread is presumably outdated, I tried to access the page: http://172.26.63.90/nomad-oasis/north/hub/home and can start as ordinary user a jupyter hub with button “Start my Server”. This spawn the jupyter hub (with a docker container “nomad_oasis_north-test” and image “jupyter/datascience-notebook”) and create a jupyter notebook.

Here’s the log of worker_oasis_north:

[I 2025-01-10 15:25:48.883 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2025-01-10 15:25:48.883 JupyterHub app:2889] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-15.1.0
[I 2025-01-10 15:25:48.883 JupyterHub app:2889] Using Spawner: builtins.DockerSpawnerWithWindowsFixes
[I 2025-01-10 15:25:48.883 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2025-01-10 15:25:48.900 JupyterHub app:1709] Writing cookie_secret to /app/jupyterhub_cookie_secret
[I 2025-01-10 15:25:48.940 alembic.runtime.migration migration:215] Context impl SQLiteImpl.
[I 2025-01-10 15:25:48.940 alembic.runtime.migration migration:218] Will assume non-transactional DDL.
[I 2025-01-10 15:25:49.615 alembic.runtime.migration migration:623] Running stamp_revision  -> 0eee8c825d24
[I 2025-01-10 15:25:50.735 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2025-01-10 15:25:51.084 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[W 2025-01-10 15:25:51.089 JupyterHub app:2382] Service nomad-service sets `admin: True`, which is deprecated in JupyterHub 2.0. You can assign now assign roles via `JupyterHub.load_roles` configuration. If you specify services in the admin role configuration, the Service admin flag will be ignored.
[I 2025-01-10 15:25:51.094 JupyterHub roles:238] Adding role admin for Service: nomad-service
[I 2025-01-10 15:25:51.218 JupyterHub app:2313] Adding API token for service: nomad-service
[I 2025-01-10 15:25:51.313 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2025-01-10 15:25:51.319 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-01-10 15:25:51.320 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-01-10 15:25:51.321 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2025-01-10 15:25:51.322 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2025-01-10 15:25:51.322 JupyterHub proxy:750] Starting proxy @ http://:9000/nomad-oasis/north
15:25:51.675 [ConfigProxy] e[32minfoe[39m: Proxying http://*:9000 to (no default)
15:25:51.677 [ConfigProxy] e[32minfoe[39m: Proxy API at http://127.0.0.1:8001/api/routes
15:25:51.834 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
[I 2025-01-10 15:25:51.839 JupyterHub app:3178] Hub API listening on http://0.0.0.0:8081/nomad-oasis/north/hub/
[I 2025-01-10 15:25:51.839 JupyterHub app:3180] Private Hub API connect url http://north:8081/nomad-oasis/north/hub/
[I 2025-01-10 15:25:51.839 JupyterHub app:3198] Adding external service nomad-service
15:25:51.841 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
[I 2025-01-10 15:25:51.841 JupyterHub proxy:477] Adding route for Hub: /nomad-oasis/north/ => http://north:8081
15:25:51.845 [ConfigProxy] e[32minfoe[39m: Adding route /nomad-oasis/north -> http://north:8081
15:25:51.846 [ConfigProxy] e[32minfoe[39m: Route added /nomad-oasis/north -> http://north:8081
15:25:51.847 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/nomad-oasis/north 
[I 2025-01-10 15:25:51.849 JupyterHub app:3245] JupyterHub is now running at http://:9000/nomad-oasis/north
15:30:51.853 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
15:35:51.853 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes

Attached also the docker-compose.yaml of the keycloak section, because I suppose something is not correct with the credentials to launch the jupyter hub with NORTH:

# keycloak user management
  keycloak:
    restart: unless-stopped
    image: quay.io/keycloak/keycloak:16.1.1
    container_name: nomad_oasis_keycloak
    environment:
      - PROXY_ADDRESS_FORWARDING=true
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=password
      - KEYCLOAK_FRONTEND_URL=http://172.26.63.90/keycloak/auth
      - KEYCLOAK_IMPORT="/tmp/nomad-realm.json"
    command:
      - "-Dkeycloak.import=/tmp/nomad-realm.json -Dkeycloak.migration.strategy=IGNORE_EXISTING"
    volumes:
      - keycloak:/opt/jboss/keycloak/standalone/data
      - ./configs/nomad-realm.json:/tmp/nomad-realm.json
    healthcheck:
      test: [ "CMD", "curl", "--fail", "--silent", "http://127.0.0.1:9990/health/live" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 30s
    ports:
      - 8080:8080
      - 8443:8443

and the part of nginx.conf:

 location /keycloak {
        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


        rewrite /keycloak/(.*) /$1 break;
        proxy_pass http://keycloak:8080;
    }

...

    location /nomad-oasis/north/ {
        client_max_body_size 500m;
        proxy_pass http://north:9000;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Scheme $scheme;

        proxy_buffering off;
    }

and nomad.yaml

services:
  api_host: '172.26.63.90'
  api_base_path: '/nomad-oasis'

oasis:
  is_oasis: true
  uses_central_user_management: false

north:
  jupyterhub_crypt_key: '978bfb2e13a8448a253c629d8dd84ff89587f30e635b753153960930cad9d36d'

keycloak:
  server_url: 'http://keycloak:8080/auth/'
  public_server_url: 'http://172.26.63.90/keycloak/auth/'
  realm_name: nomad
  username: 'admin'
  password: 'password'

...

Any suggestions are highly appreciate, how to spawn and run the jupyter notebooks. Thanks in advance.

2

UPDATE: The whole point of the port bindings in the keycloak section in docker-compose.yaml was to access the keycloak admin panel under http://<myserver>/keycloak/auth . Note the http and not https. As the approach fails and messes around with the internal docker containers I used another docker-compose.yaml and configuration files related to this thread. Maybe it’s worth to share it:

  1. Clean up your system and remove all docker-container,volumes, images… Start fresh
  2. Follow the instruction how to install NOMAD OASIS, esp. find your docker-group id
    getent group | grep docker
    but use the zip.file for local keycloak in this section
  3. Use for docker-compose this layout:
# docker-compose.yaml
services:
  # keycloak user management
  keycloak:
    restart: unless-stopped
    image: quay.io/keycloak/keycloak:16.1.1
    container_name: nomad_oasis_keycloak
    environment:
      - PROXY_ADDRESS_FORWARDING=true
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=password
      - KEYCLOAK_FRONTEND_URL=http://172.26.63.90/keycloak/auth
      - KEYCLOAK_IMPORT="/tmp/nomad-realm.json"
    command:
      - "-Dkeycloak.import=/tmp/nomad-realm.json -Dkeycloak.migration.strategy=IGNORE_EXISTING"
    volumes:
      - keycloak:/opt/jboss/keycloak/standalone/data
      - ./configs/nomad-realm.json:/tmp/nomad-realm.json
    healthcheck:
      test: [ "CMD", "curl", "--fail", "--silent", "http://127.0.0.1:9990/health/live" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 30s

  # broker for celery
  rabbitmq:
    restart: unless-stopped
    image: rabbitmq:4
    container_name: nomad_oasis_rabbitmq
    environment:
      - RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG
      - RABBITMQ_DEFAULT_USER=rabbitmq
      - RABBITMQ_DEFAULT_PASS=rabbitmq
      - RABBITMQ_DEFAULT_VHOST=/
    volumes:
      - rabbitmq:/var/lib/rabbitmq
    healthcheck:
      test: [ "CMD", "rabbitmq-diagnostics", "--silent", "--quiet", "ping" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 10s

  # the search engine
  elastic:
    restart: unless-stopped
    image: elasticsearch:7.17.24
    container_name: nomad_oasis_elastic
    environment:
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
      - discovery.type=single-node
    volumes:
      - elastic:/usr/share/elasticsearch/data
    healthcheck:
      test: [ "CMD", "curl", "--fail", "--silent", "http://elastic:9200/_cat/health" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 60s

  # the user data db
  mongo:
    restart: unless-stopped
    image: mongo:5
    container_name: nomad_oasis_mongo
    environment:
      - MONGO_DATA_DIR=/data/db
      - MONGO_LOG_DIR=/dev/null
    volumes:
      - mongo:/data/db
      - ./.volumes/mongo:/backup
    command: mongod --logpath=/dev/null # --quiet
    healthcheck:
      test: [ "CMD", "mongo", "mongo:27017/test", "--quiet", "--eval", "'db.runCommand({ping:1}).ok'" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 10s

  # nomad worker (processing)
  worker:
    restart: unless-stopped
    image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:prod
    container_name: nomad_oasis_worker
    environment:
      NOMAD_SERVICE: nomad_oasis_worker
      NOMAD_RABBITMQ_HOST: rabbitmq
      NOMAD_ELASTIC_HOST: elastic
      NOMAD_MONGO_HOST: mongo
    depends_on:
      rabbitmq:
        condition: service_healthy
      elastic:
        condition: service_healthy
      mongo:
        condition: service_healthy
    volumes:
      - ./configs/nomad.yaml:/app/nomad.yaml
      - ./.volumes/fs:/app/.volumes/fs
    command: python -m celery -A nomad.processing worker -l info -Q celery
    #./run-worker.sh

  # nomad app (api + proxy)
  app:
    restart: unless-stopped
    image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:prod
    container_name: nomad_oasis_app
    environment:
      NOMAD_SERVICE: nomad_oasis_app
      NOMAD_SERVICES_API_PORT: 80
      NOMAD_FS_EXTERNAL_WORKING_DIRECTORY: "$PWD"
      NOMAD_RABBITMQ_HOST: rabbitmq
      NOMAD_ELASTIC_HOST: elastic
      NOMAD_MONGO_HOST: mongo
      NOMAD_NORTH_HUB_HOST: north
    depends_on:
      rabbitmq:
        condition: service_healthy
      elastic:
        condition: service_healthy
      mongo:
        condition: service_healthy
      keycloak:
        condition: service_started
      north:
        condition: service_started
    volumes:
      - ./configs/nomad.yaml:/app/nomad.yaml
      - ./.volumes/fs:/app/.volumes/fs
    command: ./run.sh
    healthcheck:
      test: [ "CMD", "curl", "--fail", "--silent", "http://localhost:8000/-/health" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 10s

  # nomad remote tools hub (JupyterHUB, e.g. for AI Toolkit)
  north:
    restart: unless-stopped
    image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:prod
    container_name: nomad_oasis_north
    environment:
      NOMAD_SERVICE: nomad_oasis_north
      NOMAD_NORTH_DOCKER_NETWORK: nomad_oasis_network
      NOMAD_NORTH_HUB_CONNECT_IP: north
      NOMAD_NORTH_HUB_IP: "0.0.0.0"
      NOMAD_NORTH_HUB_HOST: north
      NOMAD_SERVICES_API_HOST: app
      NOMAD_FS_EXTERNAL_WORKING_DIRECTORY: "$PWD"
      NOMAD_RABBITMQ_HOST: rabbitmq
      NOMAD_ELASTIC_HOST: elastic
      NOMAD_MONGO_HOST: mongo
    depends_on:
      keycloak:
        condition: service_started
    volumes:
      - ./configs/nomad.yaml:/app/nomad.yaml
      - ./.volumes/fs:/app/.volumes/fs
      - /var/run/docker.sock:/var/run/docker.sock
    user: '1000:988'
    command: python -m nomad.cli admin run hub
    healthcheck:
      test: [ "CMD", "curl", "--fail", "--silent", "http://localhost:8081/nomad-oasis/north/hub/health" ]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 10s

  # nomad proxy (a reverse proxy for nomad)
  proxy:
    restart: unless-stopped
    image: nginx:stable-alpine
    container_name: nomad_oasis_proxy
    command: nginx -g 'daemon off;'
    volumes:
      - ./configs/nginx.conf:/etc/nginx/conf.d/default.conf
    depends_on:
      keycloak:
        condition: service_healthy
      app:
        condition: service_healthy
      worker:
        condition: service_started # TODO: service_healthy
      north:
        condition: service_healthy
    ports:
      - "80:80"

volumes:
  mongo:
    name: "nomad_oasis_mongo"
  elastic:
    name: "nomad_oasis_elastic"
  rabbitmq:
    name: "nomad_oasis_rabbitmq"
  keycloak:
    name: "nomad_oasis_keycloak"

networks:
  default:
    name: nomad_oasis_network

The only difference between the official guide and github entry is the line NOMAD_NORTH_HUB_HOST: north in the app section. Replace the IP 172.26.63.90 with <YourHostIP> and replace the line user: '1000:988' with your docker-group id from step 2: user: '1000:<dockergroupid> 3. Use for configs/nginx.conf`:

# nginx.conf
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen        80;
    server_name   localhost;
    proxy_set_header Host $host;

    gzip_min_length     1000;
    gzip_buffers        4 8k;
    gzip_http_version   1.0;
    gzip_disable        "msie6";
    gzip_vary           on;
    gzip on;
    gzip_proxied any;
    gzip_types
        text/css
        text/javascript
        text/xml
        text/plain
        application/javascript
        application/x-javascript
        application/json;

    location /keycloak {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        rewrite /keycloak/(.*) /$1 break;
        proxy_pass http://keycloak:8080;
    }

    location / {
        proxy_pass http://app:8000;
    }

    location ~ /nomad-oasis\/?(gui)?$ {
        rewrite ^ /nomad-oasis/gui/ permanent;
    }

    location /nomad-oasis/gui/ {
        proxy_intercept_errors on;
        error_page 404 = @redirect_to_index;
        proxy_pass http://app:8000;
    }

    location @redirect_to_index {
        rewrite ^ /nomad-oasis/gui/index.html break;
        proxy_pass http://app:8000;
    }

    location ~ \/gui\/(service-worker\.js|meta\.json)$ {
        add_header Last-Modified $date_gmt;
        add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
        if_modified_since off;
        expires off;
        etag off;
        proxy_pass http://app:8000;
    }

    location ~ /api/v1/uploads(/?$|.*/raw|.*/bundle?$)  {
        client_max_body_size 35g;
        proxy_request_buffering off;
        proxy_pass http://app:8000;
    }

    location ~ /api/v1/.*/download {
        proxy_buffering off;
        proxy_pass http://app:8000;
    }

    location /nomad-oasis/north/ {
        client_max_body_size 500m;
        proxy_pass http://north:9000;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Scheme $scheme;

        proxy_buffering off;
    }
}
  1. Use for nomad.yaml:
# nomad.yaml
services:
  api_host: '172.26.63.90'
  api_base_path: '/nomad-oasis'

oasis:
  is_oasis: true
  uses_central_user_management: false

north:
  jupyterhub_crypt_key: '978bfb2e13a8448a253c629d8dd84ff89587f30e635b753153960930cad9d36d'

keycloak:
  server_url: 'http://keycloak:8080/auth/'
  public_server_url: 'http://172.26.63.90/keycloak/auth/'
  realm_name: nomad
  username: 'admin'
  password: 'password'

meta:
  deployment: 'oasis'
  deployment_url: 'https://my-oasis.org/api'
  maintainer_email: '[email protected]'

logtransfer:
  enabled: false

logstash:
  enable: false

mongo:
    db_name: nomad_oasis_v1

elastic:
    entries_index: nomad_oasis_entries_v1
    materials_index: nomad_oasis_materials_v1

Also, replace 172.26.63.90 with <YourHostIP>
5. You should be able to login to http:\\<YourHostIP>\nomad-oasis\gui with the predefined config credentials: User test and PW password
6. Try to launch a Jupyter file either in the NORTH section or in Uploads. At first startup it may takes some time (> 3 min), and you maybe get also a nginx-error 502 Bad Gateway. Just reload the page and try again, maybe several time. At some point the JupyterHub is available and works like a charm. Thanks NOMAD! It’s really great to incorporate Jupyter with NOMAD!
7. But sometimes the error nginx-error 502 Bad Gateway is persistent (esp. in the Upload section). The only way I can figure it out is to access directly the JupyterHub page: http:\\<YourHostIP>/nomad-oasis/north/hub/home or shut down the jupyter notebook in the NORTH panel. Then try again.
8. If you spawn a jupyter notebook, there will be a docker container created with the username (here: test): nomad_oasis_north-test–jupyter

$ docker container ls
CONTAINER ID   IMAGE                                                           COMMAND                  CREATED              STATUS                        PORTS                                                 NAMES
1f8fbc838562   gitlab-registry.mpcdf.mpg.de/nomad-lab/north/jupyter:refactor   "tini -g -- start-no…"   About a minute ago   Up About a minute (healthy)   8888/tcp                                              nomad_oasis_north-test--jupyter
  1. But the whole point is to access the admin panel of keycloak: http://<myserver>/keycloak/auth and then the “Administration Console” http://<myserver>/keycloak/auth, which will prompt the known error “Sorry, HTTPS required!” To circumvent this problem do the following steps:
    9a. Find the id of the keycloak docker container and login, and change the SSL-authentication method:
$ docker container ls
CONTAINER ID   IMAGE                                                           COMMAND                  CREATED              STATUS                        PORTS                                                 NAMES                                   nomad_oasis_elastic
0ba7d5dbd677   quay.io/keycloak/keycloak:16.1.1                                "/opt/jboss/tools/do…"   14 hours ago         Up 14 hours (healthy)         8080/tcp, 8443/tcp                                    nomad_oasis_keycloak

$ docker exec -it 0ba7d5dbd677 bash
bash-4.4$ cd ~/keycloak/bin/
bash-4.4$ ./kcadm.sh config credentials --server http://keycloak:8080/auth --realm master --user admin 
bash-4.4$ ./kcadm.sh update realms/master -s sslRequired=NONE
bash-4.4$ exit
$

Now, you should have access to http://<myserver>/keycloak/auth with admin and password and you can create new user and configure your realm. Cheers!

3

Update 2: I created a new user “TestUserOasis” in the realm “nomad”. Login into the oasis and upload of files works. But if I want to access NORTH and try to launch a jupyter notebook (same also in the “Upload” section), I get the following error: “403 : Forbidden You do not have permission to access Server at /nomad-oasis/north/user/testuseroasis/jupyter/”

Screenshot_20250112_094457
Screenshot_20250112_0944571817×179 24.2 KB

The URL points to: http://172.26.63.90/nomad-oasis/north/hub/api/oauth2/authorize?client_id=jupyterhub-user-testuseroasis-jupyter&redirect_uri=%2Fnomad-oasis%2Fnorth%2Fuser%2Ftestuseroasis%2Fjupyter%2Foauth_callback&response_type=code&state=[SOMECRYPTICID]

On one hand a new docker container is spawned:

docker container ls
CONTAINER ID   IMAGE                                                           COMMAND                  CREATED         STATUS                   PORTS                                                 NAMES
090fdabe3264   gitlab-registry.mpcdf.mpg.de/nomad-lab/north/jupyter:refactor   "tini -g -- start-no…"   3 minutes ago   Up 3 minutes (healthy)   8888/tcp                                              nomad_oasis_north-testuseroasis--jupyter

But on the other hand, directly accessing http://172.26.63.90/nomad-oasis/north/hub/home shows no launched jupyter notebook. But interestingly, it seems I access the http://172.26.63.90/nomad-oasis/north/hub/home as “test” instead of “testuseroasis”?!

Screenshot_20250112_094547
Screenshot_20250112_0945471816×354 42.7 KB

Any advice on this?

4

:sweat_smile: Erratum: if this happens, just press the logout button and try to “reopen” or “launch” your notebook. AFAIK maybe a browser cache issue…

=== SOLVED: THREAD CAN BE CLOSED ===