Nomad North - Jupyter Hub

MichaelGoette · June 23, 2022, 11:01am

Hey Nomad,

I tried to test the nomad_oasis_hub, but when clicking the launch button in the gui, i get the following error. DO you have any idea?

nomad_oasis_hub | [I 2022-06-23 10:53:21.266 JupyterHub log:189] 200 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter/progress (michaelgoette@xxx) 11.71ms
nomad_oasis_hub | [D 2022-06-23 10:53:21.315 JupyterHub base:880] Initiating spawn for michaelgoette:jupyter
nomad_oasis_hub | [D 2022-06-23 10:53:21.315 JupyterHub base:887] 0/100 concurrent spawns
nomad_oasis_hub | [D 2022-06-23 10:53:21.315 JupyterHub base:892] 0 active servers
nomad_oasis_hub | [I 2022-06-23 10:53:21.325 JupyterHub provider:576] Updating oauth client jupyterhub-user-michaelgoette-jupyter
nomad_oasis_app | INFO:     172.21.0.7:49516 - "GET /nomad-oasis/api/v1/uploads?is_published=false&per_page=100 HTTP/1.1" 200 OK
nomad_oasis_hub | [D 2022-06-23 10:53:21.349 JupyterHub jupyterhub_config:146] Configure spawner with nomad volumes: {'/opt/nomad-oasis/.volumes/fs/north/users/michaelgoette': '/prefix/work', '/opt/nomad-oasis/.volumes/fs/north/shared': '/prefix/shared', '/opt/nomad-oasis/.volumes/fs/staging/1L/1LEYafbhSDWwwktm3CqE-g/raw': '/prefix/uploads/1LEYafbhSDWwwktm3CqE-g', '/opt/nomad-oasis/.volumes/fs/staging/q2/q2hGetSiSkmmJ5x9v7nUXg/raw': '/prefix/uploads/q2hGetSiSkmmJ5x9v7nUXg', '/opt/nomad-oasis/.volumes/fs/staging/pl/pldl4sVuS2uhXQWIt9AujQ/raw': '/prefix/uploads/pldl4sVuS2uhXQWIt9AujQ', '/opt/nomad-oasis/.volumes/fs/staging/fi/fikFXwrPTxuF8vJYtxN0gQ/raw': '/prefix/uploads/fikFXwrPTxuF8vJYtxN0gQ', '/opt/nomad-oasis/.volumes/fs/staging/uS/uS9-er_CSMau4Puyp43w3A/raw': '/prefix/uploads/uS9-er_CSMau4Puyp43w3A'}
nomad_oasis_hub | [D 2022-06-23 10:53:21.356 JupyterHub user:631] Calling Spawner.start for michaelgoette:jupyter
nomad_oasis_hub | [D 2022-06-23 10:53:21.357 JupyterHub jupyterhub_config:185] Configuring spawner for container jupyter-michaelgoette--jupyter
nomad_oasis_hub | [D 2022-06-23 10:53:21.357 JupyterHub jupyterhub_config:189] Configured spawner for jupyter
nomad_oasis_hub | [E 2022-06-23 10:53:21.359 JupyterHub user:720] Unhandled error starting michaelgoette's server: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
nomad_oasis_hub | [D 2022-06-23 10:53:21.359 JupyterHub user:819] Stopping michaelgoette:jupyter
nomad_oasis_hub | [D 2022-06-23 10:53:21.359 JupyterHub dockerspawner:982] Getting container 'jupyter-michaelgoette--jupyter'
nomad_oasis_hub | [E 2022-06-23 10:53:21.369 JupyterHub user:732] Failed to cleanup michaelgoette's server that failed to start
nomad_oasis_hub |     Traceback (most recent call last):
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 726, in spawn
nomad_oasis_hub |         await self.stop(spawner.name)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 823, in stop
nomad_oasis_hub |         status = await spawner.poll()
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 961, in poll
nomad_oasis_hub |         container = await self.get_object()
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 984, in get_object
nomad_oasis_hub |         obj = await self.docker("inspect_%s" % self.object_type, self.object_name)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
nomad_oasis_hub |         result = self.fn(*self.args, **self.kwargs)
nomad_oasis_hub |       File "/app/nomad/jupyterhub_config.py", line 203, in _docker
nomad_oasis_hub |         result = super()._docker(method, *args, **kwargs)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 947, in _docker
nomad_oasis_hub |         m = getattr(self.client, method)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 122, in client
nomad_oasis_hub |         client = docker.APIClient(**kwargs)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
nomad_oasis_hub |         self._version = self._retrieve_server_version()
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
nomad_oasis_hub |         f'Error while fetching server API version: {e}'
nomad_oasis_hub |     docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
nomad_oasis_hub |     
nomad_oasis_hub | [E 2022-06-23 10:53:21.369 JupyterHub web:1793] Uncaught exception POST /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter (xxx)
nomad_oasis_hub |     HTTPServerRequest(protocol='http', host='localhost', method='POST', uri='/nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter', version='HTTP/1.1', remote_ip='xxx')
nomad_oasis_hub |     Traceback (most recent call last):
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1704, in _execute
nomad_oasis_hub |         result = await result
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/apihandlers/users.py", line 472, in post
nomad_oasis_hub |         await self.spawn_single_user(user, server_name, options=options)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 997, in spawn_single_user
nomad_oasis_hub |         timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
nomad_oasis_hub |         await spawn_future
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 736, in spawn
nomad_oasis_hub |         raise e
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 635, in spawn
nomad_oasis_hub |         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
nomad_oasis_hub |       File "/app/nomad/jupyterhub_config.py", line 194, in start
nomad_oasis_hub |         return await super().start(image, extra_create_kwargs, extra_host_config)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 1255, in start
nomad_oasis_hub |         await self.pull_image(image)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 1211, in pull_image
nomad_oasis_hub |         await self.docker('inspect_image', image)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
nomad_oasis_hub |         result = self.fn(*self.args, **self.kwargs)
nomad_oasis_hub |       File "/app/nomad/jupyterhub_config.py", line 203, in _docker
nomad_oasis_hub |         result = super()._docker(method, *args, **kwargs)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 947, in _docker
nomad_oasis_hub |         m = getattr(self.client, method)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/dockerspawner/dockerspawner.py", line 122, in client
nomad_oasis_hub |         client = docker.APIClient(**kwargs)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
nomad_oasis_hub |         self._version = self._retrieve_server_version()
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
nomad_oasis_hub |         f'Error while fetching server API version: {e}'
nomad_oasis_hub |     docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
nomad_oasis_hub |     
nomad_oasis_gui | xxx - - [23/Jun/2022:10:53:21 +0000] "POST /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter HTTP/1.1" 500 35 "http://localhost:8001/nomad-oasis/gui/analyze/north" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0" "-"
nomad_oasis_hub | [E 2022-06-23 10:53:21.372 JupyterHub log:181] {
nomad_oasis_hub |       "X-Forwarded-Host": "localhost",
nomad_oasis_hub |       "X-Forwarded-Proto": "http",
nomad_oasis_hub |       "X-Forwarded-Port": "80",
nomad_oasis_hub |       "Cookie": "username-localhost-8889=[secret]; _xsrf=[secret]; PHPSESSID=[secret]; username-localhost-8888=[secret]; token=[secret]; token_team=[secret]; devicetoken=[secret]",
nomad_oasis_hub |       "Referer": "http://localhost:8001/nomad-oasis/gui/analyze/north",
nomad_oasis_hub |       "Authorization": "token [secret]",
nomad_oasis_hub |       "Sec-Fetch-Site": "same-origin",
nomad_oasis_hub |       "Sec-Fetch-Mode": "cors",
nomad_oasis_hub |       "Sec-Fetch-Dest": "empty",
nomad_oasis_hub |       "Dnt": "1",
nomad_oasis_hub |       "Origin": "http://localhost:8001",
nomad_oasis_hub |       "Accept-Encoding": "gzip, deflate, br",
nomad_oasis_hub |       "Accept-Language": "de,en-US;q=0.7,en;q=0.3",
nomad_oasis_hub |       "Accept": "application/json",
nomad_oasis_hub |       "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0",
nomad_oasis_hub |       "Content-Length": "0",
nomad_oasis_hub |       "X-Scheme": "http",
nomad_oasis_hub |       "Connection": "close",
nomad_oasis_hub |       "X-Forwarded-For": "xxx,::ffff:172.21.0.8",
nomad_oasis_hub |       "Host": "localhost",
nomad_oasis_hub |       "X-Real-Ip": "xxx"
nomad_oasis_hub |     }
nomad_oasis_hub | [E 2022-06-23 10:53:21.372 JupyterHub log:189] 500 POST /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter (michaelgoette@xxx) 59.66ms

Best Micha

mscheidgen · June 23, 2022, 11:17am

This is a very typical sign that the hub service (nomad_oasis_hub) is not allowed to use docker. Make sure

make sure you follow this doc. especially the second last paragraph of “Quick start”.
the user used to run nomad_oasis_hub must be able to use docker (docker-in-docker). Typically it has to be in the docker group. This is not the same GID on all distros. You might need to change the used group in the docker-compose.yaml.
make sure that /var/run/docker.sock is correctly mounted. Maybe this is also on a different path in your case?

MichaelGoette · June 23, 2022, 11:38am

Hey,
i am sorry, i did not read the new documentation carefully and did not change the docker group id.

When i click the launch for the basic jupyter it starts a jupyter hub and opens a new tab. When I click new I get

Further, when clicking the hyperspy launch button I get this:

nomad_oasis_hub | [E 2022-06-23 11:36:59.112 JupyterHub jupyterhub_config:128] Cannot get user uploads: {"detail":"Could not validate credentials. The given token is invalid."}
nomad_oasis_hub | [D 2022-06-23 11:36:59.119 JupyterHub user:631] Calling Spawner.start for michaelgoette:hyperspy
nomad_oasis_hub | [D 2022-06-23 11:36:59.120 JupyterHub jupyterhub_config:185] Configuring spawner for container jupyter-michaelgoette--hyperspy
nomad_oasis_hub | [E 2022-06-23 11:36:59.120 JupyterHub user:720] Unhandled error starting michaelgoette's server: 'image'
nomad_oasis_hub | [D 2022-06-23 11:36:59.120 JupyterHub user:819] Stopping michaelgoette:hyperspy
nomad_oasis_hub | [D 2022-06-23 11:36:59.120 JupyterHub dockerspawner:982] Getting container 'jupyter-michaelgoette--hyperspy'
nomad_oasis_hub | [I 2022-06-23 11:36:59.123 JupyterHub dockerspawner:989] Container 'jupyter-michaelgoette--hyperspy' is gone
nomad_oasis_hub | [W 2022-06-23 11:36:59.123 JupyterHub dockerspawner:963] Container not found: jupyter-michaelgoette--hyperspy
nomad_oasis_hub | [D 2022-06-23 11:36:59.129 JupyterHub user:845] Deleting oauth client jupyterhub-user-michaelgoette-hyperspy
nomad_oasis_hub | [D 2022-06-23 11:36:59.135 JupyterHub user:848] Finished stopping michaelgoette:hyperspy
nomad_oasis_hub | [E 2022-06-23 11:36:59.141 JupyterHub web:1793] Uncaught exception POST /nomad-oasis/north/hub/api/users/michaelgoette/servers/hyperspy (134.30.95.70)
nomad_oasis_hub |     HTTPServerRequest(protocol='http', host='localhost', method='POST', uri='/nomad-oasis/north/hub/api/users/michaelgoette/servers/hyperspy', version='HTTP/1.1', remote_ip='134.30.95.70')
nomad_oasis_hub |     Traceback (most recent call last):
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1704, in _execute
nomad_oasis_hub |         result = await result
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/apihandlers/users.py", line 472, in post
nomad_oasis_hub |         await self.spawn_single_user(user, server_name, options=options)
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 997, in spawn_single_user
nomad_oasis_hub |         timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
nomad_oasis_hub |         await spawn_future
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 736, in spawn
nomad_oasis_hub |         raise e
nomad_oasis_hub |       File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 635, in spawn
nomad_oasis_hub |         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
nomad_oasis_hub |       File "/app/nomad/jupyterhub_config.py", line 188, in start
nomad_oasis_hub |         tools[tool](self)
nomad_oasis_hub |       File "/app/nomad/jupyterhub_config.py", line 234, in configure
nomad_oasis_hub |         spawner.image = tool_json['image']
nomad_oasis_hub |     KeyError: 'image'
nomad_oasis_hub |

Best Micha

mscheidgen · June 24, 2022, 7:47am

So you upload directories get mounted and you see files from your uploads? But when you want to create a new file (i.e. a notebook) in one of the upload directories, you get this error?

I guess that something goes wrong with the mounting of these directories. Could docker exec into the started jupyter container. It should be called <username>-jupyter, or something like that. Check the owner and permissions on /home/joyvan/uploads/xxxxxxxxx folders. What are the owner and permissions on your .volumes folder and their children?

MichaelGoette · June 24, 2022, 8:22am

Ok today i did it from inside the internal network and yesterday i did it with SSH port forwarding to localhost. Today directly from the internal network it works, i think this might be the issue, i will test this again when i am home.

Thanks.

I can also start nionswift, but hyperspy still gives:

mscheidgen · June 24, 2022, 8:27am

These timeout might happen on the first open. The image has to be pulled and this takes too long. We should add something that prefetches the images automatically. I can imagine if you try this a second time on the same installation that it works.