ERROR nomad.metainfo 2022-11-14T07:39:03 Fail to generate metainfo. - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference. - nomad.metainfo.target_name: NXxpcs - nomad.service: unknown nomad service - nomad.version: 1.1.5 /usr/local/lib/python3.7/site-packages/optimade/__init__.py:18: DeprecationWarning: v0.17 of the `optimade` package will be the last to support Python 3.7. Please upgrade to Python 3.8+ to use v0.18 and later versions of `optimade`. DeprecationWarning, INFO: Started server process [34] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 127.0.0.1:49542 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:45056 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:47948 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:41184 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:42496 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:52514 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:58218 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:38842 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:35712 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:34054 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:45682 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:42380 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:38740 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:34816 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:34310 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:37026 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:45588 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:42114 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:47570 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:59444 - "GET /-/health HTTP/1.1" 200 OK INFO: 192.168.0.8:34348 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 127.0.0.1:43830 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:57968 - "GET /-/health HTTP/1.1" 200 OK INFO: 192.168.0.8:45254 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 192.168.0.8:54126 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 192.168.0.8:54130 - "GET /nomad-oasis/gui/tinymce/tinymce.min.js HTTP/1.0" 200 OK INFO: 192.168.0.8:54146 - "GET /nomad-oasis/gui/env.js HTTP/1.0" 200 OK INFO: 127.0.0.1:46666 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:60158 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:48216 - "GET /-/health HTTP/1.1" 200 OK INFO: 192.168.0.8:48392 - "GET /nomad-oasis/api/v1/info HTTP/1.0" 200 OK INFO: 192.168.0.8:48394 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 192.168.0.8:48410 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 127.0.0.1:42814 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:50724 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:49352 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:52094 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:47868 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:44204 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:56460 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:45462 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:60176 - "GET /-/health HTTP/1.1" 200 OK INFO: 192.168.0.8:33942 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 192.168.0.8:33958 - "GET /nomad-oasis/gui/tinymce/tinymce.min.js HTTP/1.0" 304 Not Modified INFO: 192.168.0.8:33970 - "GET /nomad-oasis/gui/env.js HTTP/1.0" 304 Not Modified INFO: 127.0.0.1:50802 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:59128 - "GET /-/health HTTP/1.1" 200 OK INFO: 192.168.0.8:39376 - "GET /nomad-oasis/gui/service-worker.js HTTP/1.0" 200 OK INFO: 192.168.0.8:39388 - "GET /nomad-oasis/api/v1/info HTTP/1.0" 200 OK INFO: 127.0.0.1:40136 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:55022 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:53272 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:36582 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:57950 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:56506 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:38066 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:50848 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:51978 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:38494 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:47700 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:42042 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:53038 - "GET /-/health HTTP/1.1" 200 OK WARNING urllib3.connectionpo 2022-11-14T08:00:47 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')': /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 WARNING urllib3.connectionpo 2022-11-14T08:00:47 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')': /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 WARNING urllib3.connectionpo 2022-11-14T08:00:47 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')': /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 INFO: 127.0.0.1:52514 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:42168 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:53292 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:44166 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:38000 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:41394 - "GET /-/health HTTP/1.1" 200 OK ERROR nomad.infrastructure 2022-11-14T08:01:47 cannot perform tokenauth - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn % (self.host, self.timeout), urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nomad-lab.eu timed out. (connect timeout=60)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 828, in urlopen **response_kw File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 158, in raw_get verify=self.verify) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: w1pSaXM-RkVwaxcXGfdaG-evfs26 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.app 2022-11-14T08:01:47 unexpected exception in API - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: qGbdviIC0oYdX8BB9Keu-bhleUkp - nomad.app.url: URL('http://localhost/nomad-oasis/api/v1/uploads?is_published=false&page_size=10000&order_by=upload_create_time&order=desc') - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) ERROR uvicorn.error 2022-11-14T08:01:47 Exception in ASGI application - lineno: 401 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 - process: 34 - stack_trace: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - thread_name: MainThread ERROR nomad.infrastructure 2022-11-14T08:01:47 cannot perform tokenauth - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn % (self.host, self.timeout), urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nomad-lab.eu timed out. (connect timeout=60)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 828, in urlopen **response_kw File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 158, in raw_get verify=self.verify) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: I-T2UZKMOu0pdmSsbhpQGytMji2w - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.infrastructure 2022-11-14T08:01:47 cannot perform tokenauth - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn % (self.host, self.timeout), urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nomad-lab.eu timed out. (connect timeout=60)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 828, in urlopen **response_kw File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 158, in raw_get verify=self.verify) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: chfm4-q8Vra99L6tYpEmbQrEyQbS - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.app 2022-11-14T08:01:47 unexpected exception in API - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: qGbdviIC0oYdX8BB9Keu-bhleUkp - nomad.app.url: URL('http://localhost/nomad-oasis/api/v1/uploads/command-examples') - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.app 2022-11-14T08:01:47 unexpected exception in API - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: qGbdviIC0oYdX8BB9Keu-bhleUkp - nomad.app.url: URL('http://localhost/nomad-oasis/api/v1/uploads?page_size=10&page=1&order_by=upload_create_time&order=desc') - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) ERROR uvicorn.error 2022-11-14T08:01:47 Exception in ASGI application - lineno: 401 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 - process: 34 - stack_trace: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - thread_name: MainThread ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) ERROR uvicorn.error 2022-11-14T08:01:47 Exception in ASGI application - lineno: 401 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 - process: 34 - stack_trace: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - thread_name: MainThread INFO: 127.0.0.1:38940 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:33302 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:43462 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:36332 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:41388 - "GET /-/health HTTP/1.1" 200 OK WARNING urllib3.connectionpo 2022-11-14T08:02:43 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')': /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 WARNING urllib3.connectionpo 2022-11-14T08:02:43 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')': /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 WARNING urllib3.connectionpo 2022-11-14T08:02:43 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')': /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 INFO: 127.0.0.1:54120 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:44902 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:59336 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:52922 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:50726 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:48184 - "GET /-/health HTTP/1.1" 200 OK ERROR nomad.infrastructure 2022-11-14T08:03:43 cannot perform tokenauth - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn % (self.host, self.timeout), urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nomad-lab.eu timed out. (connect timeout=60)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 828, in urlopen **response_kw File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 158, in raw_get verify=self.verify) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: lzDPKudYiXBzhtPkY-pzhkCToMA5 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.app 2022-11-14T08:03:43 unexpected exception in API - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: qGbdviIC0oYdX8BB9Keu-bhleUkp - nomad.app.url: URL('http://localhost/nomad-oasis/api/v1/uploads?page_size=10&page=1&order_by=upload_create_time&order=desc') - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) ERROR uvicorn.error 2022-11-14T08:03:43 Exception in ASGI application - lineno: 401 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 - process: 34 - stack_trace: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - thread_name: MainThread ERROR nomad.infrastructure 2022-11-14T08:03:44 cannot perform tokenauth - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn % (self.host, self.timeout), urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nomad-lab.eu timed out. (connect timeout=60)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 828, in urlopen **response_kw File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 158, in raw_get verify=self.verify) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: PpowrXNm1TP9Kz-daS878exWuJvB - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.app 2022-11-14T08:03:44 unexpected exception in API - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: qGbdviIC0oYdX8BB9Keu-bhleUkp - nomad.app.url: URL('http://localhost/nomad-oasis/api/v1/uploads?is_published=false&page_size=10000&order_by=upload_create_time&order=desc') - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) ERROR uvicorn.error 2022-11-14T08:03:44 Exception in ASGI application - lineno: 401 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 - process: 34 - stack_trace: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - thread_name: MainThread ERROR nomad.infrastructure 2022-11-14T08:03:44 cannot perform tokenauth - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn % (self.host, self.timeout), urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nomad-lab.eu timed out. (connect timeout=60)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 828, in urlopen **response_kw File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 158, in raw_get verify=self.verify) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: 0shLsh7JcBsFGZC9vswxvDr7cHCv - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR nomad.app 2022-11-14T08:03:44 unexpected exception in API - exception: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - exception_hash: qGbdviIC0oYdX8BB9Keu-bhleUkp - nomad.app.url: URL('http://localhost/nomad-oasis/api/v1/uploads/command-examples') - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) ERROR uvicorn.error 2022-11-14T08:03:44 Exception in ASGI application - lineno: 401 - nomad.commit: f476ce92 - nomad.deployment: oasis - nomad.service: unknown nomad service - nomad.version: 1.1.5 - process: 34 - stack_trace: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi result = await app(self.scope, self.receive, self.send) File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 226, in __call__ (self.listen_for_disconnect, {"receive": receive}), File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 24, in [task.result() for task in done] File "/usr/local/lib/python3.7/site-packages/starlette/responses.py", line 216, in stream_response async for chunk in self.body_iterator: File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 56, in body_stream task.result() File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 38, in coro await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 390, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__ raise exc from None File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 580, in __call__ await route.handle(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 241, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 52, in app response = await func(request) File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 210, in app dependency_overrides_provider=dependency_overrides_provider, File "/usr/local/lib/python3.7/site-packages/fastapi/dependencies/utils.py", line 550, in solve_dependencies solved = await run_in_threadpool(call, **sub_values) File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "./nomad/app/v1/routers/auth.py", line 153, in wrapper return user_dependency(**kwargs) File "./nomad/app/v1/routers/auth.py", line 73, in user_dependency user = _get_user_bearer_token_auth(kwargs.get('bearer_token')) File "./nomad/app/v1/routers/auth.py", line 187, in _get_user_bearer_token_auth user = cast(datamodel.User, infrastructure.keycloak.tokenauth(bearer_token)) File "./nomad/infrastructure.py", line 212, in tokenauth raise e File "./nomad/infrastructure.py", line 194, in tokenauth payload = self.decode_access_token(access_token) File "./nomad/infrastructure.py", line 170, in decode_access_token key = keycloak._public_keys.get(kid) File "./nomad/infrastructure.py", line 143, in _public_keys raise e File "./nomad/infrastructure.py", line 135, in _public_keys jwks = self._oidc_client.certs() File "/usr/local/lib/python3.7/site-packages/keycloak/keycloak_openid.py", line 267, in certs data_raw = self.connection.raw_get(URL_CERTS.format(**params_path)) File "/usr/local/lib/python3.7/site-packages/keycloak/connection.py", line 161, in raw_get "Can't connect to server (%s)" % e) keycloak.exceptions.KeycloakConnectionError: Can't connect to server (HTTPSConnectionPool(host='nomad-lab.eu', port=443): Max retries exceeded with url: /fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/certs (Caused by ConnectTimeoutError(, 'Connection to nomad-lab.eu timed out. (connect timeout=60)'))) - thread_name: MainThread INFO: 127.0.0.1:41996 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:54960 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:39468 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:58342 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:33696 - "GET /-/health HTTP/1.1" 200 OK INFO: 127.0.0.1:46240 - "GET /-/health HTTP/1.1" 200 OK