Attaching to nomad_oasis_north, nomad_oasis_app, nomad_oasis_worker, nomad_oasis_mongo, nomad_oasis_elastic, nomad_oasis_rabbitmq
nomad_oasis_app | ERROR    nomad.metainfo       2022-10-23T08:02:39 Fail to generate metainfo.
nomad_oasis_app |   - nomad.commit: f476ce92
nomad_oasis_app |   - nomad.deployment: oasis
nomad_oasis_app |   - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference.
nomad_oasis_app |   - nomad.metainfo.target_name: NXxpcs
nomad_oasis_app |   - nomad.service: unknown nomad service
nomad_oasis_app |   - nomad.version: 1.1.5
nomad_oasis_app | /usr/local/lib/python3.7/site-packages/optimade/__init__.py:18: DeprecationWarning: v0.17 of the `optimade` package will be the last to support Python 3.7. Please upgrade to Python 3.8+ to use v0.18 and later versions of `optimade`.
nomad_oasis_app |   DeprecationWarning,
nomad_oasis_app | INFO:     Started server process [34]
nomad_oasis_app | INFO:     Waiting for application startup.
nomad_oasis_app | WARNING  elasticsearch        2022-10-23T08:04:48 GET http://elastic:9200/ [status:N/A request:60.060s]
nomad_oasis_app |   - lineno: 294
nomad_oasis_app |   - nomad.commit: f476ce92
nomad_oasis_app |   - nomad.deployment: oasis
nomad_oasis_app |   - nomad.service: unknown nomad service
nomad_oasis_app |   - nomad.version: 1.1.5
nomad_oasis_app |   - process: 34
nomad_oasis_app |   - stack_trace: Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_app |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_app |     raise err
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_app |     sock.connect(sa)
nomad_oasis_app | socket.timeout: timed out
nomad_oasis_app | 
nomad_oasis_app | During handling of the above exception, another exception occurred:
nomad_oasis_app | 
nomad_oasis_app | Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_app |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_app |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_app |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_app |     raise value
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_app |     chunked=chunked,
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_app |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_app |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_app |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_app |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_app |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_app |     self.send(msg)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_app |     self.connect()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_app |     conn = self._new_conn()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_app |     % (self.host, self.timeout),
nomad_oasis_app | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_app | 
nomad_oasis_app |   - thread_name: MainThread
nomad_oasis_app | ERROR:    Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_app |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_app |     raise err
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_app |     sock.connect(sa)
nomad_oasis_app | socket.timeout: timed out
nomad_oasis_app | 
nomad_oasis_app | During handling of the above exception, another exception occurred:
nomad_oasis_app | 
nomad_oasis_app | Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_app |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_app |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_app |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_app |     raise value
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_app |     chunked=chunked,
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_app |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_app |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_app |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_app |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_app |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_app |     self.send(msg)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_app |     self.connect()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_app |     conn = self._new_conn()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_app |     % (self.host, self.timeout),
nomad_oasis_app | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_app | 
nomad_oasis_app | During handling of the above exception, another exception occurred:
nomad_oasis_app | 
nomad_oasis_app | Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 540, in lifespan
nomad_oasis_app |     async for item in self.lifespan_context(app):
nomad_oasis_rabbitmq | 2022-10-23 08:02:01.326466+00:00 [warn] <0.130.0> Overriding Erlang cookie using the value set in the environment
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593185+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593266+00:00 [info] <0.228.0> Feature flags:   [ ] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593298+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593350+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593429+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593457+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593479+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.593551+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.933052+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.933140+00:00 [noti] <0.228.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
nomad_oasis_rabbitmq | 2022-10-23 08:02:02.947688+00:00 [noti] <0.228.0> Logging: configured log handlers are now ACTIVE
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.144060+00:00 [info] <0.228.0> ra: starting system quorum_queues
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.144160+00:00 [info] <0.228.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@7842daa4bac1/quorum/rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.229003+00:00 [info] <0.265.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.243325+00:00 [noti] <0.270.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.252268+00:00 [info] <0.228.0> ra: starting system coordination
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.252329+00:00 [info] <0.228.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@7842daa4bac1/coordination/rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.254149+00:00 [info] <0.278.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.254406+00:00 [noti] <0.283.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.257918+00:00 [info] <0.228.0> 
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.257918+00:00 [info] <0.228.0>  Starting RabbitMQ 3.9.13 on Erlang 24.3.2 [jit]
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.257918+00:00 [info] <0.228.0>  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.257918+00:00 [info] <0.228.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
nomad_oasis_rabbitmq | 
nomad_oasis_rabbitmq |   ##  ##      RabbitMQ 3.9.13
nomad_oasis_rabbitmq |   ##  ##
nomad_oasis_rabbitmq |   ##########  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
nomad_oasis_rabbitmq |   ######  ##
nomad_oasis_rabbitmq |   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
nomad_oasis_rabbitmq | 
nomad_oasis_rabbitmq |   Erlang:      24.3.2 [jit]
nomad_oasis_rabbitmq |   TLS Library: OpenSSL - OpenSSL 1.1.1n  15 Mar 2022
nomad_oasis_rabbitmq | 
nomad_oasis_rabbitmq |   Doc guides:  https://rabbitmq.com/documentation.html
nomad_oasis_rabbitmq |   Support:     https://rabbitmq.com/contact.html
nomad_oasis_rabbitmq |   Tutorials:   https://rabbitmq.com/getstarted.html
nomad_oasis_rabbitmq |   Monitoring:  https://rabbitmq.com/monitoring.html
nomad_oasis_rabbitmq | 
nomad_oasis_rabbitmq |   Logs: /var/log/rabbitmq/rabbit@7842daa4bac1_upgrade.log
nomad_oasis_rabbitmq |         <stdout>
nomad_oasis_rabbitmq | 
nomad_oasis_rabbitmq |   Config file(s): /etc/rabbitmq/conf.d/10-default-guest-user.conf
nomad_oasis_rabbitmq |                   /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
nomad_oasis_rabbitmq | 
nomad_oasis_rabbitmq |   Starting broker...2022-10-23 08:02:03.259196+00:00 [info] <0.228.0> 
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>  node           : rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>  home dir       : /var/lib/rabbitmq
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>  config file(s) : /etc/rabbitmq/conf.d/10-default-guest-user.conf
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>                 : /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>  cookie hash    : P6euPuJ+9iKMq+WqGJEzBA==
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>  log(s)         : /var/log/rabbitmq/rabbit@7842daa4bac1_upgrade.log
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>                 : <stdout>
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.259196+00:00 [info] <0.228.0>  database dir   : /var/lib/rabbitmq/mnesia/rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420177+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420312+00:00 [info] <0.228.0> Feature flags:   [ ] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420381+00:00 [info] <0.228.0> Feature flags:   [ ] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420418+00:00 [info] <0.228.0> Feature flags:   [ ] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420450+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420540+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420573+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420611+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420640+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:03.420702+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.059776+00:00 [info] <0.228.0> Running boot step pre_boot defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.059870+00:00 [info] <0.228.0> Running boot step rabbit_global_counters defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.060089+00:00 [info] <0.228.0> Running boot step rabbit_osiris_metrics defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.060224+00:00 [info] <0.228.0> Running boot step rabbit_core_metrics defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.060874+00:00 [info] <0.228.0> Running boot step rabbit_alarm defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.064837+00:00 [info] <0.295.0> Memory high watermark set to 3142 MiB (3295536742 bytes) of 7857 MiB (8238841856 bytes) total
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.068393+00:00 [info] <0.297.0> Enabling free disk space monitoring
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.068449+00:00 [info] <0.297.0> Disk free limit set to 50MB
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.071366+00:00 [info] <0.228.0> Running boot step code_server_cache defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.071497+00:00 [info] <0.228.0> Running boot step file_handle_cache defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.071711+00:00 [info] <0.302.0> Limiting to approx 1048479 file handles (943629 sockets)
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.071832+00:00 [info] <0.303.0> FHC read buffering: OFF
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.071868+00:00 [info] <0.303.0> FHC write buffering: ON
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.072449+00:00 [info] <0.228.0> Running boot step worker_pool defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.072514+00:00 [info] <0.285.0> Will use 8 processes for default worker pool
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.072536+00:00 [info] <0.285.0> Starting worker pool 'worker_pool' with 8 processes in it
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 481, in default_lifespan
nomad_oasis_app |     await self.startup()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 516, in startup
nomad_oasis_app |     await handler()
nomad_oasis_app |   File "./nomad/app/main.py", line 96, in startup_event
nomad_oasis_app |     infrastructure.setup()
nomad_oasis_app |   File "./nomad/infrastructure.py", line 72, in setup
nomad_oasis_app |     setup_elastic()
nomad_oasis_app |   File "./nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_app |     create_v1_indices()
nomad_oasis_app |   File "./nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_app |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_app |   File "./nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_app |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_app |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_app |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_app |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_app |     raise error
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_app |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_app |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_app | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_app | 
nomad_oasis_app | ERROR    uvicorn.error        2022-10-23T08:04:48 Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_app |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_app |     raise err
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_app |     sock.connect(sa)
nomad_oasis_app | socket.timeout: timed out
nomad_oasis_app | 
nomad_oasis_app | During handling of the above exception, another exception occurred:
nomad_oasis_app | 
nomad_oasis_app | Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_app |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_app |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_app |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_app |     raise value
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_app |     chunked=chunked,
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_app |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_app |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_app |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_app |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_app |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_app |     self.send(msg)
nomad_oasis_app |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_app |     self.connect()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_app |     conn = self._new_conn()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_app |     % (self.host, self.timeout),
nomad_oasis_app | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_app | 
nomad_oasis_app | During handling of the above exception, another exception occurred:
nomad_oasis_app | 
nomad_oasis_app | Traceback (most recent call last):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 540, in lifespan
nomad_oasis_app |     async for item in self.lifespan_context(app):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 481, in default_lifespan
nomad_oasis_app |     await self.startup()
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 516, in startup
nomad_oasis_app |     await handler()
nomad_oasis_app |   File "./nomad/app/main.py", line 96, in startup_event
nomad_oasis_app |     infrastructure.setup()
nomad_oasis_app |   File "./nomad/infrastructure.py", line 72, in setup
nomad_oasis_app |     setup_elastic()
nomad_oasis_app |   File "./nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_app |     create_v1_indices()
nomad_oasis_app |   File "./nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_app |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_app |   File "./nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_app |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_app |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_app |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_app |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_app |     raise error
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_app |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_app |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_app |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.073075+00:00 [info] <0.228.0> Running boot step database defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.073345+00:00 [info] <0.228.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@7842daa4bac1 is empty. Assuming we need to join an existing cluster or initialise from scratch...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.073407+00:00 [info] <0.228.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.073448+00:00 [info] <0.228.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.073538+00:00 [info] <0.228.0> All discovered existing cluster peers:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.073568+00:00 [info] <0.228.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.077072+00:00 [noti] <0.44.0> Application mnesia exited with reason: stopped
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.157571+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.157740+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.168906+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.169049+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.169103+00:00 [info] <0.228.0> Feature flag `drop_unroutable_metric`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.169201+00:00 [info] <0.228.0> Feature flag `drop_unroutable_metric`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170203+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170255+00:00 [info] <0.228.0> Feature flags:   [~] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170286+00:00 [info] <0.228.0> Feature flags:   [ ] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170376+00:00 [info] <0.228.0> Feature flags:   [ ] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170436+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170473+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170503+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170530+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170559+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.170590+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.180755+00:00 [info] <0.228.0> Feature flag `drop_unroutable_metric`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.181999+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182052+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182074+00:00 [info] <0.228.0> Feature flags:   [ ] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182105+00:00 [info] <0.228.0> Feature flags:   [ ] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182145+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182174+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182190+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182203+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182236+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.182273+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.194414+00:00 [info] <0.228.0> Feature flag `empty_basic_get_metric`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.194505+00:00 [info] <0.228.0> Feature flag `empty_basic_get_metric`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195463+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195520+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195550+00:00 [info] <0.228.0> Feature flags:   [~] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195575+00:00 [info] <0.228.0> Feature flags:   [ ] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195615+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195648+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195674+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195712+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195750+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.195773+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.206999+00:00 [info] <0.228.0> Feature flag `empty_basic_get_metric`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209636+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209693+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209728+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209751+00:00 [info] <0.228.0> Feature flags:   [ ] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209832+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209864+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209886+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209906+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209962+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.209990+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.218875+00:00 [info] <0.228.0> Feature flag `implicit_default_bindings`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.218979+00:00 [info] <0.228.0> Feature flag `implicit_default_bindings`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220022+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220071+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220104+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220148+00:00 [info] <0.228.0> Feature flags:   [~] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220187+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220216+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220248+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220269+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220310+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.220343+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.233482+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.233618+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.233684+00:00 [info] <0.228.0> Feature flag `implicit_default_bindings`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235552+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235606+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235651+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235679+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235705+00:00 [info] <0.228.0> Feature flags:   [ ] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235768+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235794+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235818+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235841+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.235934+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.245832+00:00 [info] <0.228.0> Feature flag `maintenance_mode_status`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.245904+00:00 [info] <0.228.0> Feature flag `maintenance_mode_status`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246674+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246710+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246733+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246759+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246780+00:00 [info] <0.228.0> Feature flags:   [~] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246798+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246838+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246869+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246909+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.246942+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.255141+00:00 [info] <0.228.0> Creating table rabbit_node_maintenance_states for feature flag `maintenance_mode_status`
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.258851+00:00 [info] <0.228.0> Feature flag `maintenance_mode_status`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260437+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260482+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260515+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260544+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260567+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260648+00:00 [info] <0.228.0> Feature flags:   [ ] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260677+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260696+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260710+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.260764+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.268504+00:00 [info] <0.228.0> Feature flag `quorum_queue`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.268605+00:00 [info] <0.228.0> Feature flag `quorum_queue`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269738+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269790+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269821+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269860+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_app | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7f4652a4fa10>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_app | 
nomad_oasis_app |   - nomad.commit: f476ce92
nomad_oasis_app |   - nomad.deployment: oasis
nomad_oasis_app |   - nomad.service: unknown nomad service
nomad_oasis_app |   - nomad.version: 1.1.5
nomad_oasis_app | ERROR:    Application startup failed. Exiting.
nomad_oasis_app | ERROR    uvicorn.error        2022-10-23T08:04:48 Application startup failed. Exiting.
nomad_oasis_app |   - nomad.commit: f476ce92
nomad_oasis_app |   - nomad.deployment: oasis
nomad_oasis_app |   - nomad.service: unknown nomad service
nomad_oasis_app |   - nomad.version: 1.1.5
nomad_oasis_app | ERROR    nomad.metainfo       2022-10-23T08:05:11 Fail to generate metainfo.
nomad_oasis_app |   - nomad.commit: f476ce92
nomad_oasis_app |   - nomad.deployment: oasis
nomad_oasis_app |   - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference.
nomad_oasis_app |   - nomad.metainfo.target_name: NXxpcs
nomad_oasis_app |   - nomad.service: unknown nomad service
nomad_oasis_app |   - nomad.version: 1.1.5
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269896+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269922+00:00 [info] <0.228.0> Feature flags:   [~] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269952+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.269982+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.270007+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.270036+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.279835+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.279964+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.280026+00:00 [info] <0.228.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_queue...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.292769+00:00 [info] <0.228.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_durable_queue...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.315245+00:00 [info] <0.228.0> Feature flag `quorum_queue`:   Mnesia tables migration done
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.315383+00:00 [info] <0.228.0> Feature flag `quorum_queue`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.317791+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.317907+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318004+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318063+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318132+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318249+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318320+00:00 [info] <0.228.0> Feature flags:   [ ] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318384+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318455+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.318538+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.331972+00:00 [info] <0.228.0> Feature flag `stream_queue`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.332076+00:00 [info] <0.228.0> Feature flag `stream_queue`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333151+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333216+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333250+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333306+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333372+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333403+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333441+00:00 [info] <0.228.0> Feature flags:   [~] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333469+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333506+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.333537+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_north | ERROR    nomad.metainfo       2022-10-23T08:02:37 Fail to generate metainfo.
nomad_oasis_north |   - nomad.commit: f476ce92
nomad_oasis_north |   - nomad.deployment: oasis
nomad_oasis_north |   - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference.
nomad_oasis_north |   - nomad.metainfo.target_name: NXxpcs
nomad_oasis_north |   - nomad.service: hub
nomad_oasis_north |   - nomad.version: 1.1.5
nomad_oasis_north | [I 2022-10-23 08:02:44.146 JupyterHub app:2459] Running JupyterHub version 1.4.2
nomad_oasis_north | [I 2022-10-23 08:02:44.146 JupyterHub app:2490] Using Authenticator: builtins.NomadAuthenticator
nomad_oasis_north | [I 2022-10-23 08:02:44.146 JupyterHub app:2490] Using Spawner: builtins.NomadDockerSpawner
nomad_oasis_north | [I 2022-10-23 08:02:44.146 JupyterHub app:2490] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.2
nomad_oasis_north | [I 2022-10-23 08:02:44.151 JupyterHub app:1579] Writing cookie_secret to /app/jupyterhub_cookie_secret
nomad_oasis_north | [I 2022-10-23 08:02:44.189 alembic.runtime.migration migration:204] Context impl SQLiteImpl.
nomad_oasis_north | [I 2022-10-23 08:02:44.189 alembic.runtime.migration migration:211] Will assume non-transactional DDL.
nomad_oasis_north | [I 2022-10-23 08:02:44.201 alembic.runtime.migration migration:618] Running stamp_revision  -> 4dc2d5a8c53c
nomad_oasis_north | [I 2022-10-23 08:02:44.277 JupyterHub proxy:497] Generating new CONFIGPROXY_AUTH_TOKEN
nomad_oasis_north | [W 2022-10-23 08:02:44.280 JupyterHub app:1808] No admin users, admin interface will be unavailable.
nomad_oasis_north | [W 2022-10-23 08:02:44.280 JupyterHub app:1810] Add any administrative users to `c.Authenticator.admin_users` in config.
nomad_oasis_north | [I 2022-10-23 08:02:44.280 JupyterHub app:1839] Not using allowed_users. Any authenticated user will be allowed.
nomad_oasis_north | [I 2022-10-23 08:02:44.371 JupyterHub app:2529] Initialized 0 spawners in 0.003 seconds
nomad_oasis_north | [W 2022-10-23 08:02:44.375 JupyterHub proxy:700] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
nomad_oasis_north | [I 2022-10-23 08:02:44.375 JupyterHub proxy:703] Starting proxy @ http://:9000/nomad-oasis/north
nomad_oasis_north | 08:02:44.801 [ConfigProxy] info: Proxying http://*:9000 to (no default)
nomad_oasis_north | 08:02:44.807 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
nomad_oasis_north | 08:02:44.889 [ConfigProxy] info: 200 GET /api/routes 
nomad_oasis_north | [I 2022-10-23 08:02:44.889 JupyterHub app:2774] Hub API listening on http://0.0.0.0:8081/nomad-oasis/north/hub/
nomad_oasis_north | [I 2022-10-23 08:02:44.890 JupyterHub app:2776] Private Hub API connect url http://north:8081/nomad-oasis/north/hub/
nomad_oasis_north | 08:02:44.892 [ConfigProxy] info: 200 GET /api/routes 
nomad_oasis_north | [I 2022-10-23 08:02:44.893 JupyterHub proxy:347] Checking routes
nomad_oasis_north | [I 2022-10-23 08:02:44.893 JupyterHub proxy:432] Adding route for Hub: /nomad-oasis/north/ => http://north:8081
nomad_oasis_north | 08:02:44.896 [ConfigProxy] info: Adding route /nomad-oasis/north -> http://north:8081
nomad_oasis_north | 08:02:44.897 [ConfigProxy] info: Route added /nomad-oasis/north -> http://north:8081
nomad_oasis_north | 08:02:44.898 [ConfigProxy] info: 201 POST /api/routes/nomad-oasis/north 
nomad_oasis_north | [I 2022-10-23 08:02:44.898 JupyterHub app:2849] JupyterHub is now running at http://:9000/nomad-oasis/north
nomad_oasis_worker | ERROR    nomad.metainfo       2022-10-23T08:02:35 Fail to generate metainfo.
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference.
nomad_oasis_worker |   - nomad.metainfo.target_name: NXxpcs
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker | ERROR    celery.worker.consum 2022-10-23T08:02:59 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out.
nomad_oasis_worker | Trying again in 2.00 seconds... (1/100)
nomad_oasis_worker | 
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker | ERROR    celery.worker.consum 2022-10-23T08:03:15 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out.
nomad_oasis_worker | Trying again in 4.00 seconds... (2/100)
nomad_oasis_worker | 
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker | ERROR    celery.worker.consum 2022-10-23T08:03:33 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out.
nomad_oasis_worker | Trying again in 6.00 seconds... (3/100)
nomad_oasis_worker | 
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:44 GET http://elastic:9200/ [status:N/A request:60.061s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 35
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99e9df10>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:44 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99e9df10>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99e9df10>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 35
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.344303+00:00 [info] <0.228.0> Feature flag `stream_queue`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346207+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346273+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346309+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346364+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346443+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346479+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346523+00:00 [info] <0.228.0> Feature flags:   [x] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346547+00:00 [info] <0.228.0> Feature flags:   [ ] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346582+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.346620+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.354520+00:00 [info] <0.228.0> Feature flag `user_limits`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.354598+00:00 [info] <0.228.0> Feature flag `user_limits`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355378+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355415+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355434+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355459+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355508+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355542+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355583+00:00 [info] <0.228.0> Feature flags:   [x] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355606+00:00 [info] <0.228.0> Feature flags:   [~] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355624+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.355655+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.365264+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.365398+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.376893+00:00 [info] <0.228.0> Feature flag `user_limits`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378503+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378579+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378609+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378639+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378679+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378725+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378761+00:00 [info] <0.228.0> Feature flags:   [x] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378795+00:00 [info] <0.228.0> Feature flags:   [x] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378826+00:00 [info] <0.228.0> Feature flags:   [ ] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.378875+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.388707+00:00 [info] <0.228.0> Feature flag `virtual_host_metadata`: supported, attempt to enable...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.388806+00:00 [info] <0.228.0> Feature flag `virtual_host_metadata`: mark as enabled=state_changing
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389568+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389599+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389617+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389638+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389693+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389723+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389756+00:00 [info] <0.228.0> Feature flags:   [x] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389800+00:00 [info] <0.228.0> Feature flags:   [x] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389848+00:00 [info] <0.228.0> Feature flags:   [~] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.389887+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.397389+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.397515+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.409782+00:00 [info] <0.228.0> Feature flag `virtual_host_metadata`: mark as enabled=true
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411513+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411579+00:00 [info] <0.228.0> Feature flags:   [x] drop_unroutable_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411638+00:00 [info] <0.228.0> Feature flags:   [x] empty_basic_get_metric
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411673+00:00 [info] <0.228.0> Feature flags:   [x] implicit_default_bindings
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411700+00:00 [info] <0.228.0> Feature flags:   [x] maintenance_mode_status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411727+00:00 [info] <0.228.0> Feature flags:   [x] quorum_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411749+00:00 [info] <0.228.0> Feature flags:   [x] stream_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411788+00:00 [info] <0.228.0> Feature flags:   [x] user_limits
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411821+00:00 [info] <0.228.0> Feature flags:   [x] virtual_host_metadata
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.411848+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.422597+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.422730+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433030+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433178+00:00 [info] <0.228.0> Successfully synced tables from a peer
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433222+00:00 [info] <0.228.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433294+00:00 [info] <0.228.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433422+00:00 [info] <0.228.0> Running boot step database_sync defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433543+00:00 [info] <0.228.0> Running boot step feature_flags defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433653+00:00 [info] <0.228.0> Running boot step codec_correctness_check defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433707+00:00 [info] <0.228.0> Running boot step external_infrastructure defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433744+00:00 [info] <0.228.0> Running boot step rabbit_registry defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433834+00:00 [info] <0.228.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99e9df10>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99e9df10>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99e9df10>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:44 GET http://elastic:9200/ [status:N/A request:60.061s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 38
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:03,363Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "version[7.17.1], pid[7], build[default/docker/e5acb99f822233d62d6444ce45a4543dc1c8059a/2022-02-23T22:20:54.153567231Z], OS[Linux/5.15.0-52-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.2/17.0.2+8]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:03,369Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:03,370Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-1125632303494028097, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,489Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [aggs-matrix-stats]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,489Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [analysis-common]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,489Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [constant-keyword]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,490Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [frozen-indices]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,490Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [ingest-common]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,490Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [ingest-geoip]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,490Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [ingest-user-agent]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,491Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [kibana]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,491Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [lang-expression]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,491Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [lang-mustache]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,491Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [lang-painless]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,491Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [legacy-geo]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,492Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [mapper-extras]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,492Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [mapper-version]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,492Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [parent-join]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,492Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [percolator]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,492Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [rank-eval]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,493Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [reindex]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,493Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [repositories-metering-api]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,493Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [repository-encrypted]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,493Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [repository-url]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,494Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [runtime-fields-common]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,494Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [search-business-rules]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,494Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [searchable-snapshots]" }
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea8150>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:44 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99ea8150>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99ea8150>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 38
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.433938+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_random defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434021+00:00 [info] <0.228.0> Running boot step rabbit_event defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434173+00:00 [info] <0.228.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434241+00:00 [info] <0.228.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434300+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434363+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434473+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434541+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434609+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434701+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434792+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434873+00:00 [info] <0.228.0> Running boot step rabbit_priority_queue defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434909+00:00 [info] <0.228.0> Priority queues enabled, real BQ is rabbit_variable_queue
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.434991+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.435083+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.435143+00:00 [info] <0.228.0> Running boot step kernel_ready defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.435176+00:00 [info] <0.228.0> Running boot step rabbit_sysmon_minder defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.435325+00:00 [info] <0.228.0> Running boot step rabbit_epmd_monitor defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.435891+00:00 [info] <0.568.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.436044+00:00 [info] <0.228.0> Running boot step guid_generator defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.438254+00:00 [info] <0.228.0> Running boot step rabbit_node_monitor defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.438453+00:00 [info] <0.572.0> Starting rabbit_node_monitor
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.438610+00:00 [info] <0.228.0> Running boot step delegate_sup defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.439289+00:00 [info] <0.228.0> Running boot step rabbit_memory_monitor defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.439494+00:00 [info] <0.228.0> Running boot step core_initialized defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.439537+00:00 [info] <0.228.0> Running boot step upgrade_queues defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.448725+00:00 [info] <0.228.0> message_store upgrades: 1 to apply
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.448910+00:00 [info] <0.228.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.449107+00:00 [info] <0.228.0> message_store upgrades: No durable queues found. Skipping message store migration
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.449177+00:00 [info] <0.228.0> message_store upgrades: Removing the old message store data
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.450953+00:00 [info] <0.228.0> message_store upgrades: All upgrades applied successfully
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.461401+00:00 [info] <0.228.0> Running boot step channel_tracking defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.464828+00:00 [info] <0.228.0> Setting up a table for channel tracking on this node: tracked_channel_on_node_rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.468007+00:00 [info] <0.228.0> Setting up a table for channel tracking on this node: tracked_channel_table_per_user_on_node_rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.468279+00:00 [info] <0.228.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.468356+00:00 [info] <0.228.0> Running boot step connection_tracking defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.471604+00:00 [info] <0.228.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.475278+00:00 [info] <0.228.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.478622+00:00 [info] <0.228.0> Setting up a table for per-user connection counting on this node: tracked_connection_table_per_user_on_node_rabbit@7842daa4bac1
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.478980+00:00 [info] <0.228.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.479089+00:00 [info] <0.228.0> Running boot step rabbit_exchange_parameters defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.479200+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.479424+00:00 [info] <0.228.0> Running boot step rabbit_policies defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.479855+00:00 [info] <0.228.0> Running boot step rabbit_policy defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.479948+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_validator defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.480015+00:00 [info] <0.228.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.480085+00:00 [info] <0.228.0> Running boot step rabbit_stream_coordinator defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.480208+00:00 [info] <0.228.0> Running boot step rabbit_vhost_limit defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.480308+00:00 [info] <0.228.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.480360+00:00 [info] <0.228.0> Management plugin: using rates mode 'basic'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.480798+00:00 [info] <0.228.0> Running boot step recovery defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.481813+00:00 [info] <0.228.0> Running boot step empty_db_check defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.481865+00:00 [info] <0.228.0> Will seed default virtual host and user...
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.481954+00:00 [info] <0.228.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.499632+00:00 [info] <0.634.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@7842daa4bac1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.500576+00:00 [info] <0.634.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.503624+00:00 [info] <0.634.0> Starting message stores for vhost '/'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.503844+00:00 [info] <0.638.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.505006+00:00 [info] <0.634.0> Started message store of type transient for vhost '/'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.505167+00:00 [info] <0.642.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.505743+00:00 [warn] <0.642.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.506498+00:00 [info] <0.634.0> Started message store of type persistent for vhost '/'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.506665+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_classic_queue took 5ms
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.506720+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.506772+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.511179+00:00 [info] <0.228.0> Created user 'rabbitmq'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.513389+00:00 [info] <0.228.0> Successfully set user tags for user 'rabbitmq' to [administrator]
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.515626+00:00 [info] <0.228.0> Successfully set permissions for 'rabbitmq' in virtual host '/' to '.*', '.*', '.*'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.515720+00:00 [info] <0.228.0> Running boot step rabbit_looking_glass defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.515749+00:00 [info] <0.228.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.515944+00:00 [info] <0.228.0> Running boot step background_gc defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.516114+00:00 [info] <0.228.0> Running boot step routing_ready defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.516147+00:00 [info] <0.228.0> Running boot step pre_flight defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.516180+00:00 [info] <0.228.0> Running boot step notify_cluster defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.516213+00:00 [info] <0.228.0> Running boot step networking defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.516291+00:00 [info] <0.228.0> Running boot step definition_import_worker_pool defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.516349+00:00 [info] <0.285.0> Starting worker pool 'definition_import_pool' with 8 processes in it
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.517042+00:00 [info] <0.228.0> Running boot step cluster_name defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.517110+00:00 [info] <0.228.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-REXjfX2kemvLdAeBt38j3g'
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.519173+00:00 [info] <0.228.0> Running boot step direct_client defined by app rabbit
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.519484+00:00 [info] <0.679.0> Resetting node maintenance status
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.540735+00:00 [info] <0.706.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.540914+00:00 [info] <0.679.0> Ready to start client connection listeners
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.542608+00:00 [info] <0.750.0> started TCP listener on [::]:5672
nomad_oasis_rabbitmq |  completed with 3 plugins.
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.602060+00:00 [info] <0.679.0> Server startup complete; 3 plugins started.
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.602060+00:00 [info] <0.679.0>  * rabbitmq_prometheus
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.602060+00:00 [info] <0.679.0>  * rabbitmq_web_dispatch
nomad_oasis_rabbitmq | 2022-10-23 08:02:06.602060+00:00 [info] <0.679.0>  * rabbitmq_management_agent
nomad_oasis_rabbitmq | 2022-10-23 08:03:39.308641+00:00 [info] <0.817.0> accepting AMQP connection <0.817.0> (172.21.0.6:50728 -> 172.21.0.2:5672)
nomad_oasis_rabbitmq | 2022-10-23 08:03:39.314719+00:00 [info] <0.817.0> connection <0.817.0> (172.21.0.6:50728 -> 172.21.0.2:5672): user 'rabbitmq' authenticated and granted access to vhost '/'
nomad_oasis_rabbitmq | 2022-10-23 08:03:39.321613+00:00 [info] <0.825.0> accepting AMQP connection <0.825.0> (172.21.0.6:50740 -> 172.21.0.2:5672)
nomad_oasis_rabbitmq | 2022-10-23 08:03:39.323144+00:00 [info] <0.825.0> connection <0.825.0> (172.21.0.6:50740 -> 172.21.0.2:5672): user 'rabbitmq' authenticated and granted access to vhost '/'
nomad_oasis_rabbitmq | 2022-10-23 08:03:39.337133+00:00 [info] <0.843.0> accepting AMQP connection <0.843.0> (172.21.0.6:50746 -> 172.21.0.2:5672)
nomad_oasis_rabbitmq | 2022-10-23 08:03:39.338733+00:00 [info] <0.843.0> connection <0.843.0> (172.21.0.6:50746 -> 172.21.0.2:5672): user 'rabbitmq' authenticated and granted access to vhost '/'
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea8150>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99ea8150>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99ea8150>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:44 GET http://elastic:9200/ [status:N/A request:60.062s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 41
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea63d0>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:44 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99ea63d0>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99ea63d0>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 41
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea63d0>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99ea63d0>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99ea63d0>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:44 GET http://elastic:9200/ [status:N/A request:60.062s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 44
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,494Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [snapshot-repo-test-kit]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,495Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [spatial]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,495Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [transform]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,496Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [transport-netty4]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,496Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [unsigned-long]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,496Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [vector-tile]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,496Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [vectors]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,496Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [wildcard]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,497Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-aggregate-metric]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,497Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-analytics]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,497Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-async]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,497Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-async-search]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,498Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-autoscaling]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,498Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-ccr]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,498Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-core]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,498Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-data-streams]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,498Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-deprecation]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,499Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-enrich]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,499Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-eql]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,499Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-fleet]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,499Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-graph]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,500Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-identity-provider]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,500Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-ilm]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,500Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-logstash]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,500Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-ml]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,500Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-monitoring]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,501Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-ql]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,501Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-rollup]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,501Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-security]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,501Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-shutdown]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,501Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-sql]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,502Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-stack]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,502Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-text-structure]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,502Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-voting-only-node]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,502Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "loaded module [x-pack-watcher]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,503Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "no plugins loaded" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,552Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda5)]], net usable_space [32gb], net total_space [107.4gb], types [ext4]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,553Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "heap size [512mb], compressed ordinary object pointers [true]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:06,635Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "node name [bebbe96de06d], node ID [DcAFsVkgSeOgs9qUNOMdZA], cluster name [docker-cluster], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:11,450Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "[controller/220] [Main.cc@122] controller (64 bit): Version 7.17.1 (Build 6d8a28b39bf223) Copyright (c) 2022 Elasticsearch BV" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:11,860Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:12,722Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:12,723Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:13,333Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:13,365Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:13,405Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:13,941Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:14,474Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "initialized" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:14,474Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "starting ..." }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:14,492Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "persistent cache index loaded" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:14,492Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "deprecation component started" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:14,604Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "publish_address {172.21.0.3:9300}, bound_addresses {0.0.0.0:9300}" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,033Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,034Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "cluster UUID [FaawrxI-STe_x0453IT5hQ]" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,256Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "elected-as-master ([1] nodes joined)[{bebbe96de06d}{DcAFsVkgSeOgs9qUNOMdZA}{b7EjaTI1R9Gt2C75Es3IAQ}{172.21.0.3}{172.21.0.3:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 79, delta: master node changed {previous [], current [{bebbe96de06d}{DcAFsVkgSeOgs9qUNOMdZA}{b7EjaTI1R9Gt2C75Es3IAQ}{172.21.0.3}{172.21.0.3:9300}{cdfhilmrstw}]}" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,337Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "master node changed {previous [], current [{bebbe96de06d}{DcAFsVkgSeOgs9qUNOMdZA}{b7EjaTI1R9Gt2C75Es3IAQ}{172.21.0.3}{172.21.0.3:9300}{cdfhilmrstw}]}, term: 4, version: 79, reason: Publication{term=4, version=79}" }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,390Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "publish_address {172.21.0.3:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,391Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "started", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,674Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "license [beed1a37-55e3-434b-b7b7-f84fb50b9301] mode [basic] - valid", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,675Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,676Z", "level": "WARN", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "deprecation.elasticsearch", "timestamp": "2022-10-23T08:02:15,677Z", "level": "CRITICAL", "component": "o.e.d.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "The default behavior of disabling security on basic licenses is deprecated. In a later version of Elasticsearch, the value of [xpack.security.enabled] will default to \"true\" , regardless of the license level. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security, or explicitly disable security by setting [xpack.security.enabled] to false in elasticsearch.yml", "key": "security_implicitly_disabled", "category": "security", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:15,683Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "recovered [5] indices into cluster_state", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,084Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "updating geoip databases", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,084Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,124Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-Country.mmdb.tmp.gz]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,126Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-ASN.mmdb.tmp.gz]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,126Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-City.mmdb.tmp.gz]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,243Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2022.10.21-000001][0]]]).", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,591Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-Country.mmdb]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:16,649Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-ASN.mmdb]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:17,299Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-1125632303494028097/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-City.mmdb]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA"  }
nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T08:02:36,127Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bebbe96de06d", "message": "exception during geoip databases update", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" , 
nomad_oasis_elastic | "stacktrace": ["java.net.UnknownHostException: geoip.elastic.co",
nomad_oasis_elastic | "at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567) ~[?:?]",
nomad_oasis_elastic | "at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]",
nomad_oasis_elastic | "at java.net.Socket.connect(Socket.java:633) ~[?:?]",
nomad_oasis_elastic | "at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:299) ~[?:?]",
nomad_oasis_elastic | "at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:189) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:175) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]",
nomad_oasis_elastic | "at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]",
nomad_oasis_elastic | "at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:55) ~[ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:97) ~[ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:49) ~[ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:40) ~[ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:141) ~[ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:129) ~[ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:266) [ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.1.jar:7.17.1]",
nomad_oasis_elastic | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
nomad_oasis_elastic | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
nomad_oasis_elastic | "at java.lang.Thread.run(Thread.java:833) [?:?]"] }
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea6650>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:44 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99ea6650>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99ea6650>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 44
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea6650>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99ea6650>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99ea6650>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:44 GET http://elastic:9200/ [status:N/A request:60.061s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 47
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea5910>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:44 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99ea5910>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99ea5910>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 47
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea5910>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99ea5910>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99ea5910>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:44 GET http://elastic:9200/ [status:N/A request:60.060s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 50
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea6bd0>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:44 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99ea6bd0>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99ea6bd0>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 50
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea6bd0>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99ea6bd0>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99ea6bd0>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:45 GET http://elastic:9200/ [status:N/A request:60.060s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 53
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea6e50>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:45 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99ea6e50>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99ea6e50>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 53
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99ea6e50>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99ea6e50>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99ea6e50>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | WARNING  elasticsearch        2022-10-23T08:03:45 GET http://elastic:9200/ [status:N/A request:60.060s]
nomad_oasis_worker |   - lineno: 294
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 56
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99eaf110>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread
nomad_oasis_worker | ERROR    celery.utils.dispatc 2022-10-23T08:03:45 Signal handler <function setup at 0x7faff10feb00> raised: ConnectionError('N/A', "(<urllib3.connection.HTTPConnection object at 0x7faf99eaf110>, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7faf99eaf110>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker |   - lineno: 293
nomad_oasis_worker |   - nomad.commit: f476ce92
nomad_oasis_worker |   - nomad.deployment: oasis
nomad_oasis_worker |   - nomad.service: unknown nomad service
nomad_oasis_worker |   - nomad.version: 1.1.5
nomad_oasis_worker |   - process: 56
nomad_oasis_worker |   - stack_trace: Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
nomad_oasis_worker |     (self._dns_host, self.port), self.timeout, **extra_kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
nomad_oasis_worker |     raise err
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
nomad_oasis_worker |     sock.connect(sa)
nomad_oasis_worker | socket.timeout: timed out
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
nomad_oasis_worker |     method, url, body, retries=Retry(False), headers=request_headers, **kw
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
nomad_oasis_worker |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment
nomad_oasis_worker |     raise six.reraise(type(error), error, _stacktrace)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
nomad_oasis_worker |     raise value
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
nomad_oasis_worker |     chunked=chunked,
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
nomad_oasis_worker |     conn.request(method, url, **httplib_request_kw)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
nomad_oasis_worker |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
nomad_oasis_worker |     self._send_request(method, url, body, headers, encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
nomad_oasis_worker |     self.endheaders(body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
nomad_oasis_worker |     self._send_output(message_body, encode_chunked=encode_chunked)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
nomad_oasis_worker |     self.send(msg)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
nomad_oasis_worker |     self.connect()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
nomad_oasis_worker |     conn = self._new_conn()
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
nomad_oasis_worker |     % (self.host, self.timeout),
nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7faf99eaf110>, 'Connection to elastic timed out. (connect timeout=60)')
nomad_oasis_worker | 
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker | 
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
nomad_oasis_worker |     response = receiver(signal=self, sender=sender, **named)
nomad_oasis_worker |   File "/app/nomad/processing/base.py", line 61, in setup
nomad_oasis_worker |     infrastructure.setup()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 72, in setup
nomad_oasis_worker |     setup_elastic()
nomad_oasis_worker |   File "/app/nomad/infrastructure.py", line 102, in setup_elastic
nomad_oasis_worker |     create_v1_indices()
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices
nomad_oasis_worker |     entry_index.create_index(upsert=True)  # TODO update the existing v0 index
nomad_oasis_worker |   File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index
nomad_oasis_worker |     if not self.elastic_client.indices.exists(index=self.index_name):
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
nomad_oasis_worker |     return func(*args, params=params, headers=headers, **kwargs)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists
nomad_oasis_worker |     "HEAD", _make_path(index), params=params, headers=headers
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request
nomad_oasis_worker |     self._do_verify_elasticsearch(headers=headers, timeout=timeout)
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
nomad_oasis_worker |     raise error
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch
nomad_oasis_worker |     "GET", "/", headers=headers, timeout=timeout
nomad_oasis_worker |   File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request
nomad_oasis_worker |     raise ConnectionError("N/A", str(e), e)
nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x7faf99eaf110>, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x7faf99eaf110>, 'Connection to elastic timed out. (connect timeout=60)'))
nomad_oasis_worker | 
nomad_oasis_worker |   - thread_name: MainThread