Attaching to nomad_oasis_proxy, nomad_oasis_north, nomad_oasis_app, nomad_oasis_worker, nomad_oasis_rabbitmq, nomad_oasis_elastic, nomad_oasis_mongo nomad_oasis_app | ERROR nomad.metainfo 2022-10-23T07:53:19 Fail to generate metainfo. nomad_oasis_app | - nomad.commit: f476ce92 nomad_oasis_app | - nomad.deployment: oasis nomad_oasis_app | - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference. nomad_oasis_app | - nomad.metainfo.target_name: NXxpcs nomad_oasis_app | - nomad.service: unknown nomad service nomad_oasis_app | - nomad.version: 1.1.5 nomad_oasis_app | /usr/local/lib/python3.7/site-packages/optimade/__init__.py:18: DeprecationWarning: v0.17 of the `optimade` package will be the last to support Python 3.7. Please upgrade to Python 3.8+ to use v0.18 and later versions of `optimade`. nomad_oasis_app | DeprecationWarning, nomad_oasis_app | INFO: Started server process [34] nomad_oasis_app | INFO: Waiting for application startup. nomad_oasis_app | WARNING elasticsearch 2022-10-23T07:55:33 GET http://elastic:9200/ [status:N/A request:60.059s] nomad_oasis_app | - lineno: 294 nomad_oasis_app | - nomad.commit: f476ce92 nomad_oasis_app | - nomad.deployment: oasis nomad_oasis_app | - nomad.service: unknown nomad service nomad_oasis_app | - nomad.version: 1.1.5 nomad_oasis_app | - process: 34 nomad_oasis_app | - stack_trace: Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_app | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_app | raise err nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_app | sock.connect(sa) nomad_oasis_app | socket.timeout: timed out nomad_oasis_app | nomad_oasis_app | During handling of the above exception, another exception occurred: nomad_oasis_app | nomad_oasis_app | Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_app | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_app | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:29,019Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "version[7.17.1], pid[7], build[default/docker/e5acb99f822233d62d6444ce45a4543dc1c8059a/2022-02-23T22:20:54.153567231Z], OS[Linux/5.15.0-52-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.2/17.0.2+8]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:29,027Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:29,028Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-7160074408296794099, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,589Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [aggs-matrix-stats]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,589Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [analysis-common]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,589Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [constant-keyword]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,590Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [frozen-indices]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,590Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [ingest-common]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,590Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [ingest-geoip]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,591Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [ingest-user-agent]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,591Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [kibana]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,591Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [lang-expression]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,592Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [lang-mustache]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,592Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [lang-painless]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,592Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [legacy-geo]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,592Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [mapper-extras]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,592Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [mapper-version]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,593Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [parent-join]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,593Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [percolator]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,593Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [rank-eval]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,593Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [reindex]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,594Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [repositories-metering-api]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,594Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [repository-encrypted]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,594Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [repository-url]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,594Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [runtime-fields-common]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,595Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [search-business-rules]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,595Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [searchable-snapshots]" } nomad_oasis_app | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_app | raise value nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_app | chunked=chunked, nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_app | conn.request(method, url, **httplib_request_kw) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_app | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_app | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_app | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_app | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_app | self.send(msg) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_app | self.connect() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_app | conn = self._new_conn() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_app | % (self.host, self.timeout), nomad_oasis_app | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_app | nomad_oasis_app | - thread_name: MainThread nomad_oasis_app | ERROR: Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_app | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_app | raise err nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_app | sock.connect(sa) nomad_oasis_app | socket.timeout: timed out nomad_oasis_app | nomad_oasis_app | During handling of the above exception, another exception occurred: nomad_oasis_app | nomad_oasis_app | Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_app | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_app | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_app | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_app | raise value nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_app | chunked=chunked, nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_app | conn.request(method, url, **httplib_request_kw) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_app | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_app | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_app | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_app | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_app | self.send(msg) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_app | self.connect() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_app | conn = self._new_conn() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_app | % (self.host, self.timeout), nomad_oasis_app | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_app | nomad_oasis_app | During handling of the above exception, another exception occurred: nomad_oasis_app | nomad_oasis_app | Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 540, in lifespan nomad_oasis_app | async for item in self.lifespan_context(app): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 481, in default_lifespan nomad_oasis_app | await self.startup() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 516, in startup nomad_oasis_app | await handler() nomad_oasis_app | File "./nomad/app/main.py", line 96, in startup_event nomad_oasis_app | infrastructure.setup() nomad_oasis_app | File "./nomad/infrastructure.py", line 72, in setup nomad_oasis_app | setup_elastic() nomad_oasis_app | File "./nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_app | create_v1_indices() nomad_oasis_app | File "./nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_app | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_app | File "./nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_app | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_app | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_app | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_app | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_app | raise error nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_app | "GET", "/", headers=headers, timeout=timeout nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_app | raise ConnectionError("N/A", str(e), e) nomad_oasis_app | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_app | nomad_oasis_app | ERROR uvicorn.error 2022-10-23T07:55:33 Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_app | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_app | raise err nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_app | sock.connect(sa) nomad_oasis_app | socket.timeout: timed out nomad_oasis_app | nomad_oasis_app | During handling of the above exception, another exception occurred: nomad_oasis_app | nomad_oasis_app | Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_app | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_app | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_app | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_app | raise value nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_app | chunked=chunked, nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_app | conn.request(method, url, **httplib_request_kw) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_app | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_app | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_app | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_app | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_app | self.send(msg) nomad_oasis_app | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_app | self.connect() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_app | conn = self._new_conn() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_app | % (self.host, self.timeout), nomad_oasis_app | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_app | nomad_oasis_app | During handling of the above exception, another exception occurred: nomad_oasis_app | nomad_oasis_app | Traceback (most recent call last): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 540, in lifespan nomad_oasis_app | async for item in self.lifespan_context(app): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 481, in default_lifespan nomad_oasis_app | await self.startup() nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 516, in startup nomad_oasis_app | await handler() nomad_oasis_app | File "./nomad/app/main.py", line 96, in startup_event nomad_oasis_app | infrastructure.setup() nomad_oasis_app | File "./nomad/infrastructure.py", line 72, in setup nomad_oasis_app | setup_elastic() nomad_oasis_app | File "./nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_app | create_v1_indices() nomad_oasis_app | File "./nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_app | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_app | File "./nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_app | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_app | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_app | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_app | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_app | raise error nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_app | "GET", "/", headers=headers, timeout=timeout nomad_oasis_app | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_app | raise ConnectionError("N/A", str(e), e) nomad_oasis_app | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_app | nomad_oasis_app | - nomad.commit: f476ce92 nomad_oasis_app | - nomad.deployment: oasis nomad_oasis_app | - nomad.service: unknown nomad service nomad_oasis_app | - nomad.version: 1.1.5 nomad_oasis_app | ERROR: Application startup failed. Exiting. nomad_oasis_app | ERROR uvicorn.error 2022-10-23T07:55:33 Application startup failed. Exiting. nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,595Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [snapshot-repo-test-kit]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,595Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [spatial]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,595Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [transform]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,596Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [transport-netty4]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,596Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [unsigned-long]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,596Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [vector-tile]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,596Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [vectors]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,597Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [wildcard]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,597Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-aggregate-metric]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,597Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-analytics]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,597Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-async]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,598Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-async-search]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,598Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-autoscaling]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,598Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-ccr]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,598Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-core]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,599Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-data-streams]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,599Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-deprecation]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,599Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-enrich]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,599Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-eql]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,600Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-fleet]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,600Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-graph]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,600Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-identity-provider]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,600Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-ilm]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,601Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-logstash]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,601Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-ml]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,601Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-monitoring]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,601Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-ql]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,602Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-rollup]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,602Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-security]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,602Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-shutdown]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,602Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-sql]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,602Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-stack]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,603Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-text-structure]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,603Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-voting-only-node]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,603Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "loaded module [x-pack-watcher]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,604Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "no plugins loaded" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,650Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda5)]], net usable_space [32.5gb], net total_space [107.4gb], types [ext4]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,651Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "heap size [512mb], compressed ordinary object pointers [true]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:32,747Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "node name [bffa9d5ca0f5], node ID [DcAFsVkgSeOgs9qUNOMdZA], cluster name [docker-cluster], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:39,265Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "[controller/216] [Main.cc@122] controller (64 bit): Version 7.17.1 (Build 6d8a28b39bf223) Copyright (c) 2022 Elasticsearch BV" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:39,912Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:40,718Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:40,720Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:41,558Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:41,599Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:41,656Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "using discovery type [single-node] and seed hosts providers [settings]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:42,260Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:42,899Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "initialized" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:42,899Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "starting ..." } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:42,918Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "persistent cache index loaded" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:42,918Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "deprecation component started" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,045Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "publish_address {172.20.0.2:9300}, bound_addresses {0.0.0.0:9300}" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,569Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,570Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "cluster UUID [FaawrxI-STe_x0453IT5hQ]" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,729Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "elected-as-master ([1] nodes joined)[{bffa9d5ca0f5}{DcAFsVkgSeOgs9qUNOMdZA}{0_XG1aD9Rfmm_OZ9KfIyIA}{172.20.0.2}{172.20.0.2:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 71, delta: master node changed {previous [], current [{bffa9d5ca0f5}{DcAFsVkgSeOgs9qUNOMdZA}{0_XG1aD9Rfmm_OZ9KfIyIA}{172.20.0.2}{172.20.0.2:9300}{cdfhilmrstw}]}" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,810Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "master node changed {previous [], current [{bffa9d5ca0f5}{DcAFsVkgSeOgs9qUNOMdZA}{0_XG1aD9Rfmm_OZ9KfIyIA}{172.20.0.2}{172.20.0.2:9300}{cdfhilmrstw}]}, term: 3, version: 71, reason: Publication{term=3, version=71}" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,869Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "publish_address {172.20.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:43,869Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "started", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,251Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "license [beed1a37-55e3-434b-b7b7-f84fb50b9301] mode [basic] - valid", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,254Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,254Z", "level": "WARN", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "deprecation.elasticsearch", "timestamp": "2022-10-23T07:52:44,256Z", "level": "CRITICAL", "component": "o.e.d.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "The default behavior of disabling security on basic licenses is deprecated. In a later version of Elasticsearch, the value of [xpack.security.enabled] will default to \"true\" , regardless of the license level. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security, or explicitly disable security by setting [xpack.security.enabled] to false in elasticsearch.yml", "key": "security_implicitly_disabled", "category": "security", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,266Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "recovered [5] indices into cluster_state", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,903Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "updating geoip databases", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,904Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,959Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-Country.mmdb.tmp.gz]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,961Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-ASN.mmdb.tmp.gz]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:44,962Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-City.mmdb.tmp.gz]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:45,156Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2022.10.21-000001][0]]]).", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:45,338Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-Country.mmdb]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:45,400Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-ASN.mmdb]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:52:46,322Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-7160074408296794099/geoip-databases/DcAFsVkgSeOgs9qUNOMdZA/GeoLite2-City.mmdb]", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" } nomad_oasis_elastic | {"type": "server", "timestamp": "2022-10-23T07:53:10,274Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bffa9d5ca0f5", "message": "exception during geoip databases update", "cluster.uuid": "FaawrxI-STe_x0453IT5hQ", "node.id": "DcAFsVkgSeOgs9qUNOMdZA" , nomad_oasis_elastic | "stacktrace": ["java.net.UnknownHostException: geoip.elastic.co", nomad_oasis_elastic | "at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567) ~[?:?]", nomad_oasis_elastic | "at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]", nomad_oasis_elastic | "at java.net.Socket.connect(Socket.java:633) ~[?:?]", nomad_oasis_elastic | "at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:299) ~[?:?]", nomad_oasis_elastic | "at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:266) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:189) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:175) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]", nomad_oasis_elastic | "at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]", nomad_oasis_elastic | "at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:55) ~[ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:97) ~[ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:49) ~[ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:40) ~[ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:141) ~[ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:129) ~[ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:266) [ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.1.jar:7.17.1]", nomad_oasis_elastic | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]", nomad_oasis_elastic | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]", nomad_oasis_elastic | "at java.lang.Thread.run(Thread.java:833) [?:?]"] } nomad_oasis_app | - nomad.commit: f476ce92 nomad_oasis_app | - nomad.deployment: oasis nomad_oasis_app | - nomad.service: unknown nomad service nomad_oasis_app | - nomad.version: 1.1.5 nomad_oasis_app | ERROR nomad.metainfo 2022-10-23T07:55:54 Fail to generate metainfo. nomad_oasis_app | - nomad.commit: f476ce92 nomad_oasis_app | - nomad.deployment: oasis nomad_oasis_app | - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference. nomad_oasis_app | - nomad.metainfo.target_name: NXxpcs nomad_oasis_app | - nomad.service: unknown nomad service nomad_oasis_app | - nomad.version: 1.1.5 nomad_oasis_app | /usr/local/lib/python3.7/site-packages/optimade/__init__.py:18: DeprecationWarning: v0.17 of the `optimade` package will be the last to support Python 3.7. Please upgrade to Python 3.8+ to use v0.18 and later versions of `optimade`. nomad_oasis_app | DeprecationWarning, nomad_oasis_app | INFO: Started server process [32] nomad_oasis_app | INFO: Waiting for application startup. nomad_oasis_app | INFO: Application startup complete. nomad_oasis_app | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) nomad_oasis_app | INFO: 127.0.0.1:39610 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:48178 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:52914 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:38852 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:56812 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:47426 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:58006 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:50418 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:36720 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:33478 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:39006 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:42492 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:47214 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:41712 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:34850 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:49098 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:59088 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:60680 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:59580 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_app | INFO: 127.0.0.1:36172 - "GET /-/health HTTP/1.1" 200 OK nomad_oasis_north | ERROR nomad.metainfo 2022-10-23T07:53:15 Fail to generate metainfo. nomad_oasis_north | - nomad.commit: f476ce92 nomad_oasis_north | - nomad.deployment: oasis nomad_oasis_north | - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference. nomad_oasis_north | - nomad.metainfo.target_name: NXxpcs nomad_oasis_north | - nomad.service: hub nomad_oasis_north | - nomad.version: 1.1.5 nomad_oasis_north | [I 2022-10-23 07:53:22.595 JupyterHub app:2459] Running JupyterHub version 1.4.2 nomad_oasis_north | [I 2022-10-23 07:53:22.596 JupyterHub app:2490] Using Authenticator: builtins.NomadAuthenticator nomad_oasis_north | [I 2022-10-23 07:53:22.596 JupyterHub app:2490] Using Spawner: builtins.NomadDockerSpawner nomad_oasis_north | [I 2022-10-23 07:53:22.596 JupyterHub app:2490] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.2 nomad_oasis_north | [I 2022-10-23 07:53:22.600 JupyterHub app:1579] Writing cookie_secret to /app/jupyterhub_cookie_secret nomad_oasis_north | [I 2022-10-23 07:53:22.645 alembic.runtime.migration migration:204] Context impl SQLiteImpl. nomad_oasis_north | [I 2022-10-23 07:53:22.645 alembic.runtime.migration migration:211] Will assume non-transactional DDL. nomad_oasis_north | [I 2022-10-23 07:53:22.656 alembic.runtime.migration migration:618] Running stamp_revision -> 4dc2d5a8c53c nomad_oasis_north | [I 2022-10-23 07:53:22.736 JupyterHub proxy:497] Generating new CONFIGPROXY_AUTH_TOKEN nomad_oasis_north | [W 2022-10-23 07:53:22.740 JupyterHub app:1808] No admin users, admin interface will be unavailable. nomad_oasis_north | [W 2022-10-23 07:53:22.740 JupyterHub app:1810] Add any administrative users to `c.Authenticator.admin_users` in config. nomad_oasis_north | [I 2022-10-23 07:53:22.740 JupyterHub app:1839] Not using allowed_users. Any authenticated user will be allowed. nomad_oasis_north | [I 2022-10-23 07:53:22.841 JupyterHub app:2529] Initialized 0 spawners in 0.005 seconds nomad_oasis_north | [W 2022-10-23 07:53:22.846 JupyterHub proxy:700] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else... nomad_oasis_north | [I 2022-10-23 07:53:22.846 JupyterHub proxy:703] Starting proxy @ http://:9000/nomad-oasis/north nomad_oasis_north | 07:53:23.312 [ConfigProxy] info: Proxying http://*:9000 to (no default) nomad_oasis_north | 07:53:23.319 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes nomad_oasis_north | 07:53:23.545 [ConfigProxy] info: 200 GET /api/routes nomad_oasis_north | [I 2022-10-23 07:53:23.546 JupyterHub app:2774] Hub API listening on http://0.0.0.0:8081/nomad-oasis/north/hub/ nomad_oasis_north | [I 2022-10-23 07:53:23.546 JupyterHub app:2776] Private Hub API connect url http://north:8081/nomad-oasis/north/hub/ nomad_oasis_north | 07:53:23.550 [ConfigProxy] info: 200 GET /api/routes nomad_oasis_north | [I 2022-10-23 07:53:23.551 JupyterHub proxy:347] Checking routes nomad_oasis_north | [I 2022-10-23 07:53:23.551 JupyterHub proxy:432] Adding route for Hub: /nomad-oasis/north/ => http://north:8081 nomad_oasis_north | 07:53:23.556 [ConfigProxy] info: Adding route /nomad-oasis/north -> http://north:8081 nomad_oasis_north | 07:53:23.557 [ConfigProxy] info: Route added /nomad-oasis/north -> http://north:8081 nomad_oasis_north | 07:53:23.558 [ConfigProxy] info: 201 POST /api/routes/nomad-oasis/north nomad_oasis_north | [I 2022-10-23 07:53:23.559 JupyterHub app:2849] JupyterHub is now running at http://:9000/nomad-oasis/north nomad_oasis_north | 07:58:23.656 [ConfigProxy] info: 200 GET /api/routes nomad_oasis_north | [I 2022-10-23 07:58:23.658 JupyterHub proxy:347] Checking routes nomad_oasis_rabbitmq | 2022-10-23 07:52:26.123127+00:00 [warn] <0.130.0> Overriding Erlang cookie using the value set in the environment nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941630+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941686+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941710+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941749+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941826+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941846+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941861+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:27.941924+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:28.338449+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped nomad_oasis_rabbitmq | 2022-10-23 07:52:28.338601+00:00 [noti] <0.228.0> Logging: switching to configured handler(s); following messages may not be visible in this log output nomad_oasis_rabbitmq | 2022-10-23 07:52:28.357439+00:00 [noti] <0.228.0> Logging: configured log handlers are now ACTIVE nomad_oasis_rabbitmq | 2022-10-23 07:52:28.573374+00:00 [info] <0.228.0> ra: starting system quorum_queues nomad_oasis_rabbitmq | 2022-10-23 07:52:28.573498+00:00 [info] <0.228.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@6d86e9af2836/quorum/rabbit@6d86e9af2836 nomad_oasis_rabbitmq | 2022-10-23 07:52:28.648310+00:00 [info] <0.265.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered nomad_oasis_rabbitmq | 2022-10-23 07:52:28.662915+00:00 [noti] <0.270.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables nomad_oasis_rabbitmq | 2022-10-23 07:52:28.679219+00:00 [info] <0.228.0> ra: starting system coordination nomad_oasis_rabbitmq | 2022-10-23 07:52:28.679297+00:00 [info] <0.228.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@6d86e9af2836/coordination/rabbit@6d86e9af2836 nomad_oasis_rabbitmq | 2022-10-23 07:52:28.680897+00:00 [info] <0.278.0> ra: meta data store initialised for system coordination. 0 record(s) recovered nomad_oasis_rabbitmq | 2022-10-23 07:52:28.681164+00:00 [noti] <0.283.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables nomad_oasis_rabbitmq | 2022-10-23 07:52:28.685176+00:00 [info] <0.228.0> nomad_oasis_rabbitmq | 2022-10-23 07:52:28.685176+00:00 [info] <0.228.0> Starting RabbitMQ 3.9.13 on Erlang 24.3.2 [jit] nomad_oasis_rabbitmq | 2022-10-23 07:52:28.685176+00:00 [info] <0.228.0> Copyright (c) 2007-2022 VMware, Inc. or its affiliates. nomad_oasis_rabbitmq | 2022-10-23 07:52:28.685176+00:00 [info] <0.228.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | ## ## RabbitMQ 3.9.13 nomad_oasis_rabbitmq | ## ## nomad_oasis_rabbitmq | ########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates. nomad_oasis_rabbitmq | ###### ## nomad_oasis_rabbitmq | ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | Erlang: 24.3.2 [jit] nomad_oasis_rabbitmq | TLS Library: OpenSSL - OpenSSL 1.1.1n 15 Mar 2022 nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | Doc guides: https://rabbitmq.com/documentation.html nomad_oasis_rabbitmq | Support: https://rabbitmq.com/contact.html nomad_oasis_rabbitmq | Tutorials: https://rabbitmq.com/getstarted.html nomad_oasis_rabbitmq | Monitoring: https://rabbitmq.com/monitoring.html nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | Logs: /var/log/rabbitmq/rabbit@6d86e9af2836_upgrade.log nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | Config file(s): /etc/rabbitmq/conf.d/10-default-guest-user.conf nomad_oasis_rabbitmq | /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf nomad_oasis_rabbitmq | nomad_oasis_rabbitmq | Starting broker...2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> node : rabbit@6d86e9af2836 nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> home dir : /var/lib/rabbitmq nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> config file(s) : /etc/rabbitmq/conf.d/10-default-guest-user.conf nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> : /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> cookie hash : P6euPuJ+9iKMq+WqGJEzBA== nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> log(s) : /var/log/rabbitmq/rabbit@6d86e9af2836_upgrade.log nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> : nomad_oasis_rabbitmq | 2022-10-23 07:52:28.687412+00:00 [info] <0.228.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@6d86e9af2836 nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870162+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870291+00:00 [info] <0.228.0> Feature flags: [ ] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870346+00:00 [info] <0.228.0> Feature flags: [ ] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870397+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870430+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870518+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870550+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870580+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870610+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:28.870709+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:31.872919+00:00 [info] <0.228.0> Running boot step pre_boot defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.872988+00:00 [info] <0.228.0> Running boot step rabbit_global_counters defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.873388+00:00 [info] <0.228.0> Running boot step rabbit_osiris_metrics defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.873507+00:00 [info] <0.228.0> Running boot step rabbit_core_metrics defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.874275+00:00 [info] <0.228.0> Running boot step rabbit_alarm defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.882077+00:00 [info] <0.296.0> Memory high watermark set to 3142 MiB (3295536742 bytes) of 7857 MiB (8238841856 bytes) total nomad_oasis_rabbitmq | 2022-10-23 07:52:31.887156+00:00 [info] <0.298.0> Enabling free disk space monitoring nomad_oasis_rabbitmq | 2022-10-23 07:52:31.887220+00:00 [info] <0.298.0> Disk free limit set to 50MB nomad_oasis_rabbitmq | 2022-10-23 07:52:31.891455+00:00 [info] <0.228.0> Running boot step code_server_cache defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.891713+00:00 [info] <0.228.0> Running boot step file_handle_cache defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.892117+00:00 [info] <0.303.0> Limiting to approx 1048479 file handles (943629 sockets) nomad_oasis_rabbitmq | 2022-10-23 07:52:31.892306+00:00 [info] <0.304.0> FHC read buffering: OFF nomad_oasis_rabbitmq | 2022-10-23 07:52:31.892363+00:00 [info] <0.304.0> FHC write buffering: ON nomad_oasis_rabbitmq | 2022-10-23 07:52:31.892985+00:00 [info] <0.228.0> Running boot step worker_pool defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.893103+00:00 [info] <0.285.0> Will use 8 processes for default worker pool nomad_oasis_rabbitmq | 2022-10-23 07:52:31.893145+00:00 [info] <0.285.0> Starting worker pool 'worker_pool' with 8 processes in it nomad_oasis_rabbitmq | 2022-10-23 07:52:31.893909+00:00 [info] <0.228.0> Running boot step database defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:31.894250+00:00 [info] <0.228.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@6d86e9af2836 is empty. Assuming we need to join an existing cluster or initialise from scratch... nomad_oasis_rabbitmq | 2022-10-23 07:52:31.894337+00:00 [info] <0.228.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config nomad_oasis_rabbitmq | 2022-10-23 07:52:31.894387+00:00 [info] <0.228.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config nomad_oasis_rabbitmq | 2022-10-23 07:52:31.894486+00:00 [info] <0.228.0> All discovered existing cluster peers: nomad_oasis_rabbitmq | 2022-10-23 07:52:31.894529+00:00 [info] <0.228.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot. nomad_oasis_rabbitmq | 2022-10-23 07:52:31.898647+00:00 [noti] <0.44.0> Application mnesia exited with reason: stopped nomad_oasis_rabbitmq | 2022-10-23 07:52:31.984185+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_rabbitmq | 2022-10-23 07:52:31.984340+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_rabbitmq | 2022-10-23 07:52:31.996376+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_rabbitmq | 2022-10-23 07:52:31.996498+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_rabbitmq | 2022-10-23 07:52:31.996569+00:00 [info] <0.228.0> Feature flag `drop_unroutable_metric`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:31.996695+00:00 [info] <0.228.0> Feature flag `drop_unroutable_metric`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997684+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997745+00:00 [info] <0.228.0> Feature flags: [~] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997781+00:00 [info] <0.228.0> Feature flags: [ ] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997798+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997870+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997890+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997905+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997920+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997941+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:31.997972+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.009095+00:00 [info] <0.228.0> Feature flag `drop_unroutable_metric`: mark as enabled=true nomad_oasis_rabbitmq | 2022-10-23 07:52:32.010896+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.010972+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011006+00:00 [info] <0.228.0> Feature flags: [ ] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011032+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011058+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011085+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011133+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011162+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011191+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.011256+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.020767+00:00 [info] <0.228.0> Feature flag `empty_basic_get_metric`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.020952+00:00 [info] <0.228.0> Feature flag `empty_basic_get_metric`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022296+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022390+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022424+00:00 [info] <0.228.0> Feature flags: [~] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022475+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022517+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022543+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022576+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022628+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022720+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.022780+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.038056+00:00 [info] <0.228.0> Feature flag `empty_basic_get_metric`: mark as enabled=true nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040311+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040379+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040428+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040457+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040486+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040537+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040574+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040613+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040654+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.040701+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.050329+00:00 [info] <0.228.0> Feature flag `implicit_default_bindings`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.050419+00:00 [info] <0.228.0> Feature flag `implicit_default_bindings`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:32.051716+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052003+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052042+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052072+00:00 [info] <0.228.0> Feature flags: [~] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052115+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052162+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052191+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052216+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052295+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.052346+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.066283+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 0 retries left nomad_oasis_rabbitmq | 2022-10-23 07:52:32.066460+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_rabbitmq | 2022-10-23 07:52:32.066529+00:00 [info] <0.228.0> Feature flag `implicit_default_bindings`: mark as enabled=true nomad_oasis_worker | ERROR nomad.metainfo 2022-10-23T07:53:13 Fail to generate metainfo. nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.metainfo.exe_info: Unit NX_INT is not supported for delay_difference. nomad_oasis_worker | - nomad.metainfo.target_name: NXxpcs nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:53:44 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 2.00 seconds... (1/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:54:00 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 4.00 seconds... (2/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:54:18 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 6.00 seconds... (3/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:23 GET http://elastic:9200/ [status:N/A request:60.046s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 35 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:23 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 35 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_rabbitmq | 2022-10-23 07:52:32.068835+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.068927+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.068996+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069031+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069061+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069093+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069130+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069166+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069196+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.069244+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.080243+00:00 [info] <0.228.0> Feature flag `maintenance_mode_status`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.080350+00:00 [info] <0.228.0> Feature flag `maintenance_mode_status`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081490+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081730+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081795+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081837+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081880+00:00 [info] <0.228.0> Feature flags: [~] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081917+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.081964+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.082003+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.082051+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.082101+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.096187+00:00 [info] <0.228.0> Creating table rabbit_node_maintenance_states for feature flag `maintenance_mode_status` nomad_oasis_rabbitmq | 2022-10-23 07:52:32.107254+00:00 [info] <0.228.0> Feature flag `maintenance_mode_status`: mark as enabled=true nomad_oasis_rabbitmq | 2022-10-23 07:52:32.109759+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.109846+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.109943+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110000+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110032+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110110+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110144+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110174+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110200+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.110278+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.121671+00:00 [info] <0.228.0> Feature flag `quorum_queue`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.121776+00:00 [info] <0.228.0> Feature flag `quorum_queue`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:32.122918+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123045+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123117+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123151+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123196+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123222+00:00 [info] <0.228.0> Feature flags: [~] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123245+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123267+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123302+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.123328+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.138725+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_rabbitmq | 2022-10-23 07:52:32.139228+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_rabbitmq | 2022-10-23 07:52:32.139267+00:00 [info] <0.228.0> Feature flag `quorum_queue`: migrating Mnesia table rabbit_queue... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.158376+00:00 [info] <0.228.0> Feature flag `quorum_queue`: migrating Mnesia table rabbit_durable_queue... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.172675+00:00 [info] <0.228.0> Feature flag `quorum_queue`: Mnesia tables migration done nomad_oasis_rabbitmq | 2022-10-23 07:52:32.172755+00:00 [info] <0.228.0> Feature flag `quorum_queue`: mark as enabled=true nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174622+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174706+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174773+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174805+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174833+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174861+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174887+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174912+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174936+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.174976+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.187916+00:00 [info] <0.228.0> Feature flag `stream_queue`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.188315+00:00 [info] <0.228.0> Feature flag `stream_queue`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189634+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189716+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189757+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189822+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189879+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189933+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.189985+00:00 [info] <0.228.0> Feature flags: [~] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.190058+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.190123+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.190144+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.201645+00:00 [info] <0.228.0> Feature flag `stream_queue`: mark as enabled=true nomad_oasis_rabbitmq | 2022-10-23 07:52:32.203936+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204010+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204037+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204070+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204089+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204107+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204142+00:00 [info] <0.228.0> Feature flags: [x] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204185+00:00 [info] <0.228.0> Feature flags: [ ] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204224+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.204274+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.215655+00:00 [info] <0.228.0> Feature flag `user_limits`: supported, attempt to enable... nomad_oasis_rabbitmq | 2022-10-23 07:52:32.215985+00:00 [info] <0.228.0> Feature flag `user_limits`: mark as enabled=state_changing nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217056+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217104+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217148+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217190+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217220+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217254+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217286+00:00 [info] <0.228.0> Feature flags: [x] stream_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217334+00:00 [info] <0.228.0> Feature flags: [~] user_limits nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217365+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_rabbitmq | 2022-10-23 07:52:32.217395+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_rabbitmq | 2022-10-23 07:52:32.230377+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_rabbitmq | 2022-10-23 07:52:32.230572+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_rabbitmq | 2022-10-23 07:52:32.245124+00:00 [info] <0.228.0> Feature flag `user_limits`: mark as enabled=true nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247387+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247449+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247525+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247561+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247596+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247678+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:23 GET http://elastic:9200/ [status:N/A request:60.069s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 38 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:23 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 38 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247725+00:00 [info] <0.228.0> Feature flags: [x] stream_queue nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247765+00:00 [info] <0.228.0> Feature flags: [x] user_limits nomad_oasis_worker | raise err nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247809+00:00 [info] <0.228.0> Feature flags: [ ] virtual_host_metadata nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_rabbitmq | 2022-10-23 07:52:32.247854+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_worker | sock.connect(sa) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.261189+00:00 [info] <0.228.0> Feature flag `virtual_host_metadata`: supported, attempt to enable... nomad_oasis_worker | socket.timeout: timed out nomad_oasis_rabbitmq | 2022-10-23 07:52:32.261290+00:00 [info] <0.228.0> Feature flag `virtual_host_metadata`: mark as enabled=state_changing nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262387+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262471+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262535+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262578+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262628+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262673+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262708+00:00 [info] <0.228.0> Feature flags: [x] stream_queue nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262755+00:00 [info] <0.228.0> Feature flags: [x] user_limits nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262807+00:00 [info] <0.228.0> Feature flags: [~] virtual_host_metadata nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_rabbitmq | 2022-10-23 07:52:32.262854+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.273992+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_rabbitmq | 2022-10-23 07:52:32.274095+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_rabbitmq | 2022-10-23 07:52:32.291178+00:00 [info] <0.228.0> Feature flag `virtual_host_metadata`: mark as enabled=true nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294247+00:00 [info] <0.228.0> Feature flags: list of feature flags found: nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294366+00:00 [info] <0.228.0> Feature flags: [x] drop_unroutable_metric nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294401+00:00 [info] <0.228.0> Feature flags: [x] empty_basic_get_metric nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294445+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294473+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294535+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294561+00:00 [info] <0.228.0> Feature flags: [x] stream_queue nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294584+00:00 [info] <0.228.0> Feature flags: [x] user_limits nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294608+00:00 [info] <0.228.0> Feature flags: [x] virtual_host_metadata nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.294678+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_rabbitmq | 2022-10-23 07:52:32.307154+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_rabbitmq | 2022-10-23 07:52:32.307299+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.320534+00:00 [info] <0.228.0> Waiting for Mnesia tables for 30000 ms, 9 retries left nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.320691+00:00 [info] <0.228.0> Successfully synced tables from a peer nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.320736+00:00 [info] <0.228.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration. nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_rabbitmq | 2022-10-23 07:52:32.320823+00:00 [info] <0.228.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_rabbitmq | 2022-10-23 07:52:32.320926+00:00 [info] <0.228.0> Running boot step database_sync defined by app rabbit nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321072+00:00 [info] <0.228.0> Running boot step feature_flags defined by app rabbit nomad_oasis_worker | infrastructure.setup() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321187+00:00 [info] <0.228.0> Running boot step codec_correctness_check defined by app rabbit nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321228+00:00 [info] <0.228.0> Running boot step external_infrastructure defined by app rabbit nomad_oasis_worker | setup_elastic() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321256+00:00 [info] <0.228.0> Running boot step rabbit_registry defined by app rabbit nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321390+00:00 [info] <0.228.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321519+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_random defined by app rabbit nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321605+00:00 [info] <0.228.0> Running boot step rabbit_event defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321782+00:00 [info] <0.228.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321874+00:00 [info] <0.228.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.321960+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_direct defined by app rabbit nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322030+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322130+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_headers defined by app rabbit nomad_oasis_worker | raise error nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322230+00:00 [info] <0.228.0> Running boot step rabbit_exchange_type_topic defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322314+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322402+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322484+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322567+00:00 [info] <0.228.0> Running boot step rabbit_priority_queue defined by app rabbit nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322608+00:00 [info] <0.228.0> Priority queues enabled, real BQ is rabbit_variable_queue nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:24 GET http://elastic:9200/ [status:N/A request:60.061s] nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322707+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_client_local defined by app rabbit nomad_oasis_worker | - lineno: 294 nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322812+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322893+00:00 [info] <0.228.0> Running boot step kernel_ready defined by app rabbit nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_rabbitmq | 2022-10-23 07:52:32.322938+00:00 [info] <0.228.0> Running boot step rabbit_sysmon_minder defined by app rabbit nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_rabbitmq | 2022-10-23 07:52:32.323176+00:00 [info] <0.228.0> Running boot step rabbit_epmd_monitor defined by app rabbit nomad_oasis_worker | - process: 41 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_rabbitmq | 2022-10-23 07:52:32.324082+00:00 [info] <0.569.0> epmd monitor knows us, inter-node communication (distribution) port: 25672 nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_rabbitmq | 2022-10-23 07:52:32.324392+00:00 [info] <0.228.0> Running boot step guid_generator defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_rabbitmq | 2022-10-23 07:52:32.326995+00:00 [info] <0.228.0> Running boot step rabbit_node_monitor defined by app rabbit nomad_oasis_worker | raise err nomad_oasis_rabbitmq | 2022-10-23 07:52:32.327252+00:00 [info] <0.573.0> Starting rabbit_node_monitor nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_rabbitmq | 2022-10-23 07:52:32.327401+00:00 [info] <0.228.0> Running boot step delegate_sup defined by app rabbit nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_rabbitmq | 2022-10-23 07:52:32.328182+00:00 [info] <0.228.0> Running boot step rabbit_memory_monitor defined by app rabbit nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.328468+00:00 [info] <0.228.0> Running boot step core_initialized defined by app rabbit nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_rabbitmq | 2022-10-23 07:52:32.328510+00:00 [info] <0.228.0> Running boot step upgrade_queues defined by app rabbit nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.337586+00:00 [info] <0.228.0> message_store upgrades: 1 to apply nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_rabbitmq | 2022-10-23 07:52:32.337829+00:00 [info] <0.228.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.338106+00:00 [info] <0.228.0> message_store upgrades: No durable queues found. Skipping message store migration nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_rabbitmq | 2022-10-23 07:52:32.338186+00:00 [info] <0.228.0> message_store upgrades: Removing the old message store data nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_rabbitmq | 2022-10-23 07:52:32.338763+00:00 [info] <0.228.0> message_store upgrades: All upgrades applied successfully nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_rabbitmq | 2022-10-23 07:52:32.349471+00:00 [info] <0.228.0> Running boot step channel_tracking defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_rabbitmq | 2022-10-23 07:52:32.353144+00:00 [info] <0.228.0> Setting up a table for channel tracking on this node: tracked_channel_on_node_rabbit@6d86e9af2836 nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.356871+00:00 [info] <0.228.0> Setting up a table for channel tracking on this node: tracked_channel_table_per_user_on_node_rabbit@6d86e9af2836 nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_rabbitmq | 2022-10-23 07:52:32.357353+00:00 [info] <0.228.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit nomad_oasis_worker | raise value nomad_oasis_rabbitmq | 2022-10-23 07:52:32.357522+00:00 [info] <0.228.0> Running boot step connection_tracking defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_rabbitmq | 2022-10-23 07:52:32.361099+00:00 [info] <0.228.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@6d86e9af2836 nomad_oasis_worker | chunked=chunked, nomad_oasis_rabbitmq | 2022-10-23 07:52:32.364536+00:00 [info] <0.228.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@6d86e9af2836 nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.368662+00:00 [info] <0.228.0> Setting up a table for per-user connection counting on this node: tracked_connection_table_per_user_on_node_rabbit@6d86e9af2836 nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.369038+00:00 [info] <0.228.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.369156+00:00 [info] <0.228.0> Running boot step rabbit_exchange_parameters defined by app rabbit nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.369424+00:00 [info] <0.228.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.369708+00:00 [info] <0.228.0> Running boot step rabbit_policies defined by app rabbit nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370203+00:00 [info] <0.228.0> Running boot step rabbit_policy defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370351+00:00 [info] <0.228.0> Running boot step rabbit_queue_location_validator defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370426+00:00 [info] <0.228.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370519+00:00 [info] <0.228.0> Running boot step rabbit_stream_coordinator defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370737+00:00 [info] <0.228.0> Running boot step rabbit_vhost_limit defined by app rabbit nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370847+00:00 [info] <0.228.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_rabbitmq | 2022-10-23 07:52:32.370890+00:00 [info] <0.228.0> Management plugin: using rates mode 'basic' nomad_oasis_worker | self.send(msg) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.371333+00:00 [info] <0.228.0> Running boot step recovery defined by app rabbit nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_rabbitmq | 2022-10-23 07:52:32.372317+00:00 [info] <0.228.0> Running boot step empty_db_check defined by app rabbit nomad_oasis_worker | self.connect() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.372369+00:00 [info] <0.228.0> Will seed default virtual host and user... nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_rabbitmq | 2022-10-23 07:52:32.372479+00:00 [info] <0.228.0> Adding vhost '/' (description: 'Default virtual host', tags: []) nomad_oasis_worker | conn = self._new_conn() nomad_oasis_rabbitmq | 2022-10-23 07:52:32.389574+00:00 [info] <0.634.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@6d86e9af2836/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_rabbitmq | 2022-10-23 07:52:32.391045+00:00 [info] <0.634.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048' nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_rabbitmq | 2022-10-23 07:52:32.394513+00:00 [info] <0.634.0> Starting message stores for vhost '/' nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_rabbitmq | 2022-10-23 07:52:32.394772+00:00 [info] <0.638.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index nomad_oasis_worker | nomad_oasis_rabbitmq | 2022-10-23 07:52:32.396117+00:00 [info] <0.634.0> Started message store of type transient for vhost '/' nomad_oasis_worker | - thread_name: MainThread nomad_oasis_rabbitmq | 2022-10-23 07:52:32.396360+00:00 [info] <0.642.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:24 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_rabbitmq | 2022-10-23 07:52:32.397282+00:00 [warn] <0.642.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch nomad_oasis_worker | - lineno: 293 nomad_oasis_rabbitmq | 2022-10-23 07:52:32.398182+00:00 [info] <0.634.0> Started message store of type persistent for vhost '/' nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 41 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:24 GET http://elastic:9200/ [status:N/A request:60.059s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 44 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:24 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 44 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_rabbitmq | 2022-10-23 07:52:32.398401+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_classic_queue took 6ms nomad_oasis_rabbitmq | 2022-10-23 07:52:32.398467+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms nomad_oasis_rabbitmq | 2022-10-23 07:52:32.398502+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_stream_queue took 0ms nomad_oasis_rabbitmq | 2022-10-23 07:52:32.403033+00:00 [info] <0.228.0> Created user 'rabbitmq' nomad_oasis_rabbitmq | 2022-10-23 07:52:32.405278+00:00 [info] <0.228.0> Successfully set user tags for user 'rabbitmq' to [administrator] nomad_oasis_rabbitmq | 2022-10-23 07:52:32.407476+00:00 [info] <0.228.0> Successfully set permissions for 'rabbitmq' in virtual host '/' to '.*', '.*', '.*' nomad_oasis_rabbitmq | 2022-10-23 07:52:32.407554+00:00 [info] <0.228.0> Running boot step rabbit_looking_glass defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.407611+00:00 [info] <0.228.0> Running boot step rabbit_core_metrics_gc defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.407799+00:00 [info] <0.228.0> Running boot step background_gc defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.407981+00:00 [info] <0.228.0> Running boot step routing_ready defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.408050+00:00 [info] <0.228.0> Running boot step pre_flight defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.408088+00:00 [info] <0.228.0> Running boot step notify_cluster defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.408147+00:00 [info] <0.228.0> Running boot step networking defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.408192+00:00 [info] <0.228.0> Running boot step definition_import_worker_pool defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.408284+00:00 [info] <0.285.0> Starting worker pool 'definition_import_pool' with 8 processes in it nomad_oasis_rabbitmq | 2022-10-23 07:52:32.409064+00:00 [info] <0.228.0> Running boot step cluster_name defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.409188+00:00 [info] <0.228.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-4TP07XkcPBGRgnY_Ym5iVg' nomad_oasis_rabbitmq | 2022-10-23 07:52:32.411678+00:00 [info] <0.228.0> Running boot step direct_client defined by app rabbit nomad_oasis_rabbitmq | 2022-10-23 07:52:32.411938+00:00 [info] <0.679.0> Resetting node maintenance status nomad_oasis_rabbitmq | 2022-10-23 07:52:32.429286+00:00 [info] <0.706.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692 nomad_oasis_rabbitmq | 2022-10-23 07:52:32.429471+00:00 [info] <0.679.0> Ready to start client connection listeners nomad_oasis_rabbitmq | 2022-10-23 07:52:32.431760+00:00 [info] <0.750.0> started TCP listener on [::]:5672 nomad_oasis_rabbitmq | completed with 3 plugins. nomad_oasis_rabbitmq | 2022-10-23 07:52:32.510073+00:00 [info] <0.679.0> Server startup complete; 3 plugins started. nomad_oasis_rabbitmq | 2022-10-23 07:52:32.510073+00:00 [info] <0.679.0> * rabbitmq_prometheus nomad_oasis_rabbitmq | 2022-10-23 07:52:32.510073+00:00 [info] <0.679.0> * rabbitmq_web_dispatch nomad_oasis_rabbitmq | 2022-10-23 07:52:32.510073+00:00 [info] <0.679.0> * rabbitmq_management_agent nomad_oasis_rabbitmq | 2022-10-23 07:57:41.096226+00:00 [info] <0.984.0> accepting AMQP connection <0.984.0> (172.20.0.6:36678 -> 172.20.0.4:5672) nomad_oasis_rabbitmq | 2022-10-23 07:57:41.112011+00:00 [info] <0.984.0> connection <0.984.0> (172.20.0.6:36678 -> 172.20.0.4:5672): user 'rabbitmq' authenticated and granted access to vhost '/' nomad_oasis_rabbitmq | 2022-10-23 07:57:41.133585+00:00 [info] <0.992.0> accepting AMQP connection <0.992.0> (172.20.0.6:36692 -> 172.20.0.4:5672) nomad_oasis_rabbitmq | 2022-10-23 07:57:41.139005+00:00 [info] <0.992.0> connection <0.992.0> (172.20.0.6:36692 -> 172.20.0.4:5672): user 'rabbitmq' authenticated and granted access to vhost '/' nomad_oasis_rabbitmq | 2022-10-23 07:57:41.181719+00:00 [info] <0.1010.0> accepting AMQP connection <0.1010.0> (172.20.0.6:36698 -> 172.20.0.4:5672) nomad_oasis_rabbitmq | 2022-10-23 07:57:41.184972+00:00 [info] <0.1010.0> connection <0.1010.0> (172.20.0.6:36698 -> 172.20.0.4:5672): user 'rabbitmq' authenticated and granted access to vhost '/' nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:24 GET http://elastic:9200/ [status:N/A request:60.060s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 47 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:24 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 47 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:30 GET http://elastic:9200/ [status:N/A request:60.274s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 50 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:30 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 50 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:30 GET http://elastic:9200/ [status:N/A request:60.060s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 53 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:30 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 53 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | WARNING elasticsearch 2022-10-23T07:54:30 GET http://elastic:9200/ [status:N/A request:60.061s] nomad_oasis_worker | - lineno: 294 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 56 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.utils.dispatc 2022-10-23T07:54:30 Signal handler raised: ConnectionError('N/A', "(, 'Connection to elastic timed out. (connect timeout=60)')", ConnectTimeoutError(, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | - lineno: 293 nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | - process: 56 nomad_oasis_worker | - stack_trace: Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn nomad_oasis_worker | (self._dns_host, self.port), self.timeout, **extra_kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection nomad_oasis_worker | raise err nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection nomad_oasis_worker | sock.connect(sa) nomad_oasis_worker | socket.timeout: timed out nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request nomad_oasis_worker | method, url, body, retries=Retry(False), headers=request_headers, **kw nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen nomad_oasis_worker | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 525, in increment nomad_oasis_worker | raise six.reraise(type(error), error, _stacktrace) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise nomad_oasis_worker | raise value nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen nomad_oasis_worker | chunked=chunked, nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request nomad_oasis_worker | conn.request(method, url, **httplib_request_kw) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request nomad_oasis_worker | super(HTTPConnection, self).request(method, url, body=body, headers=headers) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1281, in request nomad_oasis_worker | self._send_request(method, url, body, headers, encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request nomad_oasis_worker | self.endheaders(body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders nomad_oasis_worker | self._send_output(message_body, encode_chunked=encode_chunked) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output nomad_oasis_worker | self.send(msg) nomad_oasis_worker | File "/usr/local/lib/python3.7/http/client.py", line 976, in send nomad_oasis_worker | self.connect() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect nomad_oasis_worker | conn = self._new_conn() nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn nomad_oasis_worker | % (self.host, self.timeout), nomad_oasis_worker | urllib3.exceptions.ConnectTimeoutError: (, 'Connection to elastic timed out. (connect timeout=60)') nomad_oasis_worker | nomad_oasis_worker | During handling of the above exception, another exception occurred: nomad_oasis_worker | nomad_oasis_worker | Traceback (most recent call last): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send nomad_oasis_worker | response = receiver(signal=self, sender=sender, **named) nomad_oasis_worker | File "/app/nomad/processing/base.py", line 61, in setup nomad_oasis_worker | infrastructure.setup() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 72, in setup nomad_oasis_worker | setup_elastic() nomad_oasis_worker | File "/app/nomad/infrastructure.py", line 102, in setup_elastic nomad_oasis_worker | create_v1_indices() nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 910, in create_indices nomad_oasis_worker | entry_index.create_index(upsert=True) # TODO update the existing v0 index nomad_oasis_worker | File "/app/nomad/metainfo/elasticsearch_extension.py", line 480, in create_index nomad_oasis_worker | if not self.elastic_client.indices.exists(index=self.index_name): nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped nomad_oasis_worker | return func(*args, params=params, headers=headers, **kwargs) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 372, in exists nomad_oasis_worker | "HEAD", _make_path(index), params=params, headers=headers nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 417, in perform_request nomad_oasis_worker | self._do_verify_elasticsearch(headers=headers, timeout=timeout) nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch nomad_oasis_worker | raise error nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 570, in _do_verify_elasticsearch nomad_oasis_worker | "GET", "/", headers=headers, timeout=timeout nomad_oasis_worker | File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 280, in perform_request nomad_oasis_worker | raise ConnectionError("N/A", str(e), e) nomad_oasis_worker | elasticsearch.exceptions.ConnectionError: ConnectionError((, 'Connection to elastic timed out. (connect timeout=60)')) caused by: ConnectTimeoutError((, 'Connection to elastic timed out. (connect timeout=60)')) nomad_oasis_worker | nomad_oasis_worker | - thread_name: MainThread nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:54:38 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 8.00 seconds... (4/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:55:00 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 10.00 seconds... (5/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:55:24 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 12.00 seconds... (6/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:55:50 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 14.00 seconds... (7/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:56:18 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 16.00 seconds... (8/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:56:49 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 18.00 seconds... (9/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5 nomad_oasis_worker | ERROR celery.worker.consum 2022-10-23T07:57:21 consumer: Cannot connect to amqp://rabbitmq:**@rabbitmq:5672//: timed out. nomad_oasis_worker | Trying again in 20.00 seconds... (10/100) nomad_oasis_worker | nomad_oasis_worker | - nomad.commit: f476ce92 nomad_oasis_worker | - nomad.deployment: oasis nomad_oasis_worker | - nomad.service: unknown nomad service nomad_oasis_worker | - nomad.version: 1.1.5