NOMAD-Oasis : Dependency failed to start

NOMAD and FAIRmat
1

I get the following:

[ec-sherinsu@u-geovis01 nomad-oasis]$ sudo docker compose up -d
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string. 
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string. 
[+] Running 8/8
 ✔ Network nomad_oasis_network     Created                                                                                                                               0.3s 
 ✔ Container nomad_oasis_rabbitmq  Healthy                                                                                                                              11.0s 
 ✔ Container nomad_oasis_elastic   Healthy                                                                                                                              17.5s 
 ✔ Container nomad_oasis_mongo     Healthy                                                                                                                               6.5s 
 ✔ Container nomad_oasis_north     Healthy                                                                                                                              22.1s 
 ✘ Container nomad_oasis_app       Error                                                                                                                               622.1s 
 ✔ Container nomad_oasis_worker    Started                                                                                                                              18.0s 
 ✔ Container nomad_oasis_proxy     Created                                                                                                                               0.0s 
dependency failed to start: container nomad_oasis_app is unhealthy
[ec-sherinsu@u-geovis01 nomad-oasis]$

I see the following in my logs:

[ec-sherinsu@u-geovis01 configs]$ sudo docker logs nomad_oasis_app
[2025-01-16 12:58:25 +0000] [6] [INFO] Starting gunicorn 21.2.0
[2025-01-16 12:58:25 +0000] [6] [INFO] Listening at: http://0.0.0.0:8000 (6)
[2025-01-16 12:58:25 +0000] [6] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2025-01-16 12:58:25 +0000] [22] [INFO] Booting worker with pid: 22
[2025-01-16 12:58:25 +0000] [23] [INFO] Booting worker with pid: 23
[2025-01-16 12:58:25 +0000] [24] [INFO] Booting worker with pid: 24
[2025-01-16 12:58:25 +0000] [25] [INFO] Booting worker with pid: 25
[2025-01-16 13:08:25 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:22)
[2025-01-16 13:08:25 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:23)
[2025-01-16 13:08:25 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:24)
[2025-01-16 13:08:25 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:25)
[2025-01-16 13:08:26 +0000] [6] [ERROR] Worker (pid:22) was sent SIGKILL! Perhaps out of memory?
[2025-01-16 13:08:26 +0000] [6] [ERROR] Worker (pid:24) was sent SIGKILL! Perhaps out of memory?
[2025-01-16 13:08:26 +0000] [6] [ERROR] Worker (pid:25) was sent SIGKILL! Perhaps out of memory?
[2025-01-16 13:08:26 +0000] [219] [INFO] Booting worker with pid: 219
[2025-01-16 13:08:26 +0000] [6] [ERROR] Worker (pid:23) was sent SIGKILL! Perhaps out of memory?
[2025-01-16 13:08:26 +0000] [220] [INFO] Booting worker with pid: 220
[2025-01-16 13:08:26 +0000] [221] [INFO] Booting worker with pid: 221
[2025-01-16 13:08:26 +0000] [222] [INFO] Booting worker with pid: 222

Anyone know how I can find out the dependency that couldn’t start? Should I enable something like detailed logging? If yes, how?

2

Hi @sherin!

This looks like an issue in the uvicorn workers that we use for the app. Could be that they are really running out of memory, but morelikely an issue with the uvicorn library.

@ahmedilyas: Did we have a similar issue with someone else, and the solution was to somehow change the parallel worker setup? I vaguely remember something like this.

5

Thanks for your reply. It was not a running out of memory issue. I tried increasing the timeout values in the app health check. I was able to start the app after that.