Running Oasis in a Kubernetes cluster

NOMAD and FAIRmat
1

Dear Nomad developers,

I would really appreciate some simple howto or documentation about how to run Oasis in a Kubernetes cluster. The docs claim this is possible, but there is no documentation as far as I can see. While I have at this point some experience with the docker or bare-linux approaches, Kubernetes is completely new to me, so I would appreciate any help I can get.

2

Internally, we use a NOMAD helm chart to run NOMAD on our k8s cluster. You can use this chart. However, the chart does not include mongodb or elasticsearch at the moment. So those would need to be manually added by you. The values.yaml associated with the chart, roughly follows the nomad.yaml config keys.

This might be a good starting point for you. Unfortunately, the whole thing is not documented or build for independent use at the moment. We want to have this at some point, but we are in the middle of too many things right now to spontaneously do it now.

3

Thanks a lot @mscheidgen,

I think this should get me started, hopefully out IT guys can help me a bit with the details. I’ll get back with more questions if I get stuck at some point.

4

Small question regarding the helm chart, I don’t need anything for the keycloak section as long as I have

oasis:
  is_oasis: true
  uses_central_user_management: true

right?

5

Yes, you should not need anything. There is only the client_secret, which is only necessary for the central NOMAD. With uses_central_user_management your Oasis will get all user information from the central NOMAD and not form keycloak directly.

6

Another bump on the road, I got a message from our IT department working on the helm chart. They managed to add the elastic and mongo somehow, but now the current issue seems to be that nomad tries to communicate with the elasticsearch via http, which is not supported with new elastic search versions and they haven’t figured out how to switch nomad to https yet. If using the older version (as a subchart) from https://github.com/elastic/helm-charts/tree/7.17/elasticsearch there is a problem of missing SecurityContext so the deployment is not working.

Any ideas?

7

Elasticsearch takes their breaking changes on major version updates more seriously than other software. We do not support version 8 and NOMAD probably won’t work with 8.

We are currently working on a helm chart for our infrastructure components (i.e. mongo and elastic) for our new cluster. I can share this, once it is ready.

8

I actually have a working configuration now (but most likely hacky as hell), CCing @fekad with whom I have discussed briefly at FAIRmat meeting. Ádam, my current config is at GitHub - ondracka/nomad-helm It includes everything except for the NORTH. Seems mostly stable and works but some stuff still keeps restarting (rabbitmq most often). I would be interested in any feedback (we can also move the discussion outside of the forum if this would be too technical).