Upload of large Files fails with a Gateway timeout

Hi, we host our own NOMAD OASIS instance (via docker) and try to get started with it.
A first attempt to upload a large archive (6GB) failed with

Unexpected Error: Gateway Timeout (504)...

Is this a known issue that might be fixed by adjusting some server parameters or does it require further investigation. If needed I can try to provide further details.

Thank you for your post.

It is very likely that this can be solved, but slight modifications to some server parameters. The NOMAD Oasis runs an nginx reverse proxy as gateway to the different services.

To get a general idea how long an upload is supposed to take: nomad is writing the file down while receiving it. An upload request should only take as long as the network bandwidth between client and nomad and the write-speed of the used filesystem suggests.

  1. If you run another reverse proxy in front of the NOMAD Oasis, try to figure out which proxy is causing the timeout, the NOMAD Oasis nginx, or your reverse proxy.
  2. The NOMAD nginx is configured by the configs/nginx.conf that you probably have downloaded from the NOMAD documentation (here)[Operating an OASIS - NOMAD Documentation]. It contains a rule for the upload API endpoint. This is were the timeout most likely is originating from. It should look something like this:
    location ~ /api/v1/uploads(/?$|.*/raw|.*/bundle?$)  {
        client_max_body_size 35g;
        proxy_request_buffering off;
        proxy_pass http://app:8000;
    }

There are three different timeout parameters:

  • proxy_connect_timeout – This is probably not it.
  • proxy_read_timeout – How long it takes the NOMAD app to respond after the file was send. It should not take NOMAD long to respond as the file is already saved while it is send piece by piece.
  • proxy_send_timeout – How long it takes between chunks of data send by the client. This should also not matter, if the client is sending continuously.

Please experiment a little with these timeouts.

It might be worthwhile experimenting with different upload options. How was the upload done that created the timeout?

  • The UI should stream the file and not cause this problem.
  • curl-X POST 'http://your-oasis/v1/api/v1/uploads?token=...' -T <local_file> should also stream an not cause the problem.
  • curl -X PUT ... -f file=@<local_file> uses a request with form-data and this might send the whole file at once and might be problematic.

Please let us know, if you can solve or not solve the problem with this information. As we are very much interested in the cause for these kind of problems.

Thanks for the quick reply, I’ll take a look into these options and report back if we manage to fix the issue.
We are not using an additional reverse proxy next to the one that came with the docker deployment and The error was reported by one of our users, who tried to upload the file via the UI.
This however also raises the question on how to update our instance, If I understand the documentation correctly the default is to just update the docker image. But how wil required l changes to the files in the original zip archive (docker-compose.yml, nginx.conf, …) be communicated if they should become necessary?

Yes, mostly it would just mean to use a new image tag. There is still an upcoming major release. As part of it, we are revising the availability of a Changelog that would inform about necessary config changes.