Hi.
I’m trying to upload a .zip file containing ~50k folders, each folder is 1 simulation consisting of 7 files, to our Oasis. And each folder contains 1 mainfile, so there should be ~50k entries. Zip file size unpacked is ~4 GB, so smaller than the 32 GB limit.
If I use the Nomad GUI to upload, it gets to “Matching” phase and then gets stuck there (I’ve waited 15 mins, no progress). If I use terminal via “curl …” to upload, it simply says “413 Request Entity Too Large”.
Is there a way to fix this? If not, I guess I would have to split the 50k folders into smaller batches, zipping each batch, upload these zips, and then make a dataset out of all 50k simulations (because we need one DOI for all 50k sims)? But how small/how many folders per .zip file will work?
Best
Fabian
Hi @fabian_li!
Getting large uploads through the network can sometimes be tricky. The fastest way to solve this in your case might be to simply break down the upload into smaller zip files. Notice that you can upload several zip files to a single upload: the contents will be automatically unzipped into this single upload. This way all your data can still live in the same upload.
As to fixing the underlying problem: From our experience, CURL has been the most reliable way of making bigger uploads. But in your case it seems that some part of your Oasis network is limiting the POST body size, resulting in the 413 HTTP message. I can try if I can reproduce this error on my end to see if the problem is in our Oasis image. If this is not the case, the problem might be on some other layer of your Oasis network setup .