Minio doesn't obey --s3-chunk-size?


#1

I noticed that Minio was writing multipart uploads as 5MB parts even though I provided “-s3-chunk-size 100M”. Does anyone else see this behaviour?


#2

That is completely under the control of the client so I wouldn’t have thought it would be minio.

How do you know the chunks are 5MB?


#3

Minio Server stores a folder with the tmp files in for for multi-part uploads (/.minio.sys/multipart/<hash>/<parts>). All the files in those folders are 5MB regardless of what setting I provide via --s3-chunk-size.

I suppose its possible the files are being transfered as 100M chunks and then stored on the Minio server-side as 5MB parts, but seems unlikely.


#4

I did a quick test using --s3-chunk-size 10M -vv --dump headers - you can see the PUT requests and these were correct for using a chunk size of 10M.

Can you have a go with minio and see what you get?

Which version of rclone are you using? I tested with the latest beta but v1.43.1 should give identical results.


#5

Thanks for trying this out.

I just tested this myself with the version I was running before (1.42) and it does indeed work fine. I can see the chunks being saved as 10MB or 50MB parts (based on what I set). It also works as intended on 1.43.

I’m really not sure what happened before then, perhaps I was looking at a tmp multipart folder for another upload where I hadn’t used the chunksize flag. But I recall deleting all tmp folders and trying multiple times before coming here.

If I manage to reproduce it I’ll post back, otherwise I’ll just assume it was user error, thanks for the help.

p.s.
noticed in 1.43 that I no longer get the 405 error for S3/Minio paths that start with a slash. Awesome!


#6

No worries. It is about time I had another play with minio - I haven’t for a while and it has come on a long way since then!

:slight_smile: