I noticed that Minio was writing multipart uploads as 5MB parts even though I provided “-s3-chunk-size 100M”. Does anyone else see this behaviour?
That is completely under the control of the client so I wouldn’t have thought it would be minio.
How do you know the chunks are 5MB?
Minio Server stores a folder with the tmp files in for for multi-part uploads (/.minio.sys/multipart/<hash>/<parts>). All the files in those folders are 5MB regardless of what setting I provide via --s3-chunk-size.
I suppose its possible the files are being transfered as 100M chunks and then stored on the Minio server-side as 5MB parts, but seems unlikely.
I did a quick test using --s3-chunk-size 10M -vv --dump headers - you can see the PUT requests and these were correct for using a chunk size of 10M.
Can you have a go with minio and see what you get?
Which version of rclone are you using? I tested with the latest beta but v1.43.1 should give identical results.
Thanks for trying this out.
I just tested this myself with the version I was running before (1.42) and it does indeed work fine. I can see the chunks being saved as 10MB or 50MB parts (based on what I set). It also works as intended on 1.43.
I’m really not sure what happened before then, perhaps I was looking at a tmp multipart folder for another upload where I hadn’t used the chunksize flag. But I recall deleting all tmp folders and trying multiple times before coming here.
If I manage to reproduce it I’ll post back, otherwise I’ll just assume it was user error, thanks for the help.
p.s.
noticed in 1.43 that I no longer get the 405 error for S3/Minio paths that start with a slash. Awesome!
No worries. It is about time I had another play with minio - I haven’t for a while and it has come on a long way since then!
