How to Disable Multipart Uploading

Help and Support
1

I'd like to disable multipart uploading for a S3 remote, but I did not find such a flag. It is also possible that my plan does not make sense.

Context:

I want to sync a set of files to a S3 Deep Glacier crypt remote. I'm aware that verification might not work if both crypt and multipart-uploading are in effect.

My plan is:

  • rclone copy my files to the S3 crypt remote, while disabling multipart uploading
  • Check which files failed to upload, rclone copy these files to a local crypt remote
  • rclone copy from the local crypt remote to the S3 remote (without crypt)

Other plans that I have considered:

rclone cryptcheck ==> This would be too slow for me, and it might not work for deep glacier.

Copy all files to local crypt first ==> I think it is too slow

Only directly upload small files ==> I suppose a file smaller than 3GB won't get bigger than 5GB after the encryption, but I'm not sure about the optimal cutoff.

2

hi,
i use aws s3 deep glacier, have many dozens of TiB stored there.

sorry, not possible to disable multi-part uploads using s3.
once a file is over 5GiB, have to use multipart.
i could be wrong but the best is --s3-upload-cutoff=5G

for non-technical and the cost of api calls, i am not able rclone crypt with aws s3 deep glacier.
i only upload local files that are already crypted; veeam backup files and .7z
so i can use rclone check

3

Hi, thanks for you reply!

To clarify, I meant "disable multipart uploading and show an error if the encrypted file is too big".

4

what error?
there would not be an error, as rclone will upload any file you asked it to.

5

The error should be "the file is too big to upload without multipart uploading"

6

as far as i know, no such error.

if you want to limit files to sizes below 5GiB.
--s3-upload-cutoff=5G --max-size=5G

7

This is one of the options I listed above.
The issue is, if a file is just a little bit smaller than 5GB, the encrypted file may be larger than 5G, and thus trigger multipart uploading.

I could probably use 4GB as the threshold, but I'm here asking whether there are better solutions.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.