Adding MD5SUM on S3

[ncw (Nick Craig-Wood)]

@Val_Schmidt

• Is there a way to use rclone to calculate the hashes and populate the metadata for files already in the s3 bucket so that a proper rclone check can be performed without error?

You could probably do it with use of the hasher backend to read the checksums then using server side copies to add the hashes... Maybe!

The S3 protocol allows a server side copy to adjust the metadata so in theory it is possible.

@Val_Schmidt

Is there a way to force rclone to calculate them, even for large files when uploading?

Rclone will do its best to calculate them but if the source backend can't give it an MD5 then it can't. For example the local backend can produce any kind of hash, whereas the crypt backend can't produce any hashes.

It might be possible to wrap the source in the hasher backend and use its facilities to generate all the MD5sums before the transfer.

@Val_Schmidt

What is the file size limit above which rclone will not calculate the hash automatically.
Thanks.

All files below --s3-upload-cutoff should have an MD5SUM generated by S3 itself. So you could just set --s3-upload-cutoff 5G for all files below 5G to have MD5SUMs. Note that 5G is the maximum size possible here. This will make the uploads slower but may be what you want.