S3 upload - MD5 Check Sum

Help and Support
1

What is the problem you are having with rclone?

Hello everyone !

I have a question, if you could spare some to time check please. After upgrading o the latest beta which fixed some issues with s3 backend, I can no longer see the key "x-amz-meta-md5chksum" filled in my recent uploads. As I could check, previous uploads using v1.50.2 have this information. Is this something I should worry about, concerning the integrity of the uploaded files?

Reference:

What is your rclone version (output from rclone version)

v1.50.2-095

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows Server 2012 R2 STD x64

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

2020/01/06 17:04:31 DEBUG : rclone: Version "v1.50.2-095-g5363afa7-fix-s3-manager-beta" finishing with parameters ["rclone" "sync" "H:\Backups" "mytest:aws-jannarelli\Backups2" "--checkers" "32" "--retries" "5" "--transfers" "10" "--s3-upload-concurrency" "10" "--contimeout=10m" "-vv" "--log-file" "C:\scripts\AWS\Logs\Log-BSFS.txt"]

2

Update:

Just did some testing with v1.50.2, which showed the same results, no "x-amz-meta-md5chksum" on the tag, so I might have correlated the behavior to the wrong cause. It's very odd, since files from 28/12/2019 on S3 (uploaded with rclone) have it.

The only other change recently I've made was to enable bucket versioning, maybe this was the cause? But again, I am just trying to know if this is an issue, or if I should not be worried at all. :slight_smile:

Thanks

3

Files of sizes below --s3-upload-cutoff (200M) by default won't have x-amz-meta-md5chksum however they should have an Etag which is a valid md5sum which is why rclone doesn't bother computing the metadata for those files.

Is that what you see?

4

Hello Nick !

The Etag is present in all uploads above, as far as I could check. What I can see is that prior to 01/02/2020, all uploads (from 200Mb up to 700Gb) have x-amz-meta-md5chksum, the latest do not have it.

Again, very odd behavior. I'm not sure if this is rclone version related or bucket versioning related.

Thanks for your help

5

Hmm I think the beta is missing the header by the look of it.

Presumably your recent files were uploaded with the beta?

I see the problem - this was introduced by this commit (which hasn't been in a release yet)

Try this fix

https://beta.rclone.org/v1.50.2-114-gb6e86b2c-beta/ (uploaded in 15-30 mins)

6

Yes, that's correct, with the version 1.50.2-095

Thanks for your support Nick, but I couldn't find the Windows version in the repository. Please let me know if it will be available and I'll test as soon as I can.

Regards,

7

The windows builder has broken...

I made a build by hand - can you try this one?

https://pub.rclone.org/rclone-v1.50.2-115-g706da80d-beta.zip

1 Like
8

That's great @ncw, thank you for building this one.

I've already set up in all my sources, it will run in this evening and I'll let you know about the results ASAP.

Regards,

1 Like
9

@ncw good news again !

I can see that the headers are back in the log and in AWS Metadata aswell ! This was a multipart upload, which means that it work correctly for all kinds of load. See the evidences below, and I appreciate your help once again.

Regards,

image

image

10

Thanks for testing :slight_smile:

This will be released in 1.51

I need to fix the windows builders!

1 Like
11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.