MD5 hash differ when using aws s3 with kms key

What is the problem you are having with rclone?

Trying to either copy or use the mount command to a S3 bucket that is set to use a KMS key for default encryption. I'm not having any issues with using the default encryption key in AWS, only when specifying a KMS key. The source server is an EC2 server running in AWS, the user account has full (admin) access to the account.

What is your rclone version (output from rclone version)

1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows server 2019, 64 bit

Which cloud storage system are you using? (eg Google Drive)

AWS S3 bucket

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Copy:
rclone copy test3.txt ab-test:ab-test2/ --vfs-cache-mode full

Mount:
rclone mount ab-test:ab-test2/ s: --vfs-cache-mode full

The rclone config contents with secrets removed.

[ab-test]
type = s3
provider = AWS-US-GOV
region = us-gov-east-1
env_auth = false
access_key_id = ...
secret_access_key = ...
acl = private
server_side_encryption = aws:kms
sse_kms_key_id = arn:aws-us-gov:kms:us-gov-east-1:...:key/...

A log from the command with the -vv flag

Copy:

2020/08/04 19:01:21 ERROR : test3.txt: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "f58b1b8be2d472d888ccaf254c5c77fb"
2020/08/04 19:01:21 INFO  : test3.txt: Removing failed copy
2020/08/04 19:01:21 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "f58b1b8be2d472d888ccaf254c5c77fb"
2020/08/04 19:01:21 ERROR : test3.txt: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "f88fe747628a8b6c86e60a79611fe300"
2020/08/04 19:01:21 INFO  : test3.txt: Removing failed copy
2020/08/04 19:01:21 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "f88fe747628a8b6c86e60a79611fe300"
2020/08/04 19:01:21 ERROR : test3.txt: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "bc53f2997decd1f012a296ba509d63a3"
2020/08/04 19:01:21 INFO  : test3.txt: Removing failed copy
2020/08/04 19:01:21 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "bc53f2997decd1f012a296ba509d63a3"
2020/08/04 19:01:21 INFO  :
Transferred:            12 / 12 Bytes, 100%, 55 Bytes/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         0.2s

2020/08/04 19:01:21 Failed to copy: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "bc53f2997decd1f012a296ba509d63a3"

Mount:

2020/08/04 16:33:55 INFO  : S3 bucket ab-test2: poll-interval is not supported by this remote
2020/08/04 16:33:55 INFO  : Cleaned the cache: objects 2 (was 0), total size 4 (was 0)
The service rclone has been started.
2020/08/04 16:34:06 ERROR : test3.txt: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "2a126c1448cb1aa9a24513cda5e1cc3b"
2020/08/04 16:34:06 INFO  : test3.txt: Removing failed copy
2020/08/04 16:34:06 ERROR : test3.txt(0xc000092f00): failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "2a126c1448cb1aa9a24513cda5e1cc3b"
2020/08/04 16:34:06 ERROR : test3.txt(0xc000092f00): RWFileHandle.Flush error: failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "2a126c1448cb1aa9a24513cda5e1cc3b"
2020/08/04 16:34:06 ERROR : IO error: failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "2a126c1448cb1aa9a24513cda5e1cc3b"
2020/08/04 16:34:06 ERROR : test3.txt: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "43874dabdc2d904385cac5715f2f5f97"
2020/08/04 16:34:06 INFO  : test3.txt: Removing failed copy
2020/08/04 16:34:06 ERROR : test3.txt(0xc000092f00): failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "43874dabdc2d904385cac5715f2f5f97"
2020/08/04 16:34:06 ERROR : test3.txt(0xc000092f00): RWFileHandle.Release error: failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "43874dabdc2d904385cac5715f2f5f97"
2020/08/04 16:34:06 ERROR : IO error: failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs "43874dabdc2d904385cac5715f2f5f97"
2020/08/04 16:34:55 INFO  : Cleaned the cache: objects 2 (was 2), total size 4 (was 4)

``

hello and welcome to the forum.

not sure what the problem is but

--vfs-cache-mode is only for mounts, so you should remove it from this command
rclone copy test3.txt ab-test:ab-test2/ --vfs-cache-mode full

Hi, thanks.

It seems the issue I'm having is related to a bug that was reported in 2017 but still not resolved. If I add --ignore-checksum the problem goes away. I would rather not enable that though...

I just read through that issue and I think the approach to take is the one in this comment https://github.com/rclone/rclone/issues/1824#issuecomment-417932743

Do you fancy helping out with this? I don't think it would be too tricky to implement

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.