Corrupted on transfer: md5 hash differ (AWS S3 to GCS)

Help and Support
1

What is the problem you are having with rclone?

corrupted on transfer: md5 hash differ (AWS S3)

Run the command 'rclone version' and share the full output of the command.

1.58.1

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

// running as a rclone webserver as kubernetes deployement in GKE cluster:

     rclone
      rcd
      --rc-addr
      :5572
      --s3-upload-cutoff
      0

// then running from a cronjob in the same cluster with the following command:
rclone rc --rc-addr=10.4.1.39:5572 --rc-user=admin --rc-pass=passwordher sync/copy srcFs=featurespace:aws_bucket/ dstFs=gcs:gcs_bucket/ -vv

The rclone config contents with secrets removed.

// gcs config in the rclone.conf file
    [gcs]
    type = google cloud storage
    project_number = xxxxxxxx
    object_acl = bucketOwnerFullControl
    bucket_acl = private
    bucket_policy_only = true
    location = europe-west2
    storage_class = STANDARD

// on webserver pod running as environment vars for the remote
      RCLONE_RC_ENABLE_METRICS:                      true
      RCLONE_CONFIG_FEATURESPACE_ACCESS_KEY_ID:      <set to the key 'access_key_id' in secret 'stg-rclone-webserver-featurespace-oci-creds'>  Optional: false
      RCLONE_CONFIG_FEATURESPACE_ENDPOINT:
      RCLONE_CONFIG_FEATURESPACE_SECRET_ACCESS_KEY:  <set to the key 'secret_access_key' in secret 'stg-rclone-webserver-featurespace-oci-creds'>  Optional: false
      RCLONE_CONFIG_FEATURESPACE_TYPE:               s3
      RCLONE_CONFIG_FEATURESPACE_REGION:             eu-west-2
      RCLONE_RC_USER:                                <set to the key 'rclone_user' in secret 'stg-rclone-webserver-creds'>      Optional: false
      RCLONE_RC_PASS:                                <set to the key 'rclone_password' in secret 'stg-rclone-webserver-creds'>  Optional: false
      RCLONE_WEB_GUI:                                true
      RCLONE_WEB_GUI_NO_OPEN_BROWSER:                true

A log from the command with the -vv flag

2022/06/01 15:16:21 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "rc" "--rc-addr=localhost:5572" "--rc-user=admin" "--rc-pass=RcloneS123" "sync/copy" "srcFs=featurespace:aws_bucket/" "dstFs=gcs:gcs_bucket/" "-vv"]
2022/06/01 15:16:21 DEBUG : 6 go routines active
2022/06/01 15:16:21 Failed to rc: Failed to read rc response: 500 Internal Server Error: {
	"error": "corrupted on transfer: md5 hash differ \"c33df3277b3c9aeb77b92fa44046f2c5\" vs \"af1e64d938386cce6e70e5f806085493\"",
	"input": {
		"dstFs": "gcs:gcs_bucket/",
		"srcFs": "featurespace:aws_bucket/"
	},
	"path": "sync/copy",
	"status": 500
}

I'm running the rclone as webserver in a gke cluster and running a cronjob within the same cluster, to copy data from a AWS S3 bucket to GCS bucket, getting the error even after setting the --s3-upload-cutoff 0

2

Have the files got gzip Transfer encoding?

3

these are just .xls and .xlsx files

4

Do all files have the problem? Or is it just .xls and .xlsx files?

Could something be modifying them on the source while rclone is transferring them, or the destination?

Do you know which checksum is correct?

5

at the moment I have only .xls and .xlsx files.

on the source AWS bucket there is KMS key set and on the destination just normal bucket.

af1e64d938386cce6e70e5f806085493 is correct.

6

Check out KMS in the docs

Key Management System (KMS)

If you are using server-side encryption with KMS then you must make sure rclone is configured with server_side_encryption = aws:kms otherwise you will find you can't transfer small objects - these will create checksum errors.

Does that help?

7

Setting the env variable RCLONE_S3_SERVER_SIDE_ENCRYPTION="aws:kms" did the trick.

Many thanks for your help Nick.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.