Multipart upload corrupted: Etag differ

[jaimedelano]

What is the problem you are having with rclone?

I'm working on getting full S3 support in the Pydio Cells S3 gateway.
The cloud storage provider quotaless.cloud uses the Pydio Cells patch which I have developed, and it has been working great overall, except when multiparts uploads are done through the crypt backend.

Once the upload is complete, Rclone sends a HEAD request to compare the e-tags. But it compares the multipart upload etag with the etag from the server, which isn't quite right. It should be comparing the MD5 hash of the source file with the MD5 etag of the destination file.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2

• os/version: rocky 9.4 (64 bit)
• os/kernel: 5.14.0-427.35.1.el9_4.x86_64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.23.3
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3 / MinIO / Quotaless

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P /data/tmp/file-1.iso secret:/

The rclone config contents with secrets removed.

[quotaless]
type = s3
provider = Minio
env_auth = false
access_key_id = XXYYZZ
secret_access_key = gatewaysecret
endpoint = https://io.quotaless.cloud
acl = bucket-owner-full-control
list_chunk = 10000000
upload_cutoff = 100M
chunk_size = 10M
encoding = Slash,InvalidUtf8,Dot,Percent,Ctl

[secret]
type = crypt
remote = quotaless:/data/personal-files/crypt
password = XXX
password2 = YYY

A log from the command with the -vv flag

2024/12/01 09:56:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/01 09:56:19 DEBUG : HTTP REQUEST (req 0xc0008bcc80)
2024/12/01 09:56:19 DEBUG : POST /data/personal-files/crypt/uisodene7dh0qvrgrp605l6pus?uploadId=64e3adb3-328e-41c4-aec5-55653616d630 HTTP/1.1
Host: io.quotaless.cloud
User-Agent: rclone/v1.68.2
Content-Length: 5030
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: de0353a4-36aa-4cb2-8eba-e6c309f8ce61
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/xml
X-Amz-Content-Sha256: c8d37ab8724c6c73f357e3c434f64341c5ab00614280f07154681f3df3db6c1b
X-Amz-Date: 20241201T095619Z
2024/12/01 09:56:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/01 09:56:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/01 09:56:19 DEBUG : HTTP RESPONSE (req 0xc0008bcc80)
2024/12/01 09:56:19 DEBUG : HTTP/2.0 200 OK
Content-Length: 384
Accept-Ranges: bytes
Cache-Control: no-cache
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sun, 01 Dec 2024 09:56:19 GMT
Etag: "ec597a6afb0de1686d6776955e52ddd5-52"
Server: openresty/1.25.3.2
Vary: Origin
X-Amz-Request-Id: 180D050B9D4D711A
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Location>https://127.0.0.1:11800/data/personal-files/crypt/uisodene7dh0qvrgrp605l6pus</Location>
<Bucket>data</Bucket>
<Key>personal-files/crypt/uisodene7dh0qvrgrp605l6pus</Key>
<ETag>&#34;ec597a6afb0de1686d6776955e52ddd5-52&#34;</ETag>
</CompleteMultipartUploadResult>
2024/12/01 09:56:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/01 09:56:19 DEBUG : uisodene7dh0qvrgrp605l6pus: multipart upload "64e3adb3-328e-41c4-aec5-55653616d630" finished
2024/12/01 09:56:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/01 09:56:19 DEBUG : HTTP REQUEST (req 0xc0008bd680)
2024/12/01 09:56:19 DEBUG : HEAD /data/personal-files/crypt/uisodene7dh0qvrgrp605l6pus HTTP/1.1
Host: io.quotaless.cloud
User-Agent: rclone/v1.68.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c710bef6-6eee-48a0-af33-132456ccc2b2
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241201T095619Z
2024/12/01 09:56:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/01 09:56:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/01 09:56:19 DEBUG : HTTP RESPONSE (req 0xc0008bd680)
2024/12/01 09:56:19 DEBUG : HTTP/2.0 200 OK
Content-Length: 537002016
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/octet-stream
Date: Sun, 01 Dec 2024 09:56:19 GMT
Etag: "3b66981af732b1ee214604352d128a85"
Last-Modified: Fri, 15 Nov 2024 18:07:00 GMT
Server: openresty/1.25.3.2
Vary: Origin
X-Amz-Meta-Btime: 1733046979.307783801
X-Amz-Meta-Md5chksum: O2aYGvcyse4hRgQ1LRKKhQ==
X-Amz-Meta-Mtime: 1731694020.54486065
X-Amz-Meta-Pydio-Node-Uuid: 2b2dab5f-573f-4f0d-83c9-cfc6144505a3
X-Amz-Request-Id: 180D050BBFCAFEE5
X-Xss-Protection: 1; mode=block
2024/12/01 09:56:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/01 09:56:19 ERROR : file-1.iso: Failed to copy: multipart upload corrupted: Etag differ: expecting ec597a6afb0de1686d6776955e52ddd5-52 but got 3b66981af732b1ee214604352d128a85
2024/12/01 09:56:19 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload corrupted: Etag differ: expecting ec597a6afb0de1686d6776955e52ddd5-52 but got 3b66981af732b1ee214604352d128a85
2024/12/01 09:56:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/01 09:56:19 DEBUG : HTTP REQUEST (req 0xc000574140)
2024/12/01 09:56:19 DEBUG : HEAD /data/personal-files/crypt/uisodene7dh0qvrgrp605l6pus HTTP/1.1
Host: io.quotaless.cloud
User-Agent: rclone/v1.68.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 632cda9b-5c5b-4076-8256-8345bb7b440d
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241201T095619Z
2024/12/01 09:56:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/01 09:56:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/01 09:56:19 DEBUG : HTTP RESPONSE (req 0xc000574140)
2024/12/01 09:56:19 DEBUG : HTTP/2.0 200 OK
Content-Length: 537002016
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/octet-stream
Date: Sun, 01 Dec 2024 09:56:19 GMT
Etag: "3b66981af732b1ee214604352d128a85"
Last-Modified: Fri, 15 Nov 2024 18:07:00 GMT
Server: openresty/1.25.3.2
Vary: Origin
X-Amz-Meta-Btime: 1733046979.307783801
X-Amz-Meta-Md5chksum: O2aYGvcyse4hRgQ1LRKKhQ==
X-Amz-Meta-Mtime: 1731694020.54486065
X-Amz-Meta-Pydio-Node-Uuid: 2b2dab5f-573f-4f0d-83c9-cfc6144505a3
X-Amz-Request-Id: 180D050BC3AAA133
X-Xss-Protection: 1; mode=block
2024/12/01 09:56:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/01 09:56:19 DEBUG : file-1.iso: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/12/01 09:56:19 DEBUG : file-1.iso: Unchanged skipping
2024/12/01 09:56:19 ERROR : Attempt 2/3 succeeded
Transferred:      512.125 MiB / 512.125 MiB, 100%, 8.549 MiB/s, ETA 0s
Elapsed time:       1m2.4s

[asdffdsa (jojothehumanmonkey)]

@jaimedelano

I'm working on getting full S3 support

where do you think the bug is, with quotaless.cloud , pydio, or your patch?

i have been uploading rclone crypt remotes to multiple s3 storage providers, for many years, never had that issue.
and just now, i did a quick test using v.168.2 and seems to work fine
maybe i mid-understood your setup?

DEBUG : file.ext: Computing md5 hash of encrypted source
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: open chunk writer: started multipart upload: H2TYapYoSZnqvmBVRDI-IoFG8US-fPFTMORvpF2W_qLupIQBGO9zsvpc7VbJQsL4mPkB2zzxrJ_rZBo9vlDfzZ5XuEEI0OJsS4vj8yk7CKlFnTwgA6Hfecm6CnDKAX7z
DEBUG : file.ext: multipart upload: starting chunk 0 size 5Mi offset 0/15.949Mi
DEBUG : file.ext: multipart upload: starting chunk 1 size 5Mi offset 5Mi/15.949Mi
DEBUG : file.ext: multipart upload: starting chunk 2 size 5Mi offset 10Mi/15.949Mi
DEBUG : file.ext: multipart upload: starting chunk 3 size 972.031Ki offset 15Mi/15.949Mi
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: multipart upload wrote chunk 4 with 995360 bytes and etag "b2d35c0b5cec750aa66e4267d229af66"
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: multipart upload wrote chunk 1 with 5242880 bytes and etag "4a2058eda399b01fad0d5d8a8cacd24a"
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: multipart upload wrote chunk 2 with 5242880 bytes and etag "d96872b5b584134db9e31a539405716c"
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: multipart upload wrote chunk 3 with 5242880 bytes and etag "67ed071e5f5d2d996a94b19724b9a9f1"
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: multipart upload "H2TYapYoSZnqvmBVRDI-IoFG8US-fPFTMORvpF2W_qLupIQBGO9zsvpc7VbJQsL4mPkB2zzxrJ_rZBo9vlDfzZ5XuEEI0OJsS4vj8yk7CKlFnTwgA6Hfecm6CnDKAX7z" finished
DEBUG : a950r94tabgfr72sm03lo4c748kmv4udj3s9mp123qi4rkd7t1t0: Multipart upload Etag: 4acda9574cf6b0c7928f23336612b9a0-4 OK
DEBUG : file.ext: md5 = 8a52e41c7b71c18f4bfba9614035e749 OK
INFO  : file.ext: Copied (new)

---

@jaimedelano

Attempt 2/3 succeeded

and in your case, the upload does complete, correct?

---

fwiw, for testing, post a complete debug log, not snippets.
pick a small file, like 15MiB and use --s3-upload-cutoff=5M

[jaimedelano]

From what I've read in the AWS specs, it seems like the issue might not be with Rclone, but in how it is being implemented in Quotaless. I'm not totally sure, though.

Here's the link to the page on the Amazon S3 API that explains the response headers: Common Response Headers - Amazon Simple Storage Service

The issue is that the upload e-tag is being checked in the wrong context. A HEAD request to the object shouldn't be comparing the multipart upload e-tag; it should be looking at the MD5 hash of the source file instead.

Despite this, the upload is successful on a second attempt, because this time the HEAD request checks the MD5 hash of the file.

[asdffdsa (jojothehumanmonkey)]

@jaimedelano

From what I've read in the AWS specs, it seems like the issue might not be with Rclone

imho, this is not a rclone issue. i have used crypt with aws, idrive and wasabi.

as for s3+etag, i was the forum member, that raised that issue and help get it implemented.
https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095

[ncw (Nick Craig-Wood)]

You need to set this quirk probably

Amazon S3

I suggest you start from provider = Other and set supported features. Search the S3 backend source for quirks and you'll see what I mean.

[jaimedelano]

It's probably fine to leave this as it is, as Rclone reads the X-Amz-Meta-Md5chksum header and checks it correctly. It seems I just made some wrong assumptions.

[asdffdsa (jojothehumanmonkey)]

@jaimedelano

It's probably fine to leave this as it is, as Rclone reads the X-Amz-Meta-Md5chksum header and checks it correctly.
It seems I just made some wrong assumptions.

imho, that is not a check.
the only way to verify file transfer is ETag.

i pointed that out a few times and i believe that ncw agrees with me.

1. It isn't a good check you are right.
2. That is correct