Does rclone (s3) support If-None-Match and If-Match Upload Headers?

[seanturner]

What is the problem you are having with rclone?

It appears to me that the If-None-Match and If-Match upload headers aren’t supported by rclone with S3.

We’re leveraging these headers with an S3 Bucket Policy to prevent overwrites on certain S3 Prefixes; however, rclone prints the following error when using copy with the –header-upload ‘If-None-Match: *

2025/11/07 15:26:02 ERROR : protected-new-new.txt: Don't know how to set key "If-None-Match" on upload

This seems to correspond to this code rclone/backend/s3/s3.go at fb895f69a1a3aeed66d1071c4bb34ac69208e506 · rclone/rclone · GitHub

But as far as I can tell this isn’t just an AWS header, it’s more of a ubiquitous request header around conditional requests.

Regardless, the expected behavior is that rclone handles the headers properly rather than log an error.

How to prevent object overwrites with conditional writes - Amazon Simple...

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.71.2
- os/version: debian 12.11 (64 bit)
- os/kernel: 6.15.11-orbstack-00542-g4f455d264886 (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.25.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

``

```
rclone copy \
    --header-upload 'If-None-Match: *' \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah/
```

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

```
rclone config redacted
2025/11/07 15:29:53 NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
; empty config
### Double check the config for sensitive info before posting publicly
```

A log from the command that you were trying to run with the -vv flag

```
rclone copy \
    --header-upload 'If-None-Match: *' \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blahhh/ \
    -vv
2025/11/07 15:39:14 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copy" "--header-upload" "If-None-Match: *" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blahhh/" "-vv"]
2025/11/07 15:39:14 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 15:39:14 NOTICE: Config file "/home/kobold/.config/rclone/rclone.conf" not found - using defaults
2025/11/07 15:39:14 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/app/api-client-generation/data/new/hyspex/vnir"
2025/11/07 15:39:14 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blahhh/"
2025/11/07 15:39:14 DEBUG : :s3: detected overridden config - adding "{QXFZx}" suffix to name
2025/11/07 15:39:14 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blahhh/" to be canonical ":s3{QXFZx}:my-bucket/flight/5/blahhh"
2025/11/07 15:39:15 DEBUG : protected-new-new.txt: Need to transfer - File not found at Destination
2025/11/07 15:39:15 ERROR : protected-new-new.txt: Don't know how to set key "If-None-Match" on upload
2025/11/07 15:39:15 DEBUG : protected-new-new.txt: md5 = 40b134ab8a3dee5dd9760a7805fd495c OK
2025/11/07 15:39:15 INFO  : protected-new-new.txt: Copied (new)
2025/11/07 15:39:15 INFO  :
Transferred:              4 B / 4 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2025/11/07 15:39:15 DEBUG : 5 go routines active
```

[seanturner]

Totally happy to submit a PR!

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

i do not have an answer to your question, but wanted to be sure that you know about
--dump flags such as --dump=headers,requests

[seanturner]

@asdffdsa

--dump=headers,requests

Cheers, I didn’t know about this!

It looks like those headers aren’t being included?

Summary with –dump

rclone copy \
    --header-upload 'If-None-Match: *' \
    --dump=headers,requests \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blahhhh/ \
    -vv
2025/11/07 15:50:49 NOTICE: Automatically setting -vv as --dump is enabled
2025/11/07 15:50:49 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copy" "--header-upload" "If-None-Match: *" "--dump=headers,requests" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blahhhh/" "-vv"]
2025/11/07 15:50:49 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 15:50:49 NOTICE: Config file "/home/kobold/.config/rclone/rclone.conf" not found - using defaults
2025/11/07 15:50:49 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/app/api-client-generation/data/new/hyspex/vnir"
2025/11/07 15:50:49 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blahhhh/"
2025/11/07 15:50:49 DEBUG : :s3: detected overridden config - adding "{QXFZx}" suffix to name
2025/11/07 15:50:49 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 15:50:49 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 15:50:49 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blahhhh/" to be canonical ":s3{QXFZx}:my-bucket/flight/5/blahhhh"
2025/11/07 15:50:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 15:50:49 DEBUG : HTTP REQUEST (req 0x40003b2a00)
2025/11/07 15:50:49 DEBUG : HEAD /flight/5/blahhhh/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.71.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 79711188-1e07-4558-b717-c1bcac77ddf9
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T155049Z
X-Amz-Security-Token: redacted

2025/11/07 15:50:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 15:50:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 15:50:49 DEBUG : HTTP RESPONSE (req 0x40003b2a00)
2025/11/07 15:50:49 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 07 Nov 2025 15:50:48 GMT
Server: AmazonS3
X-Amz-Id-2: also-redacted?
X-Amz-Request-Id: 0F51WWEVP6M1D466

2025/11/07 15:50:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 15:50:49 DEBUG : protected-new-new.txt: Need to transfer - File not found at Destination
2025/11/07 15:50:49 ERROR : protected-new-new.txt: Don't know how to set key "If-None-Match" on upload
2025/11/07 15:50:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 15:50:49 DEBUG : HTTP REQUEST (req 0x4000c66f00)
2025/11/07 15:50:49 DEBUG : PUT /flight/5/blahhhh/protected-new-new.txt?x-id=PutObject HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.71.2
Content-Length: 45
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ee47e856-7044-4c4b-8bfd-6f249f151603
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: QLE0q4o97l3Zdgp4Bf1JXA==
Content-Type: text/plain; charset=utf-8
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251107T155049Z
X-Amz-Decoded-Content-Length: 4
X-Amz-Meta-Mtime: 1762528397.452931297
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

4
333

0
x-amz-checksum-crc32:8UlftA==

2025/11/07 15:50:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 15:50:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 15:50:49 DEBUG : HTTP RESPONSE (req 0x4000c66f00)
2025/11/07 15:50:49 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 07 Nov 2025 15:50:50 GMT
Etag: "40b134ab8a3dee5dd9760a7805fd495c"
Server: AmazonS3
X-Amz-Checksum-Crc32: 8UlftA==
X-Amz-Checksum-Type: FULL_OBJECT
X-Amz-Id-2: also-redacted?
X-Amz-Request-Id: 0F5385W5KX8WC4CW
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: yfeaw_1odnw0vq2SzMXuyZ7l3UHKsWT9

2025/11/07 15:50:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 15:50:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 15:50:49 DEBUG : HTTP REQUEST (req 0x40002c8a00)
2025/11/07 15:50:49 DEBUG : HEAD /flight/5/blahhhh/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.71.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 49e18255-d828-4451-a511-98c84ddaaf05
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T155049Z
X-Amz-Security-Token: redacted

2025/11/07 15:50:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 15:50:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 15:50:49 DEBUG : HTTP RESPONSE (req 0x40002c8a00)
2025/11/07 15:50:49 DEBUG : HTTP/1.1 200 OK
Content-Length: 4
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Fri, 07 Nov 2025 15:50:50 GMT
Etag: "40b134ab8a3dee5dd9760a7805fd495c"
Last-Modified: Fri, 07 Nov 2025 15:50:50 GMT
Server: AmazonS3
X-Amz-Id-2: also-redacted?
X-Amz-Meta-Mtime: 1762528397.452931297
X-Amz-Request-Id: 0F512BESEGKYHFW3
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: yfeaw_1odnw0vq2SzMXuyZ7l3UHKsWT9

2025/11/07 15:50:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 15:50:49 DEBUG : protected-new-new.txt: md5 = 40b134ab8a3dee5dd9760a7805fd495c OK
2025/11/07 15:50:49 INFO  : protected-new-new.txt: Copied (new)
2025/11/07 15:50:49 INFO  :
Transferred:              4 B / 4 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.2s

2025/11/07 15:50:49 DEBUG : 6 go routines active

[asdffdsa (jojothehumanmonkey)]

@seanturner

It looks like those headers aren’t being included?

as a test, might try --header

[seanturner]

These ones do get handled properly in rclone but create an error in AWS’s backend

rclone copy \
    --header 'If-None-Match: *' \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah/

2025/11/07 16:07:34 NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
2025/11/07 16:07:35 ERROR : Attempt 1/3 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: 8WVE1SG32W5C8H08, HostID: 9ZivySPx7wrBf3T0h0lDAxR6bNq40HI9Wkk7iQu2cGoVMMHRjj4qXL8CKotQ8njClISDUR0zQCQ=, api error NotModified: Not Modified
2025/11/07 16:07:35 ERROR : Attempt 2/3 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: 8WVENRP2KQSK8SVG, HostID: OrrbghFoIEFrpqZWqwA5lsyAdH9ycz+y4D1aeQZBJVJvNud8uebAcNquuCV3S5uYq9FOwuZEMk8=, api error NotModified: Not Modified
2025/11/07 16:07:35 ERROR : Attempt 3/3 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: 8WVD7STXS4MQ6GH9, HostID: colXmzYsjQRZ2mrd3HWUP/tTT4uMgDp7r9qrV+RX+vHiEwAYQ4VrSdo2YKY4YLlPFudCR3BVVMc=, api error NotModified: Not Modified
2025/11/07 16:07:35 NOTICE: Failed to copy: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: 8WVD7STXS4MQ6GH9, HostID: colXmzYsjQRZ2mrd3HWUP/tTT4uMgDp7r9qrV+RX+vHiEwAYQ4VrSdo2YKY4YLlPFudCR3BVVMc=, api error NotModified: Not Modified

[asdffdsa (jojothehumanmonkey)]

@seanturner

These ones do get handled properly in rclone

yeah, i think i noticed a difference in the source code handling of --header versus –header-upload

---

@seanturner

StatusCode: 304

if the source file and dest file are the same, then that is the expected response, correct?

---

fwiw, to keep the log smaller, use --retries=1

[seanturner]

@asdffdsa

if the source file and dest file are the same, then that is the expected response, correct?

Sorry that was confusing. I knew from testing yesterday that –header automatically gives the HEAD error. Here’s debug output with the –dump included and a new prefix.

Summary

rclone copy \
    --header 'If-None-Match: *' \
    --retries=1 \
    --dump=headers,requests \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah1/
2025/11/07 16:28:30 NOTICE: Automatically setting -vv as --dump is enabled
2025/11/07 16:28:30 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copy" "--header" "If-None-Match: *" "--retries=1" "--dump=headers,requests" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blah1/"]
2025/11/07 16:28:30 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 16:28:30 NOTICE: Config file "/home/kobold/.config/rclone/rclone.conf" not found - using defaults
2025/11/07 16:28:30 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/app/api-client-generation/data/new/hyspex/vnir"
2025/11/07 16:28:30 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blah1/"
2025/11/07 16:28:30 DEBUG : :s3: detected overridden config - adding "{QXFZx}" suffix to name
2025/11/07 16:28:30 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 16:28:30 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 16:28:30 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blah1/" to be canonical ":s3{QXFZx}:my-bucket/flight/5/blah1"
2025/11/07 16:28:30 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 16:28:30 DEBUG : HTTP REQUEST (req 0x40007b6640)
2025/11/07 16:28:30 DEBUG : HEAD /flight/5/blah1/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.71.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 2fb56d6a-723e-4075-8b6e-da35288b7033
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
If-None-Match: *
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T162830Z
X-Amz-Security-Token: redacted

2025/11/07 16:28:30 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 16:28:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 16:28:31 DEBUG : HTTP RESPONSE (req 0x40007b6640)
2025/11/07 16:28:31 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 07 Nov 2025 16:28:30 GMT
Server: AmazonS3
X-Amz-Id-2: redacted
X-Amz-Request-Id: FPY060G9P9QJ0JDT

2025/11/07 16:28:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 16:28:31 DEBUG : protected-new-new.txt: Need to transfer - File not found at Destination
2025/11/07 16:28:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 16:28:31 DEBUG : HTTP REQUEST (req 0x40007b6f00)
2025/11/07 16:28:31 DEBUG : PUT /flight/5/blah1/protected-new-new.txt?x-id=PutObject HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.71.2
Content-Length: 45
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 684c53ed-f1b4-4eef-a22f-a28b3b1235f7
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: QLE0q4o97l3Zdgp4Bf1JXA==
Content-Type: text/plain; charset=utf-8
If-None-Match: *
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251107T162831Z
X-Amz-Decoded-Content-Length: 4
X-Amz-Meta-Mtime: 1762528397.452931297
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

4
333

0
x-amz-checksum-crc32:8UlftA==

2025/11/07 16:28:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 16:28:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 16:28:31 DEBUG : HTTP RESPONSE (req 0x40007b6f00)
2025/11/07 16:28:31 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 07 Nov 2025 16:28:32 GMT
Etag: "40b134ab8a3dee5dd9760a7805fd495c"
Server: AmazonS3
X-Amz-Checksum-Crc32: 8UlftA==
X-Amz-Checksum-Type: FULL_OBJECT
X-Amz-Id-2: redacted
X-Amz-Request-Id: FPY0S911Q30BYF7X
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: 28V.gwNPBnTJ.maYdyWZAxFA5WjHuL0n

2025/11/07 16:28:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 16:28:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 16:28:31 DEBUG : HTTP REQUEST (req 0x4000000a00)
2025/11/07 16:28:31 DEBUG : HEAD /flight/5/blah1/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.71.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: caba420a-9100-44be-b93e-190a364dda71
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
If-None-Match: *
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T162831Z
X-Amz-Security-Token: redacted

2025/11/07 16:28:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 16:28:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 16:28:31 DEBUG : HTTP RESPONSE (req 0x4000000a00)
2025/11/07 16:28:31 DEBUG : HTTP/1.1 304 Not Modified
Connection: close
Date: Fri, 07 Nov 2025 16:28:32 GMT
Etag: "40b134ab8a3dee5dd9760a7805fd495c"
Last-Modified: Fri, 07 Nov 2025 16:28:32 GMT
Server: AmazonS3
X-Amz-Id-2: redacted
X-Amz-Meta-Mtime: 1762528397.452931297
X-Amz-Request-Id: FPY4YWH3FW39NWST
X-Amz-Version-Id: 28V.gwNPBnTJ.maYdyWZAxFA5WjHuL0n

2025/11/07 16:28:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 16:28:31 ERROR : protected-new-new.txt: Failed to copy: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: FPY4YWH3FW39NWST, HostID: aCv13uXVbztkzzdSFMo0rVjOCPgt/q+v7FA722+MAqA6g19TZk14IR5pwlCNkiIR5LDtTWmhZKsTuYYpJm4ulQ==, api error NotModified: Not Modified
2025/11/07 16:28:31 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: FPY4YWH3FW39NWST, HostID: aCv13uXVbztkzzdSFMo0rVjOCPgt/q+v7FA722+MAqA6g19TZk14IR5pwlCNkiIR5LDtTWmhZKsTuYYpJm4ulQ==, api error NotModified: Not Modified
2025/11/07 16:28:31 INFO  :
Transferred:              4 B / 4 B, 100%, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.3s

2025/11/07 16:28:31 DEBUG : 5 go routines active
2025/11/07 16:28:31 NOTICE: Failed to copy: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: FPY4YWH3FW39NWST, HostID: aCv13uXVbztkzzdSFMo0rVjOCPgt/q+v7FA722+MAqA6g19TZk14IR5pwlCNkiIR5LDtTWmhZKsTuYYpJm4ulQ==, api error NotModified: Not Modified

@asdffdsa

yeah, i think i noticed a difference in the source code handling of --header versus –header-upload

I think this is a large part of the issue. It seems like the S3 rclone backend specifically checks for upload-headers and ignores those that are absent from the switch statement.

[asdffdsa (jojothehumanmonkey)]

@seanturner

I think this is a large part of the issue. It seems like the S3 rclone backend specifically checks for upload-headers and ignores those that are absent from the switch statement.

i agree. maybe create the PR.
and what about the 304? how should rclone handle that?

[seanturner]

@asdffdsa

and what about the 304? how should rclone handle that?

Hmm. Not sure. It seems very specific to both S3 and If-None-Match

It seems like the first HEAD request succeeds (404)

Then the POST request succeeds (200)

Then the second HEAD fails because the object exists now which is what the If-None-Match header is used for (304).

So the logic could be kind of tricky as I’m not sure if other providers have a follow up HEAD that would 304 when If-None-Match is available.

What do you think? I’ll look at getting a PR up.

[seanturner]

So the solution on my fork (which removes that Don't know how to set key "If-None-Match" on upload error message from the logs) was to include the –s3-no-head flag

Fork Diff:

diff --git i/backend/s3/s3.go w/backend/s3/s3.go
index 137b31adb..ae6546685 100644
--- i/backend/s3/s3.go
+++ w/backend/s3/s3.go
@@ -4538,6 +4538,10 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
                        ui.req.ContentLanguage = aws.String(value)
                case "content-type":
                        ui.req.ContentType = aws.String(value)
+               case "if-match":
+                       ui.req.ContentType = aws.String(value)
+               case "if-none-match":
+                       ui.req.ContentType = aws.String(value)
                case "x-amz-tagging":
                        ui.req.Tagging = aws.String(value)
                default:

–s3-no-head flag explanation

backend/s3/s3.go

Summary (this fails because a second HEAD request is sent and fails)

./rclone --version
rclone v1.72.0-beta.9285.fb895f69a
- os/version: darwin 15.7.2 (64 bit)
- os/kernel: 24.6.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.24.4
- go/linking: dynamic
- go/tags: none

./rclone copy \
    --header 'If-None-Match: *' \
    --retries=1 \
    --dump=headers,requests \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah3/
2025/11/07 12:41:06 NOTICE: Automatically setting -vv as --dump is enabled
2025/11/07 12:41:06 DEBUG : rclone: Version "v1.72.0-beta.9285.fb895f69a" starting with parameters ["./rclone" "copy" "--header" "If-None-Match: *" "--retries=1" "--dump=headers,requests" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blah3/"]
2025/11/07 12:41:06 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 12:41:06 DEBUG : Using config file from "/Users/sean/.config/rclone/rclone.conf"
2025/11/07 12:41:06 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/Users/sean/code/github/seanturner026/rclone/data/new/hyspex/vnir"
2025/11/07 12:41:06 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blah3/"
2025/11/07 12:41:06 DEBUG : :s3: detected overridden config - adding "{QXFZx}" suffix to name
2025/11/07 12:41:06 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 12:41:06 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 12:41:06 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blah3/" to be canonical ":s3{QXFZx}:my-bucket/flight/5/blah3"
2025/11/07 12:41:07 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:41:07 DEBUG : HTTP REQUEST (req 0x14000333180)
2025/11/07 12:41:07 DEBUG : HEAD /flight/5/blah3/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: bfb9d38b-8798-4c01-bbc6-dbead490a842
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
If-None-Match: *
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T174107Z
X-Amz-Security-Token: redacted

2025/11/07 12:41:07 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:41:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:41:08 DEBUG : HTTP RESPONSE (req 0x14000333180)
2025/11/07 12:41:08 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 07 Nov 2025 17:41:08 GMT
Server: AmazonS3
X-Amz-Id-2: redacted
X-Amz-Request-Id: DXZ3S3CQJKQQ7VWT

2025/11/07 12:41:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:41:08 DEBUG : protected-new-new.txt: Need to transfer - File not found at Destination
2025/11/07 12:41:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:41:08 DEBUG : HTTP REQUEST (req 0x14000435180)
2025/11/07 12:41:08 DEBUG : PUT /flight/5/blah3/protected-new-new.txt?x-id=PutObject HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Content-Length: 44
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 3eb2c1f8-e97a-472c-8a0f-07a7882b2b5f
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: Fm13rBtGoew4qjWrfmKKtQ==
Content-Type: text/plain; charset=utf-8
If-None-Match: *
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251107T174108Z
X-Amz-Decoded-Content-Length: 3
X-Amz-Meta-Mtime: 1762537136.640951798
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

3
11

0
x-amz-checksum-crc32:/GC4GQ==

2025/11/07 12:41:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:41:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:41:08 DEBUG : HTTP RESPONSE (req 0x14000435180)
2025/11/07 12:41:08 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 07 Nov 2025 17:41:09 GMT
Etag: "166d77ac1b46a1ec38aa35ab7e628ab5"
Server: AmazonS3
X-Amz-Checksum-Crc32: /GC4GQ==
X-Amz-Checksum-Type: FULL_OBJECT
X-Amz-Id-2: 7kkBG8skB6EDTvd231JbAXb63iyHxTjE7Mgaa7SwTAuRrqmbTljg+QIQkBEPBvX+izGTQhNC3sGIcFI+2Hby/w==
X-Amz-Request-Id: DXZ19S8FYEXSBR6N
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: uBhoW1IgUUZB_6v416GhS2lIzVLEVrRF

2025/11/07 12:41:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:41:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:41:08 DEBUG : HTTP REQUEST (req 0x14000500c80)
2025/11/07 12:41:08 DEBUG : HEAD /flight/5/blah3/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ee0711ee-2879-48ed-a09a-47504d79390c
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
If-None-Match: *
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T174108Z
X-Amz-Security-Token: redacted

2025/11/07 12:41:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:41:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:41:08 DEBUG : HTTP RESPONSE (req 0x14000500c80)
2025/11/07 12:41:08 DEBUG : HTTP/1.1 304 Not Modified
Connection: close
Date: Fri, 07 Nov 2025 17:41:09 GMT
Etag: "166d77ac1b46a1ec38aa35ab7e628ab5"
Last-Modified: Fri, 07 Nov 2025 17:41:09 GMT
Server: AmazonS3
X-Amz-Id-2: redacted
X-Amz-Meta-Mtime: 1762537136.640951798
X-Amz-Request-Id: DXZ9YDDQTACZ4ZYP
X-Amz-Version-Id: uBhoW1IgUUZB_6v416GhS2lIzVLEVrRF

2025/11/07 12:41:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:41:08 ERROR : protected-new-new.txt: Failed to copy: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: DXZ9YDDQTACZ4ZYP, HostID: redacted, api error NotModified: Not Modified
2025/11/07 12:41:08 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: DXZ9YDDQTACZ4ZYP, HostID: redacted, api error NotModified: Not Modified
2025/11/07 12:41:08 INFO  :
Transferred:              3 B / 3 B, 100%, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.2s

2025/11/07 12:41:08 DEBUG : 7 go routines active
2025/11/07 12:41:08 NOTICE: Failed to copy: operation error S3: HeadObject, https response error StatusCode: 304, RequestID: DXZ9YDDQTACZ4ZYP, HostID: redacted, api error NotModified: Not Modified

Summary (this succeeds and includes –s3-no-head which prevents the second HEAD request)

./rclone copy \
    --header 'If-None-Match: *' \
    --s3-no-head  \
    --retries=1 \
    --dump=headers,requests \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah5/
2025/11/07 12:50:01 NOTICE: Automatically setting -vv as --dump is enabled
2025/11/07 12:50:01 DEBUG : rclone: Version "v1.72.0-beta.9285.fb895f69a" starting with parameters ["./rclone" "copy" "--header" "If-None-Match: *" "--s3-no-head" "--retries=1" "--dump=headers,requests" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blah5/"]
2025/11/07 12:50:01 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 12:50:01 DEBUG : Using config file from "/Users/sean/.config/rclone/rclone.conf"
2025/11/07 12:50:01 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/Users/sean/code/github/seanturner026/rclone/data/new/hyspex/vnir"
2025/11/07 12:50:01 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blah5/"
2025/11/07 12:50:01 DEBUG : :s3: detected overridden config - adding "{4Fjwo}" suffix to name
2025/11/07 12:50:01 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 12:50:01 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 12:50:01 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blah5/" to be canonical ":s3{4Fjwo}:my-bucket/flight/5/blah5"
2025/11/07 12:50:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:50:02 DEBUG : HTTP REQUEST (req 0x140000f72c0)
2025/11/07 12:50:02 DEBUG : HEAD /flight/5/blah5/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: e1190295-43b5-487d-9693-adad51ca41b7
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
If-None-Match: *
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251107T175002Z
X-Amz-Security-Token: redacted

2025/11/07 12:50:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:50:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:50:03 DEBUG : HTTP RESPONSE (req 0x140000f72c0)
2025/11/07 12:50:03 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 07 Nov 2025 17:50:02 GMT
Server: AmazonS3
X-Amz-Id-2: redacted
X-Amz-Request-Id: ZSZ62RBZPPWV4QXB

2025/11/07 12:50:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:50:03 DEBUG : protected-new-new.txt: Need to transfer - File not found at Destination
2025/11/07 12:50:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:50:03 DEBUG : HTTP REQUEST (req 0x140004c2a00)
2025/11/07 12:50:03 DEBUG : PUT /flight/5/blah5/protected-new-new.txt?x-id=PutObject HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Content-Length: 44
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 37a9c7a6-54c0-49e3-9769-8c8212c342d4
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: Fm13rBtGoew4qjWrfmKKtQ==
Content-Type: text/plain; charset=utf-8
If-None-Match: *
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251107T175003Z
X-Amz-Decoded-Content-Length: 3
X-Amz-Meta-Mtime: 1762537136.640951798
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

3
11

0
x-amz-checksum-crc32:/GC4GQ==

2025/11/07 12:50:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 12:50:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:50:03 DEBUG : HTTP RESPONSE (req 0x140004c2a00)
2025/11/07 12:50:03 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 07 Nov 2025 17:50:04 GMT
Etag: "166d77ac1b46a1ec38aa35ab7e628ab5"
Server: AmazonS3
X-Amz-Checksum-Crc32: /GC4GQ==
X-Amz-Checksum-Type: FULL_OBJECT
X-Amz-Id-2: redacted
X-Amz-Request-Id: ZSZ9REE48J0JSFW1
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: z2WUCLQUxQCi4a4lX24hTndQdPJPC0WN

2025/11/07 12:50:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 12:50:03 DEBUG : protected-new-new.txt: size = 3 OK
2025/11/07 12:50:03 DEBUG : protected-new-new.txt: md5 = 166d77ac1b46a1ec38aa35ab7e628ab5 OK
2025/11/07 12:50:03 INFO  : protected-new-new.txt: Copied (new)
2025/11/07 12:50:03 INFO  :
Transferred:              3 B / 3 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.1s

2025/11/07 12:50:03 DEBUG : 7 go routines active

[asdffdsa (jojothehumanmonkey)]

@seanturner

So the solution on my fork

nice, just four lines of code!

---

@seanturner

So the logic could be kind of tricky as I’m not sure if other providers have a follow up HEAD that would 304 when If-None-Match is available.

i could do some simple testing about that.
as for other providers, rclone has a concept of quirks, which applies to S3.

[seanturner]

You know what, I think it really is just the 4 lines of code.

@@ -4538,6 +4538,10 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectIn
fo, options [
                        ui.req.ContentLanguage = aws.String(value)
                case "content-type":
                        ui.req.ContentType = aws.String(value)
+               case "if-match":
+                       ui.req.IfMatch = aws.String(value)
+               case "if-none-match":
+                       ui.req.IfNoneMatch = aws.String(value)
                case "x-amz-tagging":
                        ui.req.Tagging = aws.String(value)
                default:

My initial diff was using ui.req.ContentType which is obviously incorrect
When I updated the case statement properly this all seems to work. No issue with a second HEAD request failing.

I’ll submit a PR for this tomorrow morning

Summary

# Initial upload on protected prefix
./rclone copy \
    --header-upload 'If-None-Match: *' \
    --retries=1 \
    --dump=headers,requests \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt
2025/11/07 23:39:50 NOTICE: Automatically setting -vv as --dump is enabled
2025/11/07 23:39:50 DEBUG : rclone: Version "v1.72.0-beta.9285.fb895f69a" starting with parameters ["./rclone" "copy" "--header-upload" "If-None-Match: *" "--retries=1" "--dump=headers,requests" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt"]
2025/11/07 23:39:50 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 23:39:50 DEBUG : Using config file from "/Users/sean/.config/rclone/rclone.conf"
2025/11/07 23:39:50 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/Users/sean/code/github/seanturner026/rclone/data/new/hyspex/vnir"
2025/11/07 23:39:50 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt"
2025/11/07 23:39:50 DEBUG : :s3: detected overridden config - adding "{QXFZx}" suffix to name
2025/11/07 23:39:50 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 23:39:50 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 23:39:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:51 DEBUG : HTTP REQUEST (req 0x140006023c0)
2025/11/07 23:39:51 DEBUG : HEAD /flight/5/blah19/hyspex/vnir/protected.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 5991b63f-43af-4e3f-b219-b29375e3509e
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251108T043951Z
X-Amz-Security-Token: redacted

2025/11/07 23:39:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : HTTP RESPONSE (req 0x140006023c0)
2025/11/07 23:39:52 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 08 Nov 2025 04:39:51 GMT
Server: AmazonS3
X-Amz-Id-2: iOik43f0fujlX4/w82TGEJVoWOsuNScKB0bb0J40nubBkClcSAh4oKwgpwDra7gGKd+Tw/wv3TykmLlc2xt5+Kubfb9xmCvICzNNknl3myc=
X-Amz-Request-Id: 2WXYAXHVZAJPX2AD

2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt" to be canonical ":s3{QXFZx}:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt"
2025/11/07 23:39:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : HTTP REQUEST (req 0x140001cc280)
2025/11/07 23:39:52 DEBUG : HEAD /flight/5/blah19/hyspex/vnir/protected.txt/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ac73bb83-e69b-4762-9a3e-9b51fe28bc54
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251108T043952Z
X-Amz-Security-Token: redacted

2025/11/07 23:39:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : HTTP RESPONSE (req 0x140001cc280)
2025/11/07 23:39:52 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 08 Nov 2025 04:39:51 GMT
Server: AmazonS3
X-Amz-Id-2: VVSF/BJ7fzL3BGXkafFOZxzsm5fURiP1dW5Pm4tP1oSYuvDmTkzqcbM2jab2F7FQMsiboVTecRk+1b3v/TM4PyEjISmGboyhbzXHU3mvlio=
X-Amz-Request-Id: 2WXQBV6JJM8G80V4

2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : protected-new-new.txt: Need to transfer - File not found at Destination
2025/11/07 23:39:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : HTTP REQUEST (req 0x140001ccdc0)
2025/11/07 23:39:52 DEBUG : PUT /flight/5/blah19/hyspex/vnir/protected.txt/protected-new-new.txt?x-id=PutObject HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Content-Length: 48
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 2e34e8ab-52ac-463f-b1b5-61715273a107
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: d6MZVkYhuW+gZW4kxnlg7w==
Content-Type: text/plain; charset=utf-8
If-None-Match: *
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251108T043952Z
X-Amz-Decoded-Content-Length: 7
X-Amz-Meta-Mtime: 1762576580.436053686
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

7
111111

0
x-amz-checksum-crc32:i6nKaQ==

2025/11/07 23:39:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : HTTP RESPONSE (req 0x140001ccdc0)
2025/11/07 23:39:52 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sat, 08 Nov 2025 04:39:53 GMT
Etag: "77a319564621b96fa0656e24c67960ef"
Server: AmazonS3
X-Amz-Checksum-Crc32: i6nKaQ==
X-Amz-Checksum-Type: FULL_OBJECT
X-Amz-Id-2: HoO46tUgHIkEavAYcEhL4nccrzLpwTwed6WsFhAGKn3cRI51L3q8TfRwgE0PVqQAgoWTyOH/1PB7q24CFSU1g5+MHkwKGrhFqhirwv7lidY=
X-Amz-Request-Id: 2WXNNJ5M7H9X2HE4
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: kj.2UsZhwjFS.hyFi1e1aBIlsl7x86wp

2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : HTTP REQUEST (req 0x14000443180)
2025/11/07 23:39:52 DEBUG : HEAD /flight/5/blah19/hyspex/vnir/protected.txt/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 27ad20b3-8068-4507-bec3-3bda8fbd3ffe
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251108T043952Z
X-Amz-Security-Token: redacted

2025/11/07 23:39:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : HTTP RESPONSE (req 0x14000443180)
2025/11/07 23:39:52 DEBUG : HTTP/1.1 200 OK
Content-Length: 7
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sat, 08 Nov 2025 04:39:53 GMT
Etag: "77a319564621b96fa0656e24c67960ef"
Last-Modified: Sat, 08 Nov 2025 04:39:53 GMT
Server: AmazonS3
X-Amz-Id-2: Oh1oFxDB6zVjamZWf4vXNJDJOTEBnzB8P0Lz9WmL3I7F+d5nuqPUZyvIVWXEqRVyt7mv9KOIdGBZ3SSjtg8tSgzmPub2HPss+VLQiZiMvec=
X-Amz-Meta-Mtime: 1762576580.436053686
X-Amz-Request-Id: 2WXXX0813STZVWQQ
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: kj.2UsZhwjFS.hyFi1e1aBIlsl7x86wp

2025/11/07 23:39:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:39:52 DEBUG : protected-new-new.txt: size = 7 OK
2025/11/07 23:39:52 DEBUG : protected-new-new.txt: md5 = 77a319564621b96fa0656e24c67960ef OK
2025/11/07 23:39:52 INFO  : protected-new-new.txt: Copied (new)
2025/11/07 23:39:52 INFO  :
Transferred:              7 B / 7 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.2s

2025/11/07 23:39:52 DEBUG : 7 go routines active

# Updating file so that rclone knows to upload the file
echo 1111111 > data/new/hyspex/vnir/protected-new-new.txt

# This fails as expected because of `If-None-Match`
./rclone copy \
    --header-upload 'If-None-Match: *' \
    --retries=1 \
    --dump=headers,requests \
    --s3-provider=AWS \
    --s3-region=us-west-2 \
    --s3-env-auth=true \
    --s3-no-check-bucket \
    data/new/hyspex/vnir/protected-new-new.txt \
    :s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt
2025/11/07 23:40:03 NOTICE: Automatically setting -vv as --dump is enabled
2025/11/07 23:40:03 DEBUG : rclone: Version "v1.72.0-beta.9285.fb895f69a" starting with parameters ["./rclone" "copy" "--header-upload" "If-None-Match: *" "--retries=1" "--dump=headers,requests" "--s3-provider=AWS" "--s3-region=us-west-2" "--s3-env-auth=true" "--s3-no-check-bucket" "data/new/hyspex/vnir/protected-new-new.txt" ":s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt"]
2025/11/07 23:40:03 DEBUG : Creating backend with remote "data/new/hyspex/vnir/protected-new-new.txt"
2025/11/07 23:40:03 DEBUG : Using config file from "/Users/sean/.config/rclone/rclone.conf"
2025/11/07 23:40:03 DEBUG : fs cache: renaming child cache item "data/new/hyspex/vnir/protected-new-new.txt" to be canonical for parent "/Users/sean/code/github/seanturner026/rclone/data/new/hyspex/vnir"
2025/11/07 23:40:03 DEBUG : Creating backend with remote ":s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt"
2025/11/07 23:40:03 DEBUG : :s3: detected overridden config - adding "{QXFZx}" suffix to name
2025/11/07 23:40:03 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 23:40:03 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/11/07 23:40:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:40:04 DEBUG : HTTP REQUEST (req 0x140003d0a00)
2025/11/07 23:40:04 DEBUG : HEAD /flight/5/blah19/hyspex/vnir/protected.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: cc6259b7-92cc-42b5-8da3-b6bf3eb07279
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251108T044004Z
X-Amz-Security-Token: redacted

2025/11/07 23:40:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:40:05 DEBUG : HTTP RESPONSE (req 0x140003d0a00)
2025/11/07 23:40:05 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 08 Nov 2025 04:40:04 GMT
Server: AmazonS3
X-Amz-Id-2: aXW0lLBgTsKnc88bqPRc1sjrRoBSKAJKxIaAAyqea1LuuODCfRbEYU9sLyJnsY05WU/jdI6YXMi8ct8X66m1h3nHdqfrghznNThT1mIvxYo=
X-Amz-Request-Id: VABV48X5PR1387YY

2025/11/07 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:40:05 DEBUG : fs cache: renaming cache item ":s3:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt" to be canonical ":s3{QXFZx}:my-bucket/flight/5/blah19/hyspex/vnir/protected.txt"
2025/11/07 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:40:05 DEBUG : HTTP REQUEST (req 0x140005e23c0)
2025/11/07 23:40:05 DEBUG : HEAD /flight/5/blah19/hyspex/vnir/protected.txt/protected-new-new.txt HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 27f264bd-2739-4af2-bcf9-00d36789a1b1
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251108T044005Z
X-Amz-Security-Token: redacted

2025/11/07 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:40:05 DEBUG : HTTP RESPONSE (req 0x140005e23c0)
2025/11/07 23:40:05 DEBUG : HTTP/1.1 200 OK
Content-Length: 7
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sat, 08 Nov 2025 04:40:06 GMT
Etag: "77a319564621b96fa0656e24c67960ef"
Last-Modified: Sat, 08 Nov 2025 04:39:53 GMT
Server: AmazonS3
X-Amz-Id-2: 62gBiYjFjhEypPcukEkX1Q7IWYVEQ+mqNlW5rvrJNXchl2J0ZlZFFoNZUGYiZ9Df2d5zgN4Zc7e/aVO9UobEGCBIQJMy9lddmf9xJFXpTz0=
X-Amz-Meta-Mtime: 1762576580.436053686
X-Amz-Request-Id: VABRFR54A0JBGA7Q
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: kj.2UsZhwjFS.hyFi1e1aBIlsl7x86wp

2025/11/07 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:40:05 DEBUG : protected-new-new.txt: size = 8 (Local file system at /Users/sean/code/github/seanturner026/rclone/data/new/hyspex/vnir)
2025/11/07 23:40:05 DEBUG : protected-new-new.txt: size = 7 (S3 bucket my-bucket path flight/5/blah19/hyspex/vnir/protected.txt)
2025/11/07 23:40:05 DEBUG : protected-new-new.txt: Sizes differ
2025/11/07 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:40:05 DEBUG : HTTP REQUEST (req 0x140003d1180)
2025/11/07 23:40:05 DEBUG : PUT /flight/5/blah19/hyspex/vnir/protected.txt/protected-new-new.txt?x-id=PutObject HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9285.fb895f69a
Content-Length: 49
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c2a40e36-dc2a-4024-8627-64dc950425ac
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: aP28kgX2DunU0zE9+PuDlQ==
Content-Type: text/plain; charset=utf-8
If-None-Match: *
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251108T044005Z
X-Amz-Decoded-Content-Length: 8
X-Amz-Meta-Mtime: 1762576800.53755878
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

8
1111111

0
x-amz-checksum-crc32:OoCZnQ==

2025/11/07 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/07 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:40:05 DEBUG : HTTP RESPONSE (req 0x140003d1180)
2025/11/07 23:40:05 DEBUG : HTTP/1.1 412 Precondition Failed
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 08 Nov 2025 04:40:04 GMT
Server: AmazonS3
X-Amz-Id-2: EEAkxXx7Djckw7Ohpp/QCsETItLJO1MWDX0RTs59HjH//MLAFSo7bbcXK6sOwB9B6VZ9v1Od+e1LbJuzjG/DM6+Ysq+tfZHSihT/+ERJ/cc=
X-Amz-Request-Id: VABGCTKC5W3TXPX8

2025/11/07 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/07 23:40:05 ERROR : protected-new-new.txt: Failed to copy: operation error S3: PutObject, https response error StatusCode: 412, RequestID: VABGCTKC5W3TXPX8, HostID: EEAkxXx7Djckw7Ohpp/QCsETItLJO1MWDX0RTs59HjH//MLAFSo7bbcXK6sOwB9B6VZ9v1Od+e1LbJuzjG/DM6+Ysq+tfZHSihT/+ERJ/cc=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
2025/11/07 23:40:05 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: PutObject, https response error StatusCode: 412, RequestID: VABGCTKC5W3TXPX8, HostID: EEAkxXx7Djckw7Ohpp/QCsETItLJO1MWDX0RTs59HjH//MLAFSo7bbcXK6sOwB9B6VZ9v1Od+e1LbJuzjG/DM6+Ysq+tfZHSihT/+ERJ/cc=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
2025/11/07 23:40:05 INFO  :
Transferred:              8 B / 8 B, 100%, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s

2025/11/07 23:40:05 DEBUG : 8 go routines active
2025/11/07 23:40:05 NOTICE: Failed to copy: operation error S3: PutObject, https response error StatusCode: 412, RequestID: VABGCTKC5W3TXPX8, HostID: EEAkxXx7Djckw7Ohpp/QCsETItLJO1MWDX0RTs59HjH//MLAFSo7bbcXK6sOwB9B6VZ9v1Od+e1LbJuzjG/DM6+Ysq+tfZHSihT/+ERJ/cc=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold

[seanturner]

Draft PR here. I can make any changes before ticking the last box

s3: add support for conditional request headers

[seanturner]

I think that PR is ready to go. Not trying to rush the system or anything, mainly mentioning as much since my last post mentioned that the PR was a draft.

Thanks as well for all of the quick correspondence! rclone seems to have a cool dev community behind it.

[seanturner]

Any idea if there’s something I’m meant to do to get my PR looked at? Seems like a maintainer never ran a workflow?

[ncw (Nick Craig-Wood)]

@seanturner

I think this is a large part of the issue. It seems like the S3 rclone backend specifically checks for upload-headers and ignores those that are absent from the switch statement.

This is to do with the AWS SDK - there isn't a method for setting general headers. Most other backends don't have this restriction.

I'll take a look at your PR later today. It seems to be doing the right thing!

[seanturner]

Hi again

I think I’ve got another change ready which should add support for these headers to Multi-Part Uploads.

Mind if I flick another PR over? The main change is to ensure that the CompleteMultiPartUploadInput (pkg.go.dev) includes the headers. It appears that my previous PR exposes the headers and the only change needed is to add the attribute references in CompleteMultiPartUploadInput (which doesn’t appear to need any conditional logic due to the default value)

Local branch, able to PUT successfully with MPU and with Bucket Policy enabled that requires the header

./rclone --version
rclone v1.72.0-beta.9300.83b83f776

• os/version: darwin 15.7.2 (64 bit)
• os/kernel: 24.6.0 (arm64)
• os/type: darwin
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.24.4
• go/linking: dynamic
• go/tags: none

./rclone copy
--header-upload 'If-None-Match: *'
--retries=1 --dump=headers
--s3-provider=AWS
--s3-region=us-west-2
--s3-env-auth=true --s3-upload-cutoff=1M
--s3-no-check-bucket
data/new/hyspex/vnir/251031_165226790_Camera_1.iiq
:s3:BUCKET_NAME/flight/5/blah20/hyspex/vnir/iiq-again.txt

omitting many other requests for brevity

2025/11/17 09:57:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:18 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload wrote chunk 20 with 5242880 bytes and etag "8f4949d53e8543afb3682ebb20327896"
2025/11/17 09:57:18 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload: starting chunk 22 size 3.577Mi offset 110Mi/113.577Mi
2025/11/17 09:57:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 09:57:18 DEBUG : HTTP REQUEST (req 0x14000b14a00)
2025/11/17 09:57:18 DEBUG : PUT /flight/5/blah20/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq?partNumber=23&uploadId=Bj729lKwu88eG7yCtpkGVwX8ObWPteOwem_EORc.tk46_xOn2A4fbQ54dSz8ZoNSIrrLwOg_UM7ApTVyHZfYDUTg1OKRE.n_TnPzG7KThK3LWctefNLPttJ2XlgYTO5qG0_mQ4E4yoK5amzWEDmhow--&x-id=UploadPart HTTP/1.1
Host: BUCKET_NAME.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9300.83b83f776
Content-Length: 3751060
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: cde9c2cf-f83a-485c-bf37-ef332c346302
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Encoding: aws-chunked
Content-Md5: JXOuyzYy17WqrQE0lVl19g==
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251117T145718Z
X-Amz-Decoded-Content-Length: 3751014
X-Amz-Security-Token: redacted
X-Amz-Trailer: x-amz-checksum-crc32

2025/11/17 09:57:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 09:57:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:24 DEBUG : HTTP RESPONSE (req 0x14000b14a00)
2025/11/17 09:57:24 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 17 Nov 2025 14:57:19 GMT
Etag: "2573aecb3632d7b5aaad0134955975f6"
Server: AmazonS3
X-Amz-Checksum-Crc32: P5gLaA==
X-Amz-Id-2: ezs5wXqORjpzNKcPP5qAjn0yaArapZV8+8c29zfL+ev/cDjjRBqbQwE7MtDqL9H05fP4XPsmXTE=
X-Amz-Request-Id: 6TKW19HZJ45R8YNA
X-Amz-Server-Side-Encryption: AES256

2025/11/17 09:57:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:24 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload wrote chunk 23 with 3751014 bytes and etag "2573aecb3632d7b5aaad0134955975f6"
2025/11/17 09:57:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:28 DEBUG : HTTP RESPONSE (req 0x14000b14780)
2025/11/17 09:57:28 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 17 Nov 2025 14:57:03 GMT
Etag: "7fef1c529a5560201e0318518a8cabc2"
Server: AmazonS3
X-Amz-Checksum-Crc32: Hhy0rw==
X-Amz-Id-2: k8lgncJt0S/Usd0jDpUIYBaJ0Z6Sqa7Ez+OPNxn539Tzb7uKuK+0lWTBtA0qXHNX9U7o3Eao200=
X-Amz-Request-Id: S9PTA4Z2QJ3VJ8KN
X-Amz-Server-Side-Encryption: AES256

2025/11/17 09:57:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:28 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload wrote chunk 19 with 5242880 bytes and etag "7fef1c529a5560201e0318518a8cabc2"
2025/11/17 09:57:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:29 DEBUG : HTTP RESPONSE (req 0x14000898640)
2025/11/17 09:57:29 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 17 Nov 2025 14:57:14 GMT
Etag: "c42083627a7fe23803426478451fa65f"
Server: AmazonS3
X-Amz-Checksum-Crc32: NYGa9A==
X-Amz-Id-2: C3qvffoXRxCgtPJnMdB0bgjVTgQtadKEnBS/t7FrTFoDR60nW8JnbJwhdEz2eMe4sWCQK7+dAAw=
X-Amz-Request-Id: J0M8SKZ34C4K22P6
X-Amz-Server-Side-Encryption: AES256

2025/11/17 09:57:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:29 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload wrote chunk 21 with 5242880 bytes and etag "c42083627a7fe23803426478451fa65f"
2025/11/17 09:57:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:31 DEBUG : HTTP RESPONSE (req 0x140006b8780)
2025/11/17 09:57:31 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 17 Nov 2025 14:57:14 GMT
Etag: "1b6f615f79f3516fc216552d729f096a"
Server: AmazonS3
X-Amz-Checksum-Crc32: t/bYyQ==
X-Amz-Id-2: u9C0tEI/FOWN00+9bPwS2l6ZxpV6mRnnxH+QzGwaazw0ExMFVZ1dgY97t1Nc2I/a8Xu4Nn9i/zM=
X-Amz-Request-Id: J0M30NRRJ5PKH4A3
X-Amz-Server-Side-Encryption: AES256

this PUT shows that the If-None-Match header is appearing

2025/11/17 09:57:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:31 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload wrote chunk 22 with 5242880 bytes and etag "1b6f615f79f3516fc216552d729f096a"
2025/11/17 09:57:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 09:57:31 DEBUG : HTTP REQUEST (req 0x140002e17c0)
2025/11/17 09:57:31 DEBUG : POST /flight/5/blah20/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq?uploadId=Bj729lKwu88eG7yCtpkGVwX8ObWPteOwem_EORc.tk46_xOn2A4fbQ54dSz8ZoNSIrrLwOg_UM7ApTVyHZfYDUTg1OKRE.n_TnPzG7KThK3LWctefNLPttJ2XlgYTO5qG0_mQ4E4yoK5amzWEDmhow-- HTTP/1.1
Host: BUCKET_NAME.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9300.83b83f776
Content-Length: 2275
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: f1c7ee13-39ec-4b19-b70d-e7fac085604e
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/xml
If-None-Match: *
X-Amz-Content-Sha256: 0e762c45cda66a449cec380a7ef4ddf153efe33227b2cba65c20aab82fea9127
X-Amz-Date: 20251117T145731Z
X-Amz-Security-Token: redacted

2025/11/17 09:57:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 09:57:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:31 DEBUG : HTTP RESPONSE (req 0x140002e17c0)
2025/11/17 09:57:31 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 17 Nov 2025 14:57:32 GMT
Server: AmazonS3
X-Amz-Id-2: cgvH3e1Ibwm1irFCMVvtAuFmrZbXY7MXThIoINlyT7wDFZoLiJ3BRxjMwlSLuHn8++QHm5v3bhc=
X-Amz-Request-Id: 0AFFHRC8VZKHPBX2
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: baSobQuA_DUjIJJFCYXStCQdDDw7gJCJ

2025/11/17 09:57:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:31 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload "Bj729lKwu88eG7yCtpkGVwX8ObWPteOwem_EORc.tk46_xOn2A4fbQ54dSz8ZoNSIrrLwOg_UM7ApTVyHZfYDUTg1OKRE.n_TnPzG7KThK3LWctefNLPttJ2XlgYTO5qG0_mQ4E4yoK5amzWEDmhow--" finished
2025/11/17 09:57:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 09:57:31 DEBUG : HTTP REQUEST (req 0x14000b15180)
2025/11/17 09:57:31 DEBUG : HEAD /flight/5/blah20/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq HTTP/1.1
Host: BUCKET_NAME.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9300.83b83f776
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d7d22640-1b93-448a-9dfd-00d109c23d83
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251117T145731Z
X-Amz-Security-Token: redacted

2025/11/17 09:57:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 09:57:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:31 DEBUG : HTTP RESPONSE (req 0x14000b15180)
2025/11/17 09:57:31 DEBUG : HTTP/1.1 200 OK
Content-Length: 119094374
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Mon, 17 Nov 2025 14:57:32 GMT
Etag: "dea59c897a8f0dcc511b1d40da83511f-23"
Last-Modified: Mon, 17 Nov 2025 14:55:59 GMT
Server: AmazonS3
X-Amz-Id-2: GzEPmLKBnBDIoPjUG5kxZcaagCjCmICIunPYBsYKzXM9lKEbPA5dG/S5cIK4tQCM1u2t8kdR/bk=
X-Amz-Meta-Md5chksum: 3DIUhavNn/31+S0C5ctmJA==
X-Amz-Meta-Mtime: 1762963423.291793376
X-Amz-Request-Id: 0AFAWM2R4ADEXN7D
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: baSobQuA_DUjIJJFCYXStCQdDDw7gJCJ

2025/11/17 09:57:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 09:57:31 DEBUG : 251031_165226790_Camera_1.iiq: Multipart upload Etag: dea59c897a8f0dcc511b1d40da83511f-23 OK
2025/11/17 09:57:31 DEBUG : 251031_165226790_Camera_1.iiq: size = 119094374 OK
2025/11/17 09:57:31 DEBUG : 251031_165226790_Camera_1.iiq: md5 = dc321485abcd9ffdf5f92d02e5cb6624 OK
2025/11/17 09:57:31 INFO : 251031_165226790_Camera_1.iiq: Copied (new)
2025/11/17 09:57:31 INFO :
Transferred: 113.577 MiB / 113.577 MiB, 100%, 644.024 KiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1m33.0s

2025/11/17 09:57:31 DEBUG : 11 go routines active

Local branch, I reset to commit 83b83f776 where I added the headers previously and get a 403 because the header is missing

git reset --hard 83b83f776
HEAD is now at 83b83f776 s3: add support for --upload-header If-Match and If-None-Match

make
fatal: No names found, cannot describe anything.
go build -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=v1.72.0-beta.9300.83b83f776.original-s3-conditional-header-commit"
GitHub - rclone/rclone: "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
mkdir -p go env GOPATH/bin/
cp -av rclonego env GOEXE go env GOPATH/bin/rclonego env GOEXE.new
rclone -> /Users/sean/go/bin/rclone.new
mv -v go env GOPATH/bin/rclonego env GOEXE.new go env GOPATH/bin/rclonego env GOEXE
/Users/sean/go/bin/rclone.new -> /Users/sean/go/bin/rclone

./rclone --version
rclone v1.72.0-beta.9300.83b83f776.original-s3-conditional-header-commit

• os/version: darwin 15.7.2 (64 bit)
• os/kernel: 24.6.0 (arm64)
• os/type: darwin
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.24.4
• go/linking: dynamic
• go/tags: none

./rclone copy
--header-upload 'If-None-Match: *'
--retries=1 --dump=headers
--s3-provider=AWS
--s3-region=us-west-2
--s3-env-auth=true --s3-upload-cutoff=1M
--s3-no-check-bucket
data/new/hyspex/vnir/251031_165226790_Camera_1.iiq
:s3:BUCKET_NAME/flight/5/blah22/hyspex/vnir/iiq-again.txt

omitting many other requests for brevity

This is the last PUT which should include the HEADER (but doesn't) and therefore 403s due to the Bucket Policy

2025/11/17 13:21:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 13:21:58 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload wrote chunk 22 with 5242880 bytes and etag "1b6f615f79f3516fc216552d729f096a"
2025/11/17 13:21:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 13:21:58 DEBUG : HTTP REQUEST (req 0x1400066fcc0)
2025/11/17 13:21:58 DEBUG : POST /flight/5/blah22/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq?uploadId=jTyYmd1x92u3WeqlmHbChVCoQZdG.Uo5n.6FQH3ygXwTHnYgKMUoQ3vnYy4JLeMp53dUhrHSeLgLb6IquRWmMh5PNt65pgREAHJixi2xn0oXcBLIvehDgB4Ma8mJ4QYpJQlUvvkwhIJ_fLFQx_DfdQ-- HTTP/1.1
Host: BUCKET_NAME.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9300.83b83f776.original-s3-conditional-header-commit
Content-Length: 2275
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d0789f6b-a1c3-47c6-a4d6-0de981670aae
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/xml
X-Amz-Content-Sha256: 0e762c45cda66a449cec380a7ef4ddf153efe33227b2cba65c20aab82fea9127
X-Amz-Date: 20251117T182158Z
X-Amz-Security-Token: redacted

2025/11/17 13:21:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 13:21:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 13:21:58 DEBUG : HTTP RESPONSE (req 0x1400066fcc0)
2025/11/17 13:21:58 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 17 Nov 2025 18:21:58 GMT
Server: AmazonS3
X-Amz-Id-2: jZh1MPeTGNBnCywwvnRI0BZt256/gxNyfDkdWsTCR7YFGKkDaU1yYq5qnr4cYTQrdgEE2DZ54LM=
X-Amz-Request-Id: CRCSFD1AXVFK00S7

2025/11/17 13:21:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 13:21:58 DEBUG : 251031_165226790_Camera_1.iiq: Cancelling multipart upload
2025/11/17 13:21:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 13:21:58 DEBUG : HTTP REQUEST (req 0x1400081a640)
2025/11/17 13:21:58 DEBUG : DELETE /flight/5/blah22/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq?uploadId=jTyYmd1x92u3WeqlmHbChVCoQZdG.Uo5n.6FQH3ygXwTHnYgKMUoQ3vnYy4JLeMp53dUhrHSeLgLb6IquRWmMh5PNt65pgREAHJixi2xn0oXcBLIvehDgB4Ma8mJ4QYpJQlUvvkwhIJ_fLFQx_DfdQ--&x-id=AbortMultipartUpload HTTP/1.1
Host: BUCKET_NAME.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.72.0-beta.9300.83b83f776.original-s3-conditional-header-commit
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: a5c3590c-fc1d-4f66-9273-f61fac5452cc
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251117T182158Z
X-Amz-Security-Token: redacted

2025/11/17 13:21:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/11/17 13:21:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 13:21:58 DEBUG : HTTP RESPONSE (req 0x1400081a640)
2025/11/17 13:21:58 DEBUG : HTTP/1.1 204 No Content
Date: Mon, 17 Nov 2025 18:21:59 GMT
Server: AmazonS3
X-Amz-Id-2: 0U2+HT3mvcubAppsdhRMt0Pu+qVBVWCGtKdsUH1muV9RWVTJxkJgyru6jkxlq2Lr9WqCIYcf+ZA=
X-Amz-Request-Id: CRCV2D33FBMJC6BV

2025/11/17 13:21:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/11/17 13:21:58 DEBUG : 251031_165226790_Camera_1.iiq: multipart upload "jTyYmd1x92u3WeqlmHbChVCoQZdG.Uo5n.6FQH3ygXwTHnYgKMUoQ3vnYy4JLeMp53dUhrHSeLgLb6IquRWmMh5PNt65pgREAHJixi2xn0oXcBLIvehDgB4Ma8mJ4QYpJQlUvvkwhIJ_fLFQx_DfdQ--" aborted
2025/11/17 13:21:58 ERROR : 251031_165226790_Camera_1.iiq: Failed to copy: multipart upload: failed to finalise: failed to complete multipart upload "jTyYmd1x92u3WeqlmHbChVCoQZdG.Uo5n.6FQH3ygXwTHnYgKMUoQ3vnYy4JLeMp53dUhrHSeLgLb6IquRWmMh5PNt65pgREAHJixi2xn0oXcBLIvehDgB4Ma8mJ4QYpJQlUvvkwhIJ_fLFQx_DfdQ--": operation error S3: CompleteMultipartUpload, https response error StatusCode: 403, RequestID: CRCSFD1AXVFK00S7, HostID: jZh1MPeTGNBnCywwvnRI0BZt256/gxNyfDkdWsTCR7YFGKkDaU1yYq5qnr4cYTQrdgEE2DZ54LM=, api error AccessDenied: User: arn:aws:sts::513288716225:assumed-role/AWSReservedSSO_engineers_806de1e8b28d3806/sean.turner is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::BUCKET_NAME/flight/5/blah22/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq" with an explicit deny in a resource-based policy
2025/11/17 13:21:58 ERROR : Attempt 1/1 failed with 1 errors and: multipart upload: failed to finalise: failed to complete multipart upload "jTyYmd1x92u3WeqlmHbChVCoQZdG.Uo5n.6FQH3ygXwTHnYgKMUoQ3vnYy4JLeMp53dUhrHSeLgLb6IquRWmMh5PNt65pgREAHJixi2xn0oXcBLIvehDgB4Ma8mJ4QYpJQlUvvkwhIJ_fLFQx_DfdQ--": operation error S3: CompleteMultipartUpload, https response error StatusCode: 403, RequestID: CRCSFD1AXVFK00S7, HostID: jZh1MPeTGNBnCywwvnRI0BZt256/gxNyfDkdWsTCR7YFGKkDaU1yYq5qnr4cYTQrdgEE2DZ54LM=, api error AccessDenied: User: arn:aws:sts::513288716225:assumed-role/AWSReservedSSO_engineers_806de1e8b28d3806/sean.turner is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::BUCKET_NAME/flight/5/blah22/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq" with an explicit deny in a resource-based policy
2025/11/17 13:21:58 INFO :
Transferred: 113.577 MiB / 113.577 MiB, 100%, 686.745 KiB/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 1m33.1s

2025/11/17 13:21:58 DEBUG : 13 go routines active
2025/11/17 13:21:58 NOTICE: Failed to copy: multipart upload: failed to finalise: failed to complete multipart upload "jTyYmd1x92u3WeqlmHbChVCoQZdG.Uo5n.6FQH3ygXwTHnYgKMUoQ3vnYy4JLeMp53dUhrHSeLgLb6IquRWmMh5PNt65pgREAHJixi2xn0oXcBLIvehDgB4Ma8mJ4QYpJQlUvvkwhIJ_fLFQx_DfdQ--": operation error S3: CompleteMultipartUpload, https response error StatusCode: 403, RequestID: CRCSFD1AXVFK00S7, HostID: jZh1MPeTGNBnCywwvnRI0BZt256/gxNyfDkdWsTCR7YFGKkDaU1yYq5qnr4cYTQrdgEE2DZ54LM=, api error AccessDenied: User: arn:aws:sts::513288716225:assumed-role/AWSReservedSSO_engineers_806de1e8b28d3806/sean.turner is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::BUCKET_NAME/flight/5/blah22/hyspex/vnir/iiq-again.txt/251031_165226790_Camera_1.iiq" with an explicit deny in a resource-based policy

[ncw (Nick Craig-Wood)]

Go for it - thanks

[sanjay_kumar]

Rclone does not currently support If-None-Match or If-Match headers for S3 uploads. The --header-upload flag works for some HTTP backends, but the S3 backend doesn’t allow setting these conditional upload headers, which is why you see the error. As a workaround, you can use flags like --ignore-existing or --update to avoid overwriting files, or check if the object exists with rclone ls before uploading. For strict conditional uploads based on these headers, you would need to use the AWS CLI instead.