Problem copying to S3 bucket

Tony_Williams · January 20, 2025, 1:20am

What is the problem you are having with rclone?

I cannot get Rclone uploading some files to an S3 bucket.

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -vv --log-file=log.txt --config ${PWD}/rclone.conf copy ${PWD}/LICENSE nvr-backup:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[nvr-backup]
type = s3
provider = AWS
env_auth = false
region = ap-southeast-2
location_constraint = ap-southeast-2
acl = public-read-write
sse_kms_key_id = XXX
access_key_id = XXX
secret_access_key = XXX
session_token = XXX

A log from the command that you were trying to run with the `-vv` flag


2025/01/20 12:00:04 DEBUG : rclone: Version "v1.69.0" starting with parameters ["rclone" "-vv" "--log-file=log.txt" "--config" "/Users/tonyw/OneDrive/work/[directory]/rclone.conf" "copy" "/Users/tonyw/OneDrive/work/[directory]/LICENSE" "nvr-backup:/"]
2025/01/20 12:00:04 DEBUG : Creating backend with remote "/Users/tonyw/OneDrive/work/[directory]/LICENSE"
2025/01/20 12:00:04 DEBUG : Using config file from "/Users/tonyw/OneDrive/work/[directory]/rclone.conf"
2025/01/20 12:00:04 DEBUG : fs cache: renaming child cache item "/Users/tonyw/OneDrive/work/[directory]/LICENSE" to be canonical for parent "/Users/tonyw/OneDrive/work/[directory]"
2025/01/20 12:00:04 DEBUG : Creating backend with remote "nvr-backup:/"
2025/01/20 12:00:04 DEBUG : fs cache: renaming cache item "nvr-backup:/" to be canonical "nvr-backup:"
2025/01/20 12:00:04 ERROR : Attempt 1/3 failed with 1 errors and: operation error S3: HeadObject, serialization failed: serialization failed: input member Key must not be empty
2025/01/20 12:00:04 ERROR : Attempt 2/3 failed with 1 errors and: operation error S3: HeadObject, serialization failed: serialization failed: input member Key must not be empty
2025/01/20 12:00:04 ERROR : Attempt 3/3 failed with 1 errors and: operation error S3: HeadObject, serialization failed: serialization failed: input member Key must not be empty
2025/01/20 12:00:04 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2025/01/20 12:00:04 DEBUG : 4 go routines active
2025/01/20 12:00:04 NOTICE: Failed to copy: operation error S3: HeadObject, serialization failed: serialization failed: input member Key must not be empty

I can't get copy to work. I can copy a file using the same credentials and the aws command aws s3 cp ./makefile s3://nvr-backup-340752817366

rclone seems to authenticate OK - the command rclone --config ${PWD}/rclone.conf ls nvr-backup:/ gives me a list of the files in the bucket.

the command rclone --config ${PWD}/rclone.conf copy nvr-backup:/nvr-backup-340752817366/test.txt ${PWD} works fine. It's only the other way.

I have tried nvr-backup: and nvr-backup:/ and nvr-backup:/nvr-backup-340752817366/ but none work.

I also tried copyto but got the same problem.

asdffdsa · January 20, 2025, 2:11pm

welcome to the forum,

cannot copy files to the root of the S3 remote. need to copy file into a bucket.

try
rclone -vv --log-file=log.txt --config ${PWD}/rclone.conf copy ${PWD}/LICENSE nvr-backup:nvr-backup-340752817366

and to keep the logs smaller, use --retries=1

Tony_Williams · January 20, 2025, 11:44pm

Thanks for your help.

I thought I had already tried that and I think I did. The log now looks like

2025/01/21 10:42:16 DEBUG : rclone: Version "v1.69.0" starting with parameters ["rclone" "-vv" "--log-file=log.txt" "--retries=1" "--config" "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/rclone.conf" "copy" "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/LICENSE" "nvr-backup:nvr-backup-340752817366"]
2025/01/21 10:42:16 DEBUG : Creating backend with remote "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/LICENSE"
2025/01/21 10:42:16 DEBUG : Using config file from "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/rclone.conf"
2025/01/21 10:42:16 DEBUG : fs cache: renaming child cache item "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/LICENSE" to be canonical for parent "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco"
2025/01/21 10:42:16 DEBUG : Creating backend with remote "nvr-backup:nvr-backup-340752817366"
2025/01/21 10:42:16 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: Z97SKG992E4R5J5T, HostID: P2fPKenaiYibn1vCdAeVSWQCkPjMWIyGIBU4N0ZfDxb/PlvCdiHVIMbRMhoPMwM/3bfbnvUq2Ps=, api error BadRequest: Bad Request
2025/01/21 10:42:16 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2025/01/21 10:42:16 DEBUG : 4 go routines active
2025/01/21 10:42:16 NOTICE: Failed to copy: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: Z97SKG992E4R5J5T, HostID: P2fPKenaiYibn1vCdAeVSWQCkPjMWIyGIBU4N0ZfDxb/PlvCdiHVIMbRMhoPMwM/3bfbnvUq2Ps=, api error BadRequest: Bad Request

asdffdsa · January 20, 2025, 11:50pm

how are you creating the session_token and then feeding it to rclone?

for a deeper look at the api calls, try --dump flag, such as --dump=headers

Tony_Williams · January 21, 2025, 12:06am

If I dump the headers it looks like I get a 403 for a CreateBucket call. I will get CreateBucket added to the permissions and see if that fixes it.

asdffdsa · January 21, 2025, 12:07am

i prefer not to add that permission and saves a api call.
by using --s3-no-check-bucket

Tony_Williams · January 21, 2025, 4:01am

I now have the permissions I need (I think). Now the log says -

2025/01/21 14:49:57 DEBUG : rclone: Version "v1.69.0" starting with parameters ["rclone" "-vv" "--dump=headers" "--log-file=log.txt" "--retries=1" "--config" "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/rclone.conf" "copy" "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/LICENSE" "nvr-backup:nvr-backup-340752817366/"]
2025/01/21 14:49:57 DEBUG : Creating backend with remote "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/LICENSE"
2025/01/21 14:49:57 DEBUG : Using config file from "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/rclone.conf"
2025/01/21 14:49:57 DEBUG : fs cache: renaming child cache item "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco/LICENSE" to be canonical for parent "/Users/tony.williams/Onedrive/work/unifi-protect-backup-invenco"
2025/01/21 14:49:57 DEBUG : Creating backend with remote "nvr-backup:nvr-backup-340752817366/"
2025/01/21 14:49:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/01/21 14:49:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/01/21 14:49:57 DEBUG : fs cache: renaming cache item "nvr-backup:nvr-backup-340752817366/" to be canonical "nvr-backup:nvr-backup-340752817366"
2025/01/21 14:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/21 14:49:57 DEBUG : HTTP REQUEST (req 0x140004e6280)
2025/01/21 14:49:57 DEBUG : HEAD /LICENSE HTTP/1.1
Host: nvr-backup-340752817366.s3.ap-southeast-2.amazonaws.com
User-Agent: rclone/v1.69.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: bc777376-92c2-4821-b74a-2de66e5a95e1
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250121T034957Z
X-Amz-Security-Token: 
2025/01/21 14:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/21 14:49:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/21 14:49:57 DEBUG : HTTP RESPONSE (req 0x140004e6280)
2025/01/21 14:49:57 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 21 Jan 2025 03:49:57 GMT
Server: AmazonS3
X-Amz-Delete-Marker: true
X-Amz-Id-2: PZ4eS+mq0M+g8ZM2S2J9hjH3R7E4BBCKoaw/Pqts1iyZcEwj5Nmj2uyRc3qpUaOAcW1iRHa6/q/9lq/Qjm/srQ==
X-Amz-Request-Id: 5WK9SBS0EGZVF0MP
X-Amz-Version-Id: koLai9gvuHQ.cSBtOF9VbY7GlHJFYJGv


2025/01/21 14:49:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/21 14:49:57 DEBUG : LICENSE: Need to transfer - File not found at Destination
2025/01/21 14:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/21 14:49:57 DEBUG : HTTP REQUEST (req 0x140004e6dc0)
2025/01/21 14:49:57 DEBUG : PUT / HTTP/1.1
Host: nvr-backup-340752817366.s3.ap-southeast-2.amazonaws.com
User-Agent: rclone/v1.69.0
Content-Length: 158
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 1c8d2c83-5370-4e2e-891b-0b3fd13541a8
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/xml
X-Amz-Content-Sha256: 8bfaf12a44a138dc6f38eb5c291558f2c551b0f18ab6e2a5e7f9a00176b25240
X-Amz-Date: 20250121T034957Z
X-Amz-Security-Token: [redacted]

2025/01/21 14:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/21 14:49:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/21 14:49:58 DEBUG : HTTP RESPONSE (req 0x140004e6dc0)
2025/01/21 14:49:58 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 21 Jan 2025 03:49:57 GMT
Server: AmazonS3
X-Amz-Bucket-Region: ap-southeast-2
X-Amz-Id-2: kjzlXksBJN9/fw+5RN/U2CrYIUmRZHjxa04dx82yxfY6ErjG3Q25PXovejic2FjBTq9P7Tc3njvwPyi0Q1yv0A==
X-Amz-Request-Id: 5WK2816XAF0E2GEH

2025/01/21 14:49:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/21 14:49:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/21 14:49:58 DEBUG : HTTP REQUEST (req 0x140004e7680)
2025/01/21 14:49:58 DEBUG : PUT /LICENSE?x-id=PutObject HTTP/1.1
Host: nvr-backup-340752817366.s3.ap-southeast-2.amazonaws.com
User-Agent: rclone/v1.69.0
Content-Length: 1074
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 131d297a-c0ba-4036-a70c-70d7ea1b92d6
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Md5: kfjxlCq+JVC2SnrPLEIELg==
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20250121T034958Z
X-Amz-Meta-Mtime: 1737086626.736314954
X-Amz-Security-Token: [redacted]
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:s3:::nvr-backup-340752817366

2025/01/21 14:49:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/21 14:49:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/21 14:49:58 DEBUG : HTTP RESPONSE (req 0x140004e7680)
2025/01/21 14:49:58 DEBUG : HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 21 Jan 2025 03:49:57 GMT
Server: AmazonS3
X-Amz-Id-2: dVxIhKwMc21FzYVrKyJe1GaJD3kJ4Ym3tYxs+AII7E06f4C0NslbJZnw/wcPyBC2c8QGyVYE55D6SWky7znsyg==
X-Amz-Request-Id: ZFW9QRBP1E3SGR78

2025/01/21 14:49:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/21 14:49:58 ERROR : LICENSE: Failed to copy: operation error S3: PutObject, https response error StatusCode: 400, RequestID: ZFW9QRBP1E3SGR78, HostID: dVxIhKwMc21FzYVrKyJe1GaJD3kJ4Ym3tYxs+AII7E06f4C0NslbJZnw/wcPyBC2c8QGyVYE55D6SWky7znsyg==, api error InvalidArgument: Server Side Encryption with AWS KMS managed key requires HTTP header x-amz-server-side-encryption : aws:kms
2025/01/21 14:49:58 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: PutObject, https response error StatusCode: 400, RequestID: ZFW9QRBP1E3SGR78, HostID: dVxIhKwMc21FzYVrKyJe1GaJD3kJ4Ym3tYxs+AII7E06f4C0NslbJZnw/wcPyBC2c8QGyVYE55D6SWky7znsyg==, api error InvalidArgument: Server Side Encryption with AWS KMS managed key requires HTTP header x-amz-server-side-encryption : aws:kms
2025/01/21 14:49:58 INFO  : 
Transferred:   	    1.049 KiB / 1.049 KiB, 100%, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.4s

2025/01/21 14:49:58 DEBUG : 5 go routines active
2025/01/21 14:49:58 NOTICE: Failed to copy: operation error S3: PutObject, https response error StatusCode: 400, RequestID: ZFW9QRBP1E3SGR78, HostID: dVxIhKwMc21FzYVrKyJe1GaJD3kJ4Ym3tYxs+AII7E06f4C0NslbJZnw/wcPyBC2c8QGyVYE55D6SWky7znsyg==, api error InvalidArgument: Server Side Encryption with AWS KMS managed key requires HTTP header x-amz-server-side-encryption : aws:kms

asdffdsa · January 21, 2025, 4:34pm

either it is an issue with the x-amz-server-side-encryption and how you setup the remote.
or
S3 backend updated to use AWS SDKv2 as v1 is now unsupported.
i would test an older version of rclone, such as v1.67.0

system · February 20, 2025, 4:34pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.