putBucket request on every object copy

What is the problem you are having with rclone?

When I try to copy a file to a remote S3 backend, rclone requires the access key to have permissions to create a new bucket, even when I'm uploading to an existing bucket.

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: centos 6.9 (64 bit)
  • os/kernel: 2.6.32-696.30.1.el6.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3 compatible API from Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy nothing.img test:rclonetest12321

The rclone config contents with secrets removed.

[test]
type = s3
provider = Minio
acl = private
access_key_id = xxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxx
endpoint = https://s3.us-west-002.backblazeb2.com

A log from the command with the -vv flag

2021/12/03 16:26:57 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "--log-file=log.txt" "--log-level=DEBUG" "--dump" "bodies,requests,responses" "nothing.img" "test:rclonetest12321"]
2021/12/03 16:26:57 DEBUG : Creating backend with remote "nothing.img"
2021/12/03 16:26:57 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/03 16:26:57 DEBUG : fs cache: adding new entry for parent of "nothing.img", "/root"
2021/12/03 16:26:57 DEBUG : Creating backend with remote "test:rclonetest12321"
2021/12/03 16:26:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/12/03 16:26:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HEAD /rclonetest12321/nothing.img HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z

2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : HTTP RESPONSE (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HTTP/1.1 404
Content-Length: 220
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: adfRuj2v0bkBvB3fdbkU=
X-Amz-Request-Id: 03fafe527292b397

2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : nothing.img: Need to transfer - File not found at Destination
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc00069e700)
2021/12/03 16:26:57 DEBUG : PUT /rclonetest12321 HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z
Accept-Encoding: gzip

2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : HTTP RESPONSE (req 0xc00069e700)
2021/12/03 16:26:57 DEBUG : HTTP/1.1 403
Content-Length: 139
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: aMnJmnmHDMVUzyjRaM8U4UDEoNXg4G2H9
X-Amz-Request-Id: 5b9d10e9ccf4ef56

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>not entitled</Message>
</Error>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 ERROR : nothing.img: Failed to copy: AccessDenied: not entitled
	status code: 403, request id: 5b9d10e9ccf4ef56, host id: aMnJmnmHDMVUzyjRaM8U4UDEoNXg4G2H9
2021/12/03 16:26:57 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: not entitled
	status code: 403, request id: 5b9d10e9ccf4ef56, host id: aMnJmnmHDMVUzyjRaM8U4UDEoNXg4G2H9
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HEAD /rclonetest12321/nothing.img HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z

2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : HTTP RESPONSE (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HTTP/1.1 404
Content-Length: 220
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: adTtuEGt3bk1vA3eRbr0=
X-Amz-Request-Id: a82bf2cb69a11855

2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : nothing.img: Need to transfer - File not found at Destination
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc0004a7300)
2021/12/03 16:26:57 DEBUG : PUT /rclonetest12321 HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z
Accept-Encoding: gzip

2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : HTTP RESPONSE (req 0xc0004a7300)
2021/12/03 16:26:58 DEBUG : HTTP/1.1 403
Content-Length: 139
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: aMpJmtmEWMV4zTDSIM8M4ozECNYw41WGO
X-Amz-Request-Id: eceaf969fff4e5bc

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>not entitled</Message>
</Error>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 ERROR : nothing.img: Failed to copy: AccessDenied: not entitled
	status code: 403, request id: eceaf969fff4e5bc, host id: aMpJmtmEWMV4zTDSIM8M4ozECNYw41WGO
2021/12/03 16:26:58 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: not entitled
	status code: 403, request id: eceaf969fff4e5bc, host id: aMpJmtmEWMV4zTDSIM8M4ozECNYw41WGO
2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : HTTP REQUEST (req 0xc0003f7700)
2021/12/03 16:26:58 DEBUG : HEAD /rclonetest12321/nothing.img HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192658Z

2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : HTTP RESPONSE (req 0xc0003f7700)
2021/12/03 16:26:58 DEBUG : HTTP/1.1 404
Content-Length: 220
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: adaRugGsZbplvKHejblE=
X-Amz-Request-Id: 2209262efe1219a4

2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : nothing.img: Need to transfer - File not found at Destination
2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : HTTP REQUEST (req 0xc00069f000)
2021/12/03 16:26:58 DEBUG : PUT /rclonetest12321 HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192658Z
Accept-Encoding: gzip

2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : HTTP RESPONSE (req 0xc00069f000)
2021/12/03 16:26:58 DEBUG : HTTP/1.1 403
Content-Length: 139
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
X-Amz-Request-Id: 9e6268d3366f2c9f

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>not entitled</Message>
</Error>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 ERROR : nothing.img: Failed to copy: AccessDenied: not entitled
	status code: 403, request id: 9e6268d3366f2c9f, host id: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
2021/12/03 16:26:58 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: not entitled
	status code: 403, request id: 9e6268d3366f2c9f, host id: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
2021/12/03 16:26:58 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.9s

2021/12/03 16:26:58 DEBUG : 5 go routines active
2021/12/03 16:26:58 Failed to copy: AccessDenied: not entitled
	status code: 403, request id: 9e6268d3366f2c9f, host id: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi

When I use other tools to copy files with the same credentials, I have no problems.

If I allow the credentials to create new buckets, it receives a 409 Conflict saying the bucket already exists and the file is successfully copied.

Why does rclone requires the create bucket permission when I'm uploading to an existing bucket? Is there any way to change that behavior?

hi,
for my s3 buckets,

  • always pre-create the buckets.
  • locked down bucket polices.
  • use MFA session tokens.

so i use --s3-no-check-bucket

note: b2 is not fully s3 compliant.

Hi Jojo,

I know B2 is not fully s3 compliant, but since other tools had no problem with my credentials while using their S3 compatible API, the problem seemed to be related to rclone's behavior, which indeed it was. The --s3-no-check-bucket changes that behavior to precisely what I was looking for. Thanks for pointing that out!

Kind regards

there is a back story with rclone and the need for --s3-no-check-bucket.

right now, i cannot remember the exact details.

tho it is friday afternoon, and soon i will be able to think clearly with a few guinness extra stout in me. :beers:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.