What is the problem you are having with rclone?
When I try to copy a file to a remote S3 backend, rclone requires the access key to have permissions to create a new bucket, even when I'm uploading to an existing bucket.
What is your rclone version (output from rclone version
)
rclone v1.57.0
- os/version: centos 6.9 (64 bit)
- os/kernel: 2.6.32-696.30.1.el6.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3 compatible API from Backblaze B2
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy nothing.img test:rclonetest12321
The rclone config contents with secrets removed.
[test]
type = s3
provider = Minio
acl = private
access_key_id = xxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxx
endpoint = https://s3.us-west-002.backblazeb2.com
A log from the command with the -vv
flag
2021/12/03 16:26:57 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "--log-file=log.txt" "--log-level=DEBUG" "--dump" "bodies,requests,responses" "nothing.img" "test:rclonetest12321"]
2021/12/03 16:26:57 DEBUG : Creating backend with remote "nothing.img"
2021/12/03 16:26:57 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/03 16:26:57 DEBUG : fs cache: adding new entry for parent of "nothing.img", "/root"
2021/12/03 16:26:57 DEBUG : Creating backend with remote "test:rclonetest12321"
2021/12/03 16:26:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/12/03 16:26:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HEAD /rclonetest12321/nothing.img HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : HTTP RESPONSE (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HTTP/1.1 404
Content-Length: 220
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: adfRuj2v0bkBvB3fdbkU=
X-Amz-Request-Id: 03fafe527292b397
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : nothing.img: Need to transfer - File not found at Destination
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc00069e700)
2021/12/03 16:26:57 DEBUG : PUT /rclonetest12321 HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z
Accept-Encoding: gzip
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : HTTP RESPONSE (req 0xc00069e700)
2021/12/03 16:26:57 DEBUG : HTTP/1.1 403
Content-Length: 139
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: aMnJmnmHDMVUzyjRaM8U4UDEoNXg4G2H9
X-Amz-Request-Id: 5b9d10e9ccf4ef56
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
<Code>AccessDenied</Code>
<Message>not entitled</Message>
</Error>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 ERROR : nothing.img: Failed to copy: AccessDenied: not entitled
status code: 403, request id: 5b9d10e9ccf4ef56, host id: aMnJmnmHDMVUzyjRaM8U4UDEoNXg4G2H9
2021/12/03 16:26:57 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: not entitled
status code: 403, request id: 5b9d10e9ccf4ef56, host id: aMnJmnmHDMVUzyjRaM8U4UDEoNXg4G2H9
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HEAD /rclonetest12321/nothing.img HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : HTTP RESPONSE (req 0xc00069e400)
2021/12/03 16:26:57 DEBUG : HTTP/1.1 404
Content-Length: 220
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: adTtuEGt3bk1vA3eRbr0=
X-Amz-Request-Id: a82bf2cb69a11855
2021/12/03 16:26:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:57 DEBUG : nothing.img: Need to transfer - File not found at Destination
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:57 DEBUG : HTTP REQUEST (req 0xc0004a7300)
2021/12/03 16:26:57 DEBUG : PUT /rclonetest12321 HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192657Z
Accept-Encoding: gzip
2021/12/03 16:26:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : HTTP RESPONSE (req 0xc0004a7300)
2021/12/03 16:26:58 DEBUG : HTTP/1.1 403
Content-Length: 139
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: aMpJmtmEWMV4zTDSIM8M4ozECNYw41WGO
X-Amz-Request-Id: eceaf969fff4e5bc
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
<Code>AccessDenied</Code>
<Message>not entitled</Message>
</Error>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 ERROR : nothing.img: Failed to copy: AccessDenied: not entitled
status code: 403, request id: eceaf969fff4e5bc, host id: aMpJmtmEWMV4zTDSIM8M4ozECNYw41WGO
2021/12/03 16:26:58 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: not entitled
status code: 403, request id: eceaf969fff4e5bc, host id: aMpJmtmEWMV4zTDSIM8M4ozECNYw41WGO
2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : HTTP REQUEST (req 0xc0003f7700)
2021/12/03 16:26:58 DEBUG : HEAD /rclonetest12321/nothing.img HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192658Z
2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : HTTP RESPONSE (req 0xc0003f7700)
2021/12/03 16:26:58 DEBUG : HTTP/1.1 404
Content-Length: 220
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: adaRugGsZbplvKHejblE=
X-Amz-Request-Id: 2209262efe1219a4
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : nothing.img: Need to transfer - File not found at Destination
2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : HTTP REQUEST (req 0xc00069f000)
2021/12/03 16:26:58 DEBUG : PUT /rclonetest12321 HTTP/1.1
Host: s3.us-west-002.backblazeb2.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211203T192658Z
Accept-Encoding: gzip
2021/12/03 16:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 DEBUG : HTTP RESPONSE (req 0xc00069f000)
2021/12/03 16:26:58 DEBUG : HTTP/1.1 403
Content-Length: 139
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Fri, 03 Dec 2021 19:26:57 GMT
X-Amz-Id-2: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
X-Amz-Request-Id: 9e6268d3366f2c9f
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
<Code>AccessDenied</Code>
<Message>not entitled</Message>
</Error>
2021/12/03 16:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/12/03 16:26:58 ERROR : nothing.img: Failed to copy: AccessDenied: not entitled
status code: 403, request id: 9e6268d3366f2c9f, host id: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
2021/12/03 16:26:58 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: not entitled
status code: 403, request id: 9e6268d3366f2c9f, host id: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
2021/12/03 16:26:58 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.9s
2021/12/03 16:26:58 DEBUG : 5 go routines active
2021/12/03 16:26:58 Failed to copy: AccessDenied: not entitled
status code: 403, request id: 9e6268d3366f2c9f, host id: aMl1ma2GEMRUzPDSoMy44qjEJNdE4bmFi
When I use other tools to copy files with the same credentials, I have no problems.
If I allow the credentials to create new buckets, it receives a 409 Conflict saying the bucket already exists and the file is successfully copied.
Why does rclone requires the create bucket permission when I'm uploading to an existing bucket? Is there any way to change that behavior?