Required s3 permissions for `--backup-dir`

What is the problem you are having with rclone?

as documented here, these permissions do not work with sync --backup-dir.
the initital sync works, including creating the subfolders.
when rclone needs to create the folder for --backup-dir, wasabi complains AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
but no bucket is being created, just a sub-folder.
i added s3:CreateBucket but still get accessed denied

When using the sync subcommand of rclone the following minimum permissions are required to be available on the bucket being written to:

    ListBucket
    DeleteObject
    GetObject
    PutObject
    PutObjectACL

What is your rclone version (output from rclone version)

v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

W10.20H2.64

Which cloud storage system are you using? (eg Google Drive)

wasabi - s3 clone

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe sync C:\data\rclone\scripts\new\test wasabi_en07_useren07:en07/newtest/backup/ --backup-dir=wasabi_en07_useren07:en07/newtest/archive/111111111.111111 -vv --dump=bodies --retries=1 --low-level-retries=1 --log-file=rclone.log

The rclone config contents with secrets removed.

[wasabi_en07_useren07]
type = s3
provider = Wasabi
access_key_id = 
secret_access_key = 
endpoint = s3.us-east-2.wasabisys.com

A log from the command with the -vv flag

DEBUG : Using RCLONE_CONFIG_PASS password.
DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.55.1" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "sync" "C:\\data\\rclone\\scripts\\new\\test" "wasabi_en07_useren07:en07/newtest/backup/" "--backup-dir=wasabi_en07_useren07:en07/newtest/archive/111111111.111111" "-vv" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-file=rclone.log"]
DEBUG : Creating backend with remote "C:\\data\\rclone\\scripts\\new\\test"
DEBUG : fs cache: renaming cache item "C:\\data\\rclone\\scripts\\new\\test" to be canonical "//?/C:/data/rclone/scripts/new/test"
DEBUG : Creating backend with remote "wasabi_en07_useren07:en07/newtest/backup/"
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000695100)
DEBUG : HEAD /en07/newtest/backup HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204139Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000695100)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Id-2: 0GEkuVJowf9kl7IF47NvPrBS1MYUMkQkS6vtTwqJk2hOqgSuFOoCl9Ws4ipJvMPaib2pEBd5I80r
X-Amz-Request-Id: 35A653229FFC782A

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : fs cache: renaming cache item "wasabi_en07_useren07:en07/newtest/backup/" to be canonical "wasabi_en07_useren07:en07/newtest/backup"
DEBUG : Creating backend with remote "wasabi_en07_useren07:en07/newtest/archive/111111111.111111"
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000216900)
DEBUG : HEAD /en07/newtest/archive/111111111.111111 HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000216900)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head4)
X-Amz-Id-2: xKgf43z7UcTQ9syurYyuQE8p6yeeYs3NEGGcSYNsthSFODluvX5ixcEMV+Scapuzc6T9BIdur5Iz
X-Amz-Request-Id: DDC587F4AEAB6FD2

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000216300)
DEBUG : GET /en07?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=newtest%2Fbackup%2F HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000216300)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: 3TTrDQLRyRDLpjfYKCYU0AIVQ8wqllzZOzG0F05lv3YRWrZ99Ui7uYoBh1xu4h2x/MESZokXjSso
X-Amz-Request-Id: DCEFCFC4EBF94444

1b5
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>en07</Name><Prefix>newtest/backup/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>newtest/backup/01/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>newtest/backup/02/</Prefix></CommonPrefixes></ListBucketResult>
0

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0000c3600)
DEBUG : GET /en07?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=newtest%2Fbackup%2F02%2F HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000694400)
DEBUG : GET /en07?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=newtest%2Fbackup%2F01%2F HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0000c3600)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: qaWtXlx6MSJC2T/PGKheaiHbhsCxRQEnQ9/A7dCKi3Ti4f9/DbY79lTid/c7Rn9TcGuxRt9hkCSU
X-Amz-Request-Id: 4CB8BCB8DAF31C64

285
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>en07</Name><Prefix>newtest/backup/02/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>newtest/backup/02/02.txt</Key><LastModified>2021-05-18T20:11:26.000Z</LastModified><ETag>&quot;6512bd43d9caa6e02c990b0a82652dca&quot;</ETag><Size>2</Size><Owner><ID>FE01BF6C3155146B8F93018D9C4338B3232AB1D27066FA8472A2BC6BF1C3F51D</ID><DisplayName>yoyomeltz</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
0

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : 02/02.txt: Sizes differ (src 3 vs dst 2)
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000216a00)
DEBUG : HEAD /en07/newtest/archive/111111111.111111/02/02.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000694400)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: dmoZ5wutkPiC3lhg+ZxPkWHGO6dV70Gq6Y8C2D6+nYeE9PvduxEoPDVBMx5Rjcx+pfCi2mbpPvD1
X-Amz-Request-Id: C2CD70060F2B9710

28a
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>en07</Name><Prefix>newtest/backup/01/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>newtest/backup/01/01.txt</Key><LastModified>2021-05-18T20:11:26.000Z</LastModified><ETag>&quot;614dc937dae234e2cb063a1236ebccbe&quot;</ETag><Size>396702</Size><Owner><ID>FE01BF6C3155146B8F93018D9C4338B3232AB1D27066FA8472A2BC6BF1C3F51D</ID><DisplayName>yoyomeltz</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
0

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : S3 bucket en07 path newtest/backup: Waiting for checks to finish
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000694f00)
DEBUG : HEAD /en07/newtest/backup/01/01.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000216a00)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head4)
X-Amz-Id-2: w5hY3nuKg615S3hbgHceBINbkGq4z3dpJh/TGiFzwtmFDykGx5w3rgI8b5Kv6J6k3SMJT2294e2W
X-Amz-Request-Id: E0DFE9130E0B3C09

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00041b600)
DEBUG : PUT /en07 HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000694f00)
DEBUG : HTTP/1.1 200 OK
Content-Length: 396702
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Tue, 18 May 2021 20:41:40 GMT
Etag: "614dc937dae234e2cb063a1236ebccbe"
Last-Modified: Tue, 18 May 2021 20:11:26 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Id-2: Del4pix4ruDMF7ybbiBRCait9yCJQznZDn4oLurT5o2+4At8USgKZs7bnBgiciQWcv5jnHi3qEji
X-Amz-Meta-Mtime: 1619131413.9853072
X-Amz-Request-Id: B9817E1B2F5501EB

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : 01/01.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
DEBUG : 01/01.txt: Unchanged skipping
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00041b600)
DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head4)
X-Amz-Id-2: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
X-Amz-Request-Id: 6052E611FF47AE0D

161
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07</Message><RequestId>6052E611FF47AE0D</RequestId><HostId>QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc</HostId></Error>
0

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
ERROR : 02/02.txt: Failed to copy: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
	status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
ERROR : 02/02.txt: Not deleting source as copy failed: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
	status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
DEBUG : S3 bucket en07 path newtest/backup: Waiting for transfers to finish
ERROR : S3 bucket en07 path newtest/backup: not deleting files as there were IO errors
ERROR : S3 bucket en07 path newtest/backup: not deleting directories as there were IO errors
ERROR : Attempt 1/1 failed with 1 errors and: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
	status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         0.3s

DEBUG : 8 go routines active
Failed to sync: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
	status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc

and here is the s3 policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::100000065159:user/user.en07"
      },
      "Action": [
        "s3:CreateBucket",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::en07/*",
        "arn:aws:s3:::en07"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "arn:aws:s3:::en07"
    }
  ]
}

I suspect adding the flag (or config parameter)

  --s3-no-check-bucket     If set, don't attempt to check the bucket exists or create it

Will fix it..

--s3-no-check-bucket does workaround the error.

i guess there is some logic i do not understand.

--backup-dir only works within the same bucket as the main dir.
once rclone decides it needs to move a file into the backup dir, rclone should know that the bucket exists.
so there is no need for the PUT /en07?

There is certainly something whacky going on!

In your trace above you can see rclone listing the /en07 bucket successfully. Rclone should set its "bucket exists" flag at that point.

However what is actually happening is that rclone is using a different remote for the backup-dir - and this remote doesn't have the knowledge that the bucket exists.

So that is why it happens...

This problem is suprisingly common. I think I might get rclone to always check the bucket exists (by listing 1 file out of it) before trying to create it. This will fix the problem at the expense of 1 more transaction on bucket create which isn't very common.

I did have the idea of making a Chroot call for rclone backends which would make --backup-dir more efficient and fix this problem too.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.