What is the problem you are having with rclone?
as documented here, these permissions do not work with sync --backup-dir
.
the initital sync works, including creating the subfolders.
when rclone needs to create the folder for --backup-dir
, wasabi complains AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
but no bucket is being created, just a sub-folder.
i added s3:CreateBucket
but still get accessed denied
When using the sync subcommand of rclone the following minimum permissions are required to be available on the bucket being written to:
ListBucket
DeleteObject
GetObject
PutObject
PutObjectACL
What is your rclone version (output from rclone version
)
v1.55.1
Which OS you are using and how many bits (eg Windows 7, 64 bit)
W10.20H2.64
Which cloud storage system are you using? (eg Google Drive)
wasabi - s3 clone
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone.exe sync C:\data\rclone\scripts\new\test wasabi_en07_useren07:en07/newtest/backup/ --backup-dir=wasabi_en07_useren07:en07/newtest/archive/111111111.111111 -vv --dump=bodies --retries=1 --low-level-retries=1 --log-file=rclone.log
The rclone config contents with secrets removed.
[wasabi_en07_useren07]
type = s3
provider = Wasabi
access_key_id =
secret_access_key =
endpoint = s3.us-east-2.wasabisys.com
A log from the command with the -vv
flag
DEBUG : Using RCLONE_CONFIG_PASS password.
DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.55.1" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "sync" "C:\\data\\rclone\\scripts\\new\\test" "wasabi_en07_useren07:en07/newtest/backup/" "--backup-dir=wasabi_en07_useren07:en07/newtest/archive/111111111.111111" "-vv" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-file=rclone.log"]
DEBUG : Creating backend with remote "C:\\data\\rclone\\scripts\\new\\test"
DEBUG : fs cache: renaming cache item "C:\\data\\rclone\\scripts\\new\\test" to be canonical "//?/C:/data/rclone/scripts/new/test"
DEBUG : Creating backend with remote "wasabi_en07_useren07:en07/newtest/backup/"
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000695100)
DEBUG : HEAD /en07/newtest/backup HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204139Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000695100)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Id-2: 0GEkuVJowf9kl7IF47NvPrBS1MYUMkQkS6vtTwqJk2hOqgSuFOoCl9Ws4ipJvMPaib2pEBd5I80r
X-Amz-Request-Id: 35A653229FFC782A
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : fs cache: renaming cache item "wasabi_en07_useren07:en07/newtest/backup/" to be canonical "wasabi_en07_useren07:en07/newtest/backup"
DEBUG : Creating backend with remote "wasabi_en07_useren07:en07/newtest/archive/111111111.111111"
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000216900)
DEBUG : HEAD /en07/newtest/archive/111111111.111111 HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000216900)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head4)
X-Amz-Id-2: xKgf43z7UcTQ9syurYyuQE8p6yeeYs3NEGGcSYNsthSFODluvX5ixcEMV+Scapuzc6T9BIdur5Iz
X-Amz-Request-Id: DDC587F4AEAB6FD2
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000216300)
DEBUG : GET /en07?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=newtest%2Fbackup%2F HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000216300)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: 3TTrDQLRyRDLpjfYKCYU0AIVQ8wqllzZOzG0F05lv3YRWrZ99Ui7uYoBh1xu4h2x/MESZokXjSso
X-Amz-Request-Id: DCEFCFC4EBF94444
1b5
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>en07</Name><Prefix>newtest/backup/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>newtest/backup/01/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>newtest/backup/02/</Prefix></CommonPrefixes></ListBucketResult>
0
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0000c3600)
DEBUG : GET /en07?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=newtest%2Fbackup%2F02%2F HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000694400)
DEBUG : GET /en07?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=newtest%2Fbackup%2F01%2F HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0000c3600)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: qaWtXlx6MSJC2T/PGKheaiHbhsCxRQEnQ9/A7dCKi3Ti4f9/DbY79lTid/c7Rn9TcGuxRt9hkCSU
X-Amz-Request-Id: 4CB8BCB8DAF31C64
285
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>en07</Name><Prefix>newtest/backup/02/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>newtest/backup/02/02.txt</Key><LastModified>2021-05-18T20:11:26.000Z</LastModified><ETag>"6512bd43d9caa6e02c990b0a82652dca"</ETag><Size>2</Size><Owner><ID>FE01BF6C3155146B8F93018D9C4338B3232AB1D27066FA8472A2BC6BF1C3F51D</ID><DisplayName>yoyomeltz</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
0
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : 02/02.txt: Sizes differ (src 3 vs dst 2)
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000216a00)
DEBUG : HEAD /en07/newtest/archive/111111111.111111/02/02.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000694400)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: dmoZ5wutkPiC3lhg+ZxPkWHGO6dV70Gq6Y8C2D6+nYeE9PvduxEoPDVBMx5Rjcx+pfCi2mbpPvD1
X-Amz-Request-Id: C2CD70060F2B9710
28a
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>en07</Name><Prefix>newtest/backup/01/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>newtest/backup/01/01.txt</Key><LastModified>2021-05-18T20:11:26.000Z</LastModified><ETag>"614dc937dae234e2cb063a1236ebccbe"</ETag><Size>396702</Size><Owner><ID>FE01BF6C3155146B8F93018D9C4338B3232AB1D27066FA8472A2BC6BF1C3F51D</ID><DisplayName>yoyomeltz</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
0
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : S3 bucket en07 path newtest/backup: Waiting for checks to finish
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000694f00)
DEBUG : HEAD /en07/newtest/backup/01/01.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000216a00)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head4)
X-Amz-Id-2: w5hY3nuKg615S3hbgHceBINbkGq4z3dpJh/TGiFzwtmFDykGx5w3rgI8b5Kv6J6k3SMJT2294e2W
X-Amz-Request-Id: E0DFE9130E0B3C09
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00041b600)
DEBUG : PUT /en07 HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.55.1
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210518T204140Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000694f00)
DEBUG : HTTP/1.1 200 OK
Content-Length: 396702
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Tue, 18 May 2021 20:41:40 GMT
Etag: "614dc937dae234e2cb063a1236ebccbe"
Last-Modified: Tue, 18 May 2021 20:11:26 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head1)
X-Amz-Id-2: Del4pix4ruDMF7ybbiBRCait9yCJQznZDn4oLurT5o2+4At8USgKZs7bnBgiciQWcv5jnHi3qEji
X-Amz-Meta-Mtime: 1619131413.9853072
X-Amz-Request-Id: B9817E1B2F5501EB
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : 01/01.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
DEBUG : 01/01.txt: Unchanged skipping
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00041b600)
DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 18 May 2021 20:41:40 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head4)
X-Amz-Id-2: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
X-Amz-Request-Id: 6052E611FF47AE0D
161
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07</Message><RequestId>6052E611FF47AE0D</RequestId><HostId>QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc</HostId></Error>
0
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
ERROR : 02/02.txt: Failed to copy: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
ERROR : 02/02.txt: Not deleting source as copy failed: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
DEBUG : S3 bucket en07 path newtest/backup: Waiting for transfers to finish
ERROR : S3 bucket en07 path newtest/backup: not deleting files as there were IO errors
ERROR : S3 bucket en07 path newtest/backup: not deleting directories as there were IO errors
ERROR : Attempt 1/1 failed with 1 errors and: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Checks: 3 / 3, 100%
Elapsed time: 0.3s
DEBUG : 8 go routines active
Failed to sync: AccessDenied: User: arn:aws:iam::100000065159:user/user.en07 is not authorized to perform: s3:CreateBucket on resource: arn:aws:s3:::en07
status code: 403, request id: 6052E611FF47AE0D, host id: QP/0M70iCl28kupThwZsELWUs7UNMCUWW4BH/zsLlkylqqoTJE0I+vXCuLU5JFL6DQtU532RH5mc
and here is the s3 policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::100000065159:user/user.en07"
},
"Action": [
"s3:CreateBucket",
"s3:ListBucket",
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::en07/*",
"arn:aws:s3:::en07"
]
},
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::en07"
}
]
}