What is the problem you are having with rclone?
Hello! I'm trying to configure my AWS S3 backend. I have gotten cat
to work, but I haven't been able to get copy
nor sync
to work; both result in AccessDenied
.
Any help would be appreciated. Thanks!
What is your rclone version (output from rclone version
)
Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone --version
rclone v1.56.0
- os/version: darwin 11.6 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.6
- go/linking: dynamic
- go/tags: cmount
uname -a
Darwin ajf-ops.lan 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
This works:
rclone --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/hello.html
<html>
<head>
<title>Hello!</title>
</head>
<body>
Hello!
</body>
But this fails:
rclone --config ~/.rclone --s3-profile staging-ops sync test.html staging_s3:/sco-assets-v1/test1
2021/09/14 21:57:17 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBXPG33ZGV0DMKC</RequestId><HostId>FA47KESoWf6270Srostz5XZztxaBK4IbPgikrvOT0bBhAEAJfKKPr4MDQhhnlgKcADh0YTvi2s4=</HostId></Error>
2021/09/14 21:57:17 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBXPG33ZGV0DMKC</RequestId><HostId>FA47KESoWf6270Srostz5XZztxaBK4IbPgikrvOT0bBhAEAJfKKPr4MDQhhnlgKcADh0YTvi2s4=</HostId></Error>
2021/09/14 21:57:17 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBV9QK4R8HR4BZ8</RequestId><HostId>0ldsaFAVKPmzj9t6CrgwhhJiobISaeyCl9Uv7Z4KVLZxR6LtDOqzAn8qDJjFOHIgAFRD5QB5kT4=</HostId></Error>
2021/09/14 21:57:17 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBV9QK4R8HR4BZ8</RequestId><HostId>0ldsaFAVKPmzj9t6CrgwhhJiobISaeyCl9Uv7Z4KVLZxR6LtDOqzAn8qDJjFOHIgAFRD5QB5kT4=</HostId></Error>
2021/09/14 21:57:18 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2AF0Z9NR3C77VE06</RequestId><HostId>VDGfu56QlyVN5oyhK8wxRCtTHX69rWKRHDXb+3jI+ykPRmsKlgM5pRAQnUbX/RIG1SDHuvJ9UeA=</HostId></Error>
2021/09/14 21:57:18 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2AF0Z9NR3C77VE06</RequestId><HostId>VDGfu56QlyVN5oyhK8wxRCtTHX69rWKRHDXb+3jI+ykPRmsKlgM5pRAQnUbX/RIG1SDHuvJ9UeA=</HostId></Error>
2021/09/14 21:57:18 Failed to sync: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2AF0Z9NR3C77VE06</RequestId><HostId>VDGfu56QlyVN5oyhK8wxRCtTHX69rWKRHDXb+3jI+ykPRmsKlgM5pRAQnUbX/RIG1SDHuvJ9UeA=</HostId></Error>
The rclone config contents with secrets removed.
[staging_gdrive]
type = drive
client_id = xxx.apps.googleusercontent.com
client_secret = yyy
scope = drive
service_account_file = /Users/me/Downloads/foo.json
[staging_s3]
type = s3
provider = AWS
env_auth = true
region = us-east-1
acl = authenticated-read
server_side_encryption = AES256
no_check_bucket = true
(Tried it both with and without no_check_bucket
, failed both ways)
The staging-ops
AWS identity has the managd AdministratorAccess role, which is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
A log from the command with the -vv
flag
2021/09/14 22:18:57 DEBUG : rclone: Version "v1.56.0" starting with parameters ["Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone" "-vv" "--config" "/Users/anthony_foiani/.rclone" "--s3-profile" "staging-ops" "sync" "test.html" "staging_s3:/sco-assets-v1/test1"]
2021/09/14 22:18:57 DEBUG : Creating backend with remote "test.html"
2021/09/14 22:18:57 DEBUG : Using config file from "/Users/anthony_foiani/.rclone"
2021/09/14 22:18:57 DEBUG : fs cache: adding new entry for parent of "test.html", "/Users/anthony_foiani"
2021/09/14 22:18:57 DEBUG : Creating backend with remote "staging_s3:/sco-assets-v1/test1"
2021/09/14 22:18:57 DEBUG : staging_s3: detected overridden config - adding "{V5-qR}" suffix to name
2021/09/14 22:18:57 DEBUG : fs cache: renaming cache item "staging_s3:/sco-assets-v1/test1" to be canonical "staging_s3{V5-qR}:sco-assets-v1/test1"
2021/09/14 22:18:58 DEBUG : test.html: Need to transfer - File not found at Destination
2021/09/14 22:18:58 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5D6RCYAP0GA83V</RequestId><HostId>9dRxexxtEJ1irosxXDOe2yfm5sZBQYFwzsG90eWN3h46JmSyc/TeDgHlNWmsmL6FGc0M0x7hrF+FPJJqm9zJVQ==</HostId></Error>
2021/09/14 22:18:58 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5D6RCYAP0GA83V</RequestId><HostId>9dRxexxtEJ1irosxXDOe2yfm5sZBQYFwzsG90eWN3h46JmSyc/TeDgHlNWmsmL6FGc0M0x7hrF+FPJJqm9zJVQ==</HostId></Error>
2021/09/14 22:18:58 DEBUG : test.html: Need to transfer - File not found at Destination
2021/09/14 22:18:58 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5ENM8XMC7H53K0</RequestId><HostId>SNQ5GxgFztfkeD5frq6Lnd2cFgf1xcc823dqx6wQ6+D3nedSfGDBRXm55437Sp9ERjp1LeZuTwDi//igrHN0Jw==</HostId></Error>
2021/09/14 22:18:58 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5ENM8XMC7H53K0</RequestId><HostId>SNQ5GxgFztfkeD5frq6Lnd2cFgf1xcc823dqx6wQ6+D3nedSfGDBRXm55437Sp9ERjp1LeZuTwDi//igrHN0Jw==</HostId></Error>
2021/09/14 22:18:58 DEBUG : test.html: Need to transfer - File not found at Destination
2021/09/14 22:18:58 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W53KV38W9VQ21T3</RequestId><HostId>n9zC9R4scfLn3L8AZclQe5msf+ngmiLFmX5puDKcq6RrXsdvo35dZK8lquVCqbMc4lAvccxh4HmV16Q9/kXdHA==</HostId></Error>
2021/09/14 22:18:58 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W53KV38W9VQ21T3</RequestId><HostId>n9zC9R4scfLn3L8AZclQe5msf+ngmiLFmX5puDKcq6RrXsdvo35dZK8lquVCqbMc4lAvccxh4HmV16Q9/kXdHA==</HostId></Error>
2021/09/14 22:18:58 INFO :
Transferred: 330 / 330 Byte, 100%, 0 Byte/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 1.7s
2021/09/14 22:18:58 DEBUG : 5 go routines active
2021/09/14 22:18:58 Failed to sync: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W53KV38W9VQ21T3</RequestId><HostId>n9zC9R4scfLn3L8AZclQe5msf+ngmiLFmX5puDKcq6RrXsdvo35dZK8lquVCqbMc4lAvccxh4HmV16Q9/kXdHA==</HostId></Error>
Adding in the --dump bodies
option, I see that it's trying to do a PUT / HTTP/1.1
(!), and that gives the failure:
2021/09/14 20:58:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 20:58:40 DEBUG : HTTP REQUEST (req 0xc000b55200)
2021/09/14 20:58:40 DEBUG : PUT / HTTP/1.1
Host: sco-assets-v1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.56.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: authenticated-read
X-Amz-Content-Sha256: xxx
X-Amz-Date: 20210914T205840Z
Accept-Encoding: gzip
2021/09/14 20:58:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 20:58:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 20:58:41 DEBUG : HTTP RESPONSE (req 0xc000b55200)
2021/09/14 20:58:41 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 14 Sep 2021 20:58:40 GMT
Server: AmazonS3
X-Amz-Id-2: xxx
X-Amz-Request-Id: xxx
f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>xxx</RequestId><HostId>Fz8dEbryifXPVKKIKb18Ah1oBtBCZSUSw/nlC3WMFP7/LMrYLtEUtcaiGX4gnqM21U+KmcXDhIU=</HostId></Error>
0
2021/09/14 20:58:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
It seems to be failing on CreateBucket
:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "xxx",
"arn": "arn:aws:iam::xxx:user/blah",
"accountId": "xxx",
"accessKeyId": "yyy",
"userName": "anthony_foiani_ops"
},
"eventTime": "2021-09-14T21:33:09Z",
"eventSource": "s3.amazonaws.com",
"eventName": "CreateBucket",
"awsRegion": "us-east-1",
"sourceIPAddress": "w.x.y.z",
"userAgent": "[rclone/v1.56.0]",
"errorCode": "AccessDenied",
"errorMessage": "Access Denied",
"requestParameters": {
"bucketName": "sco-assets-v1",
"Host": "sco-assets-v1.s3.us-east-1.amazonaws.com",
"x-amz-acl": "authenticated-read"
},
"responseElements": null,
"additionalEventData": {
"SignatureVersion": "SigV4",
"CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"bytesTransferredIn": 0,
"AuthenticationMethod": "AuthHeader",
"x-amz-id-2": "foo",
"bytesTransferredOut": 243
},
"requestID": "bar",
"eventID": "baz",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "xxx",
"eventCategory": "Management"
}