AWS S3: Can `cat` but not `copy`/`sync`?

What is the problem you are having with rclone?

Hello! I'm trying to configure my AWS S3 backend. I have gotten cat to work, but I haven't been able to get copy nor sync to work; both result in AccessDenied.

Any help would be appreciated. Thanks!

What is your rclone version (output from rclone version)

Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone --version
rclone v1.56.0
- os/version: darwin 11.6 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.6
- go/linking: dynamic
- go/tags: cmount

uname -a
Darwin ajf-ops.lan 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This works:

rclone --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/hello.html
<html>
  <head>
    <title>Hello!</title>
  </head>
  <body>
    Hello!
  </body>

But this fails:

rclone --config ~/.rclone --s3-profile staging-ops sync test.html staging_s3:/sco-assets-v1/test1
2021/09/14 21:57:17 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBXPG33ZGV0DMKC</RequestId><HostId>FA47KESoWf6270Srostz5XZztxaBK4IbPgikrvOT0bBhAEAJfKKPr4MDQhhnlgKcADh0YTvi2s4=</HostId></Error>
2021/09/14 21:57:17 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBXPG33ZGV0DMKC</RequestId><HostId>FA47KESoWf6270Srostz5XZztxaBK4IbPgikrvOT0bBhAEAJfKKPr4MDQhhnlgKcADh0YTvi2s4=</HostId></Error>
2021/09/14 21:57:17 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBV9QK4R8HR4BZ8</RequestId><HostId>0ldsaFAVKPmzj9t6CrgwhhJiobISaeyCl9Uv7Z4KVLZxR6LtDOqzAn8qDJjFOHIgAFRD5QB5kT4=</HostId></Error>
2021/09/14 21:57:17 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>MVBV9QK4R8HR4BZ8</RequestId><HostId>0ldsaFAVKPmzj9t6CrgwhhJiobISaeyCl9Uv7Z4KVLZxR6LtDOqzAn8qDJjFOHIgAFRD5QB5kT4=</HostId></Error>
2021/09/14 21:57:18 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2AF0Z9NR3C77VE06</RequestId><HostId>VDGfu56QlyVN5oyhK8wxRCtTHX69rWKRHDXb+3jI+ykPRmsKlgM5pRAQnUbX/RIG1SDHuvJ9UeA=</HostId></Error>
2021/09/14 21:57:18 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2AF0Z9NR3C77VE06</RequestId><HostId>VDGfu56QlyVN5oyhK8wxRCtTHX69rWKRHDXb+3jI+ykPRmsKlgM5pRAQnUbX/RIG1SDHuvJ9UeA=</HostId></Error>
2021/09/14 21:57:18 Failed to sync: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2AF0Z9NR3C77VE06</RequestId><HostId>VDGfu56QlyVN5oyhK8wxRCtTHX69rWKRHDXb+3jI+ykPRmsKlgM5pRAQnUbX/RIG1SDHuvJ9UeA=</HostId></Error>

The rclone config contents with secrets removed.

[staging_gdrive]
type = drive
client_id = xxx.apps.googleusercontent.com
client_secret = yyy
scope = drive
service_account_file = /Users/me/Downloads/foo.json

[staging_s3]
type = s3
provider = AWS
env_auth = true
region = us-east-1
acl = authenticated-read
server_side_encryption = AES256
no_check_bucket = true

(Tried it both with and without no_check_bucket, failed both ways)

The staging-ops AWS identity has the managd AdministratorAccess role, which is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

A log from the command with the -vv flag

2021/09/14 22:18:57 DEBUG : rclone: Version "v1.56.0" starting with parameters ["Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone" "-vv" "--config" "/Users/anthony_foiani/.rclone" "--s3-profile" "staging-ops" "sync" "test.html" "staging_s3:/sco-assets-v1/test1"]
2021/09/14 22:18:57 DEBUG : Creating backend with remote "test.html"
2021/09/14 22:18:57 DEBUG : Using config file from "/Users/anthony_foiani/.rclone"
2021/09/14 22:18:57 DEBUG : fs cache: adding new entry for parent of "test.html", "/Users/anthony_foiani"
2021/09/14 22:18:57 DEBUG : Creating backend with remote "staging_s3:/sco-assets-v1/test1"
2021/09/14 22:18:57 DEBUG : staging_s3: detected overridden config - adding "{V5-qR}" suffix to name
2021/09/14 22:18:57 DEBUG : fs cache: renaming cache item "staging_s3:/sco-assets-v1/test1" to be canonical "staging_s3{V5-qR}:sco-assets-v1/test1"
2021/09/14 22:18:58 DEBUG : test.html: Need to transfer - File not found at Destination
2021/09/14 22:18:58 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5D6RCYAP0GA83V</RequestId><HostId>9dRxexxtEJ1irosxXDOe2yfm5sZBQYFwzsG90eWN3h46JmSyc/TeDgHlNWmsmL6FGc0M0x7hrF+FPJJqm9zJVQ==</HostId></Error>
2021/09/14 22:18:58 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5D6RCYAP0GA83V</RequestId><HostId>9dRxexxtEJ1irosxXDOe2yfm5sZBQYFwzsG90eWN3h46JmSyc/TeDgHlNWmsmL6FGc0M0x7hrF+FPJJqm9zJVQ==</HostId></Error>
2021/09/14 22:18:58 DEBUG : test.html: Need to transfer - File not found at Destination
2021/09/14 22:18:58 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5ENM8XMC7H53K0</RequestId><HostId>SNQ5GxgFztfkeD5frq6Lnd2cFgf1xcc823dqx6wQ6+D3nedSfGDBRXm55437Sp9ERjp1LeZuTwDi//igrHN0Jw==</HostId></Error>
2021/09/14 22:18:58 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W5ENM8XMC7H53K0</RequestId><HostId>SNQ5GxgFztfkeD5frq6Lnd2cFgf1xcc823dqx6wQ6+D3nedSfGDBRXm55437Sp9ERjp1LeZuTwDi//igrHN0Jw==</HostId></Error>
2021/09/14 22:18:58 DEBUG : test.html: Need to transfer - File not found at Destination
2021/09/14 22:18:58 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W53KV38W9VQ21T3</RequestId><HostId>n9zC9R4scfLn3L8AZclQe5msf+ngmiLFmX5puDKcq6RrXsdvo35dZK8lquVCqbMc4lAvccxh4HmV16Q9/kXdHA==</HostId></Error>
2021/09/14 22:18:58 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W53KV38W9VQ21T3</RequestId><HostId>n9zC9R4scfLn3L8AZclQe5msf+ngmiLFmX5puDKcq6RrXsdvo35dZK8lquVCqbMc4lAvccxh4HmV16Q9/kXdHA==</HostId></Error>
2021/09/14 22:18:58 INFO  : 
Transferred:   	        330 / 330 Byte, 100%, 0 Byte/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         1.7s

2021/09/14 22:18:58 DEBUG : 5 go routines active
2021/09/14 22:18:58 Failed to sync: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5W53KV38W9VQ21T3</RequestId><HostId>n9zC9R4scfLn3L8AZclQe5msf+ngmiLFmX5puDKcq6RrXsdvo35dZK8lquVCqbMc4lAvccxh4HmV16Q9/kXdHA==</HostId></Error>

Adding in the --dump bodies option, I see that it's trying to do a PUT / HTTP/1.1 (!), and that gives the failure:

2021/09/14 20:58:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 20:58:40 DEBUG : HTTP REQUEST (req 0xc000b55200)
2021/09/14 20:58:40 DEBUG : PUT / HTTP/1.1
Host: sco-assets-v1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.56.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: authenticated-read
X-Amz-Content-Sha256: xxx
X-Amz-Date: 20210914T205840Z
Accept-Encoding: gzip

2021/09/14 20:58:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 20:58:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 20:58:41 DEBUG : HTTP RESPONSE (req 0xc000b55200)
2021/09/14 20:58:41 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 14 Sep 2021 20:58:40 GMT
Server: AmazonS3
X-Amz-Id-2: xxx
X-Amz-Request-Id: xxx

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>xxx</RequestId><HostId>Fz8dEbryifXPVKKIKb18Ah1oBtBCZSUSw/nlC3WMFP7/LMrYLtEUtcaiGX4gnqM21U+KmcXDhIU=</HostId></Error>
0

2021/09/14 20:58:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

It seems to be failing on CreateBucket:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "xxx",
        "arn": "arn:aws:iam::xxx:user/blah",
        "accountId": "xxx",
        "accessKeyId": "yyy",
        "userName": "anthony_foiani_ops"
    },
    "eventTime": "2021-09-14T21:33:09Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "CreateBucket",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "w.x.y.z",
    "userAgent": "[rclone/v1.56.0]",
    "errorCode": "AccessDenied",
    "errorMessage": "Access Denied",
    "requestParameters": {
        "bucketName": "sco-assets-v1",
        "Host": "sco-assets-v1.s3.us-east-1.amazonaws.com",
        "x-amz-acl": "authenticated-read"
    },
    "responseElements": null,
    "additionalEventData": {
        "SignatureVersion": "SigV4",
        "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "bytesTransferredIn": 0,
        "AuthenticationMethod": "AuthHeader",
        "x-amz-id-2": "foo",
        "bytesTransferredOut": 243
    },
    "requestID": "bar",
    "eventID": "baz",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "xxx",
    "eventCategory": "Management"
}

Checked to see that it wasn't a "public can read" situation; here, my other accounts can't even cat the object:

$ rclone --config ~/.rclone --s3-profile alpha-ops cat staging_s3:sco-assets-v1/hello.html
2021/09/14 23:04:12 Failed to cat: AccessDenied: Access Denied
	status code: 403, request id: ZYKTTVRY8552X2M0, host id: WaMryjCPbgL9nWEMxCfy+86v2Xv2kYcDiW/lfrNqTM9bCdH9AM3KRdimeryeD9fuU2bng76Ljyk=

$ rclone --config ~/.rclone --s3-profile production-ops cat staging_s3:sco-assets-v1/hello.html
2021/09/14 23:04:23 Failed to cat: AccessDenied: Access Denied
	status code: 403, request id: XN0ET1M4H1XSKPSV, host id: 

$ rclone --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/hello.html
<html>
  <head>
    <title>Hello!</title>
  </head>
  <body>
    Hello!
  </body>
</html>

hello and welcome to the forum,

as there is not a --dump for the cat command, just a quick guess...

the two commands has a small difference, between the bucket and folder

  • cat staging_s3:sco-assets-v1 does not have a slash character
  • sync test.html staging_s3:/sco-assets-v1/test1 does have a slash character

hello and welcome to the forum,

Thanks!

as there is not a --dump for the cat command, just a quick guess...

the two commands has a small difference, between the bucket and folder

  • cat staging_s3:sco-assets-v1 does not have a slash character

Hm. I thought I did ... cat staging_s3://sco-assets-v1/hello.html (so it does have a slash in there)?

Yes. See (1) below.

All of these destinations fail the same way:

$ rclone --config ~/.rclone --s3-profile staging-ops copy test.html staging_s3:sco-assets-v1
2021/09/14 23:37:11 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>EN146D4PM0BWJS2D</RequestId><HostId>Tcl1pM+KaKazHQVK8uVkj1ucohV7PtWqxGhOM8g4LFP3wjg5j2/M8pW5UfUivXSl+DVg3za9Qk0=</HostId></Error>
2021/09/14 23:37:11 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>EN146D4PM0BWJS2D</RequestId><HostId>Tcl1pM+KaKazHQVK8uVkj1ucohV7PtWqxGhOM8g4LFP3wjg5j2/M8pW5UfUivXSl+DVg3za9Qk0=</HostId></Error>
2021/09/14 23:37:12 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4T9HHZCP0GMNFABA</RequestId><HostId>RItsHhk08SEN6n+eW2KUW0R0yT2St7trzTmHbVMbeaP+mZKa8zEzGmO+2lIr6HgaBi1pbZADfqw=</HostId></Error>
2021/09/14 23:37:12 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4T9HHZCP0GMNFABA</RequestId><HostId>RItsHhk08SEN6n+eW2KUW0R0yT2St7trzTmHbVMbeaP+mZKa8zEzGmO+2lIr6HgaBi1pbZADfqw=</HostId></Error>
2021/09/14 23:37:12 ERROR : test.html: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4T9XCNQF2M9273D4</RequestId><HostId>OBKc8K5wjVy7q2sjePzIi5Jk8/uu/7yA22I8CWvxQ9muIkyAcWZiiAAURL7U+qUM+ZwbCO7ViII=</HostId></Error>
2021/09/14 23:37:12 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4T9XCNQF2M9273D4</RequestId><HostId>OBKc8K5wjVy7q2sjePzIi5Jk8/uu/7yA22I8CWvxQ9muIkyAcWZiiAAURL7U+qUM+ZwbCO7ViII=</HostId></Error>
2021/09/14 23:37:12 Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4T9XCNQF2M9273D4</RequestId><HostId>OBKc8K5wjVy7q2sjePzIi5Jk8/uu/7yA22I8CWvxQ9muIkyAcWZiiAAURL7U+qUM+ZwbCO7ViII=</HostId></Error>

$ rclone --config ~/.rclone --s3-profile staging-ops copy test.html staging_s3:sco-assets-v1/
[same]

$ rclone --config ~/.rclone --s3-profile staging-ops copy test.html staging_s3:/sco-assets-v1
[same]

$ rclone --config ~/.rclone --s3-profile staging-ops copy test.html staging_s3:/sco-assets-v1/
[same]

$ rclone --config ~/.rclone --s3-profile staging-ops copy test.html staging_s3://sco-assets-v1
[same]

$ rclone --config ~/.rclone --s3-profile staging-ops copy test.html staging_s3://sco-assets-v1/
[same]
  • sync test.html staging_s3:/sco-assets-v1/test1 does have a slash character

I think I tried it both ways? Yes, I tried that one both ways, too. Failed in the same manner each
time.

Tried it with double slashes, too (s3://sco-assets-v1) and that failed as well.

I found at least one discussion regarding slashes, but it seemed to conclude that, since slashes are a legal character in S3 object names, it was hard to do any trimming.

Which makes me feel that I'm missing something fundamental about how rclone arguments are mapped to remote destinations. I'm really confused, though, that I can get cat to work, can upload content
with the same aws profile (with aws s3 cp, see below)... but I can't get rclone to write to it?

# Make sure there's nothing there.
$ aws --profile=staging-ops s3 rm s3://sco-assets-v1/test.html
delete: s3://sco-assets-v1/test.html

# Make sure it doesn't exist. (Aside: I'm a little startled at the 0 exit status here...)
$ rclone --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/test.html ; echo $?
0

# Now upload with the `aws` CLI tool:
$ aws --profile=staging-ops s3 cp test.html s3://sco-assets-v1/
upload: ./test.html to s3://sco-assets-v1/test.html

# And retrieve it:
$ rclone --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/test.html ; echo $?
<html>
  <head>
    <title>test of staging</title>
  </head>
  <body>
    <p>Testing...</p>
  </body>
</html>
0

I can see the buckets no problem, and the files I've added manually:

$ rclone --config ~/.rclone --s3-profile staging-ops lsd staging_s3:
                                           [...]
          -1 2021-09-13 16:34:40        -1 sco-assets-v1
                                           [...]

$ rclone --config ~/.rclone --s3-profile staging-ops ls staging_s3:sco-assets-v1
       90 hello.html
      110 test.html

(1)

$ rclone -vv --dump bodies --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/hello.html
Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone -vv --dump bodies --config ~/.rclone --s3-profile staging-ops cat staging_s3:sco-assets-v1/hello.html
2021/09/14 23:19:25 DEBUG : rclone: Version "v1.56.0" starting with parameters ["Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone" "-vv" "--dump" "bodies" "--config" "/Users/anthony_foiani/.rclone" "--s3-profile" "staging-ops" "cat" "staging_s3:sco-assets-v1/hello.html"]
2021/09/14 23:19:25 DEBUG : Creating backend with remote "staging_s3:sco-assets-v1/hello.html"
2021/09/14 23:19:25 DEBUG : Using config file from "/Users/anthony_foiani/.rclone"
2021/09/14 23:19:25 DEBUG : staging_s3: detected overridden config - adding "{V5-qR}" suffix to name
2021/09/14 23:19:25 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/09/14 23:19:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 23:19:25 DEBUG : HTTP REQUEST (req 0xc000213000)
2021/09/14 23:19:25 DEBUG : HEAD /hello.html HTTP/1.1
Host: sco-assets-v1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.56.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210914T231925Z

2021/09/14 23:19:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 23:19:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 23:19:25 DEBUG : HTTP RESPONSE (req 0xc000213000)
2021/09/14 23:19:25 DEBUG : HTTP/1.1 200 OK
Content-Length: 90
Accept-Ranges: bytes
Content-Type: text/html
Date: Tue, 14 Sep 2021 23:19:26 GMT
Etag: "27232bbb94bc4607860caa5d55cbc23c"
Last-Modified: Mon, 13 Sep 2021 17:48:44 GMT
Server: AmazonS3
X-Amz-Id-2: +VSk5KnmBRvcOkl2nOuracv41WfbqukMOBGb2yFUIcJvZEl02GGxvp0NadB/lwixBxdhP9rZi/E=
X-Amz-Request-Id: 05JFB4PJYE0VM320
X-Amz-Version-Id: null

2021/09/14 23:19:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 23:19:25 DEBUG : fs cache: adding new entry for parent of "staging_s3:sco-assets-v1/hello.html", "staging_s3{V5-qR}:sco-assets-v1"
2021/09/14 23:19:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 23:19:25 DEBUG : HTTP REQUEST (req 0xc000213800)
2021/09/14 23:19:25 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=1000&prefix= HTTP/1.1
Host: sco-assets-v1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.56.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210914T231925Z
Accept-Encoding: gzip

2021/09/14 23:19:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 23:19:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 23:19:25 DEBUG : HTTP RESPONSE (req 0xc000213800)
2021/09/14 23:19:25 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 14 Sep 2021 23:19:26 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 1EE8wrh6c6vl9i+bcQqhWAruJbRuyhYjNOuXL3zfKw9/JgI3Ly9UuFfh9LjOmdRG2d8y5gOZHdI=
X-Amz-Request-Id: 05J7Z6ZT9RC2X39X

3bd
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>sco-assets-v1</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>hello.html</Key><LastModified>2021-09-13T17:48:44.000Z</LastModified><ETag>&quot;27232bbb94bc4607860caa5d55cbc23c&quot;</ETag><Size>90</Size><Owner><ID>4ab9a4c39ea17f6a958c55444a8032889a20cd5134208a04b5b4c6afb54bbf48</ID><DisplayName>doug+staging</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>test.html</Key><LastModified>2021-09-14T23:16:38.000Z</LastModified><ETag>&quot;56c84f4ff89dbf9c88fb69c33001e526&quot;</ETag><Size>110</Size><Owner><ID>4ab9a4c39ea17f6a958c55444a8032889a20cd5134208a04b5b4c6afb54bbf48</ID><DisplayName>doug+staging</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
0

2021/09/14 23:19:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 23:19:25 DEBUG : test.html: Excluded
2021/09/14 23:19:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 23:19:25 DEBUG : HTTP REQUEST (req 0xc000137500)
2021/09/14 23:19:25 DEBUG : GET /hello.html HTTP/1.1
Host: sco-assets-v1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.56.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210914T231925Z
Accept-Encoding: gzip

2021/09/14 23:19:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/14 23:19:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/14 23:19:26 DEBUG : HTTP RESPONSE (req 0xc000137500)
2021/09/14 23:19:26 DEBUG : HTTP/1.1 200 OK
Content-Length: 90
Accept-Ranges: bytes
Content-Type: text/html
Date: Tue, 14 Sep 2021 23:19:26 GMT
Etag: "27232bbb94bc4607860caa5d55cbc23c"
Last-Modified: Mon, 13 Sep 2021 17:48:44 GMT
Server: AmazonS3
X-Amz-Id-2: +7N5Pf5fATQu+cSbXCzaJr6HxhZLdwRBFdHgvdeVGrQ1wftzKuHZDv+Kf4sm380V1khBenH7rBI=
X-Amz-Request-Id: 05J75CQDEYE9C2PF
X-Amz-Version-Id: null

<html>
  <head>
    <title>Hello!</title>
  </head>
  <body>
    Hello!
  </body>
</html>
2021/09/14 23:19:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<html>
  <head>
    <title>Hello!</title>
  </head>
  <body>
    Hello!
  </body>
</html>
$ rclone -vv --config ~/.rclone --s3-profile staging-ops sync test.html staging_s3:/sco-assets-v1
2021/09/14 23:20:52 DEBUG : rclone: Version "v1.56.0" starting with parameters ["Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone" "-vv" "--config" "/Users/anthony_foiani/.rclone" "--s3-profile" "staging-ops" "sync" "test.html" "staging_s3:/sco-assets-v1"]
2021/09/14 23:20:52 DEBUG : Creating backend with remote "test.html"
2021/09/14 23:20:52 DEBUG : Using config file from "/Users/anthony_foiani/.rclone"
2021/09/14 23:20:52 DEBUG : fs cache: adding new entry for parent of "test.html", "/Users/anthony_foiani"
2021/09/14 23:20:52 DEBUG : Creating backend with remote "staging_s3:/sco-assets-v1"
2021/09/14 23:20:52 DEBUG : staging_s3: detected overridden config - adding "{V5-qR}" suffix to name
2021/09/14 23:20:52 DEBUG : fs cache: renaming cache item "staging_s3:/sco-assets-v1" to be canonical "staging_s3{V5-qR}:sco-assets-v1"
2021/09/14 23:20:53 DEBUG : test.html: Modification times differ by 3h1m41.910875886s: 2021-09-14 20:14:56.089124114 +0000 GMT, 2021-09-14 23:16:38 +0000 UTC
2021/09/14 23:20:53 DEBUG : test.html: md5 = 56c84f4ff89dbf9c88fb69c33001e526 OK
2021/09/14 23:20:53 ERROR : test.html: Failed to set modification time: AccessDenied: Access Denied
	status code: 403, request id: 6JN9RZFH1P17XZQB, host id: y4lUryBiboHQiBtCQHzOnNAu1cL/Z28WXVsgdPXLYc/1YDZZBqqWlCByghIsgQ5pX9zeC/vUl99WePfoWTHYUw==
2021/09/14 23:20:53 DEBUG : test.html: Unchanged skipping
2021/09/14 23:20:53 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: 6JN9RZFH1P17XZQB, host id: y4lUryBiboHQiBtCQHzOnNAu1cL/Z28WXVsgdPXLYc/1YDZZBqqWlCByghIsgQ5pX9zeC/vUl99WePfoWTHYUw==
2021/09/14 23:20:53 DEBUG : test.html: Modification times differ by 3h1m41.910875886s: 2021-09-14 20:14:56.089124114 +0000 GMT, 2021-09-14 23:16:38 +0000 UTC
2021/09/14 23:20:53 DEBUG : test.html: md5 = 56c84f4ff89dbf9c88fb69c33001e526 OK
2021/09/14 23:20:53 ERROR : test.html: Failed to set modification time: AccessDenied: Access Denied
	status code: 403, request id: 6JN681G7R69AAFYQ, host id: 9SKojTwK8UWbJ9PXbztdgV5I/JqGfnSHnUNIinxIH8VOjhbeRseXwbkcQ9JWcKY3Hm9ueW0Z0gDynANPr2Xrkg==
2021/09/14 23:20:53 DEBUG : test.html: Unchanged skipping
2021/09/14 23:20:53 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: 6JN681G7R69AAFYQ, host id: 9SKojTwK8UWbJ9PXbztdgV5I/JqGfnSHnUNIinxIH8VOjhbeRseXwbkcQ9JWcKY3Hm9ueW0Z0gDynANPr2Xrkg==
2021/09/14 23:20:53 DEBUG : test.html: Modification times differ by 3h1m41.910875886s: 2021-09-14 20:14:56.089124114 +0000 GMT, 2021-09-14 23:16:38 +0000 UTC
2021/09/14 23:20:53 DEBUG : test.html: md5 = 56c84f4ff89dbf9c88fb69c33001e526 OK
2021/09/14 23:20:53 ERROR : test.html: Failed to set modification time: AccessDenied: Access Denied
	status code: 403, request id: 6JN3ACXB1HEPTRHW, host id: hmTqi5OSD+eMTjoCjMit/cqBQySAEaMhK11UtgxH5nzyDJe8o4OSzPgkvFj2sWMlyY4rGrjrdR12V1Lnq5ox1Q==
2021/09/14 23:20:53 DEBUG : test.html: Unchanged skipping
2021/09/14 23:20:53 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: 6JN3ACXB1HEPTRHW, host id: hmTqi5OSD+eMTjoCjMit/cqBQySAEaMhK11UtgxH5nzyDJe8o4OSzPgkvFj2sWMlyY4rGrjrdR12V1Lnq5ox1Q==
2021/09/14 23:20:53 INFO  :
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         1.1s

2021/09/14 23:20:53 DEBUG : 6 go routines active
2021/09/14 23:20:53 Failed to sync: AccessDenied: Access Denied
	status code: 403, request id: 6JN3ACXB1HEPTRHW, host id: hmTqi5OSD+eMTjoCjMit/cqBQySAEaMhK11UtgxH5nzyDJe8o4OSzPgkvFj2sWMlyY4rGrjrdR12V1Lnq5ox1Q==

$ rclone -vv --config ~/.rclone --s3-profile staging-ops sync test.html staging_s3:/sco-assets-v1/
2021/09/14 23:20:55 DEBUG : rclone: Version "v1.56.0" starting with parameters ["Downloads/rclone/rclone-v1.56.0-osx-amd64/rclone" "-vv" "--config" "/Users/anthony_foiani/.rclone" "--s3-profile" "staging-ops" "sync" "test.html" "staging_s3:/sco-assets-v1/"]
2021/09/14 23:20:55 DEBUG : Creating backend with remote "test.html"
2021/09/14 23:20:55 DEBUG : Using config file from "/Users/anthony_foiani/.rclone"
2021/09/14 23:20:55 DEBUG : fs cache: adding new entry for parent of "test.html", "/Users/anthony_foiani"
2021/09/14 23:20:55 DEBUG : Creating backend with remote "staging_s3:/sco-assets-v1/"
2021/09/14 23:20:55 DEBUG : staging_s3: detected overridden config - adding "{V5-qR}" suffix to name
2021/09/14 23:20:55 DEBUG : fs cache: renaming cache item "staging_s3:/sco-assets-v1/" to be canonical "staging_s3{V5-qR}:sco-assets-v1"
2021/09/14 23:20:56 DEBUG : test.html: Modification times differ by 3h1m41.910875886s: 2021-09-14 20:14:56.089124114 +0000 GMT, 2021-09-14 23:16:38 +0000 UTC
2021/09/14 23:20:56 DEBUG : test.html: md5 = 56c84f4ff89dbf9c88fb69c33001e526 OK
2021/09/14 23:20:56 ERROR : test.html: Failed to set modification time: AccessDenied: Access Denied
	status code: 403, request id: ZJ9KQ29ZYQ5AS1QR, host id: 9UfGa8HI1fz0Vzn/pzy3Nr2+a8TQAnexwX5lqkWxDNKBPZxWcN6ZpyuOt/lh80C3nDegUJzugkwNgQ+b/pdthQ==
2021/09/14 23:20:56 DEBUG : test.html: Unchanged skipping
2021/09/14 23:20:56 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: ZJ9KQ29ZYQ5AS1QR, host id: 9UfGa8HI1fz0Vzn/pzy3Nr2+a8TQAnexwX5lqkWxDNKBPZxWcN6ZpyuOt/lh80C3nDegUJzugkwNgQ+b/pdthQ==
2021/09/14 23:20:56 DEBUG : test.html: Modification times differ by 3h1m41.910875886s: 2021-09-14 20:14:56.089124114 +0000 GMT, 2021-09-14 23:16:38 +0000 UTC
2021/09/14 23:20:56 DEBUG : test.html: md5 = 56c84f4ff89dbf9c88fb69c33001e526 OK
2021/09/14 23:20:56 ERROR : test.html: Failed to set modification time: AccessDenied: Access Denied
	status code: 403, request id: ZJ9P33DJTB1KKXPC, host id: f8XscFWBgFw4vvexoytwbYB1ee8xLy1CtCRkOTFWyCcXIt/EYXVvHi4buJSGwt2KbqFf/+wqL22zyVozGIgzZw==
2021/09/14 23:20:56 DEBUG : test.html: Unchanged skipping
2021/09/14 23:20:56 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: ZJ9P33DJTB1KKXPC, host id: f8XscFWBgFw4vvexoytwbYB1ee8xLy1CtCRkOTFWyCcXIt/EYXVvHi4buJSGwt2KbqFf/+wqL22zyVozGIgzZw==
2021/09/14 23:20:56 DEBUG : test.html: Modification times differ by 3h1m41.910875886s: 2021-09-14 20:14:56.089124114 +0000 GMT, 2021-09-14 23:16:38 +0000 UTC
2021/09/14 23:20:56 DEBUG : test.html: md5 = 56c84f4ff89dbf9c88fb69c33001e526 OK
2021/09/14 23:20:56 ERROR : test.html: Failed to set modification time: AccessDenied: Access Denied
	status code: 403, request id: ZJ9Q53DEYP2DEF10, host id: fOJXYckbjyquTYPuWaaghk014vknGZo6sbiExTGWu4+BFjzrdkdCUH9L9OCZcPs6vhKQGtA2RYqZe1NOnOxNHg==
2021/09/14 23:20:56 DEBUG : test.html: Unchanged skipping
2021/09/14 23:20:56 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: ZJ9Q53DEYP2DEF10, host id: fOJXYckbjyquTYPuWaaghk014vknGZo6sbiExTGWu4+BFjzrdkdCUH9L9OCZcPs6vhKQGtA2RYqZe1NOnOxNHg==
2021/09/14 23:20:56 INFO  :
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         1.0s

2021/09/14 23:20:56 DEBUG : 6 go routines active
2021/09/14 23:20:56 Failed to sync: AccessDenied: Access Denied
	status code: 403, request id: ZJ9Q53DEYP2DEF10, host id: fOJXYckbjyquTYPuWaaghk014vknGZo6sbiExTGWu4+BFjzrdkdCUH9L9OCZcPs6vhKQGtA2RYqZe1NOnOxNHg==

like i said, just a guess even though rclone will remove that slash
fs cache: renaming cache item "staging_s3:/sco-assets-v1" to be canonical "staging_s3{V5-qR}:sco-assets-v1"

like i said, just a guess even though rclone will remove that slash

Ah, I get it this time. Yes, I see what you're saying about the canonical name.

Thanks for the explanation. Now to figure out why it's complaining at me...

you have a complex use-case.

--config ~/.rclone and --s3-profile staging-ops
is there a reason to use both in the same command?

Womp womp. TL;DR: my user does not have the privileges to set the ACL that I'm requesting in my rclone config. That is, this setting:

acl = authenticated-read

Conflicts with the (recommended) Block Public Access setting on the S3 bucket.

Thanks very much for your help!

I'll leave the rest of my notes here, in case they're helpful to others.

Have a great day!

Best,
Anthony


you have a complex use-case.

Yeah. :confused:

My ultimate goal is to mirror content from Google Drive to S3 (from where it will then be served out via CloudFront). I was hoping to use rclone to do the mirroring, running it every 5m (or some kind of daemon mode, but I figured that getting it working in a "one-shot" mode would be a good start).

I also thought I'd test the two components individually first:

  1. Read content from GDrive
  2. Write content to S3

I successfully read from GDrive, with the config shown in one of my earlier messages:

[staging_gdrive]
type = drive
client_id = blah.apps.googleusercontent.com
client_secret = quux
scope = drive
service_account_file = /Users/me/Downloads/sco-assets-v1-staging-64109ee431ef.json

I agree that specifying a profile name for my AWS credentials is a bit odd, but it will eventually be running either with secrets (e.g., from AWS Secrets Manager) or a task/instance IAM identity.

Either way, it seems that it's picking up my identity correctly -- I'm not able to even read the file with any other account, just this one (which is what I expect). Moving that setting to my config has it working same as it was previously:

[staging_s3]
type = s3
provider = AWS
profile = staging-ops
env_auth = true
region = us-east-1
acl = authenticated-read
server_side_encryption = AES256
no_check_bucket = true

Still fails to copy. (I also tried it with/without server_side_encryption.)

It looks like it's trying to do CreateBucket on /, and that seems to be the bit that's blowing up.

Can / should / must I specify the bucket name in my config? I wouldn't think so, since otherwise lsd wouldn't work (or, at least, not work the same way).

Regardless, thanks again for continuing to look into this with me!

i do specify the bucket name, using an alias remote, here the bucket name is kdbx

[kdbx_remote]
type = s3
provider = Wasabi
access_key_id = 
secret_access_key = 
endpoint = 

[kdbx]
type = alias
remote = kdbx_remote:kdbx

rclone ls kdbx:

1 Like

That makes perfect sense: one stanza for the backend provider, then an alias stanza for each resource on that backend. I'll definitely keep that in mind.

Thanks again!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.