S3 copy - 403 error Access Denied

Suspected Bug
#1

What is the problem you are having with rclone?

Copy to S3 stopped functioning as of 12/28/2020

I've tried with 3 different bucket-acl types:
default ""
public-read-write
public-read

I've tried the --s3-no-check-bucket flag, --s3-server-side-encryption aws:kms, and --s3-v2-auth (separately of course) and they all produce the same error more or less. v2-auth gives a 400 status code instead of 403.

People with similar problems all seemed to have them fixed by new releases but I'm on the newest stable past where their issues were resolved so I'm a bit stumped.

What is your rclone version (output from rclone version)

1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P ./test2.txt remote:XXX/XX/XXX

The rclone config contents with secrets removed.

type = s3
provider = AWS
env_auth = false
access_key_id = XXXXXX
secret_access_key = XXXXXX
region = eu-central-1
acl = public-read

A log from the command with the -vv flag

2021/01/07 13:50:40 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "-vv" "-P" "./test2.txt" "XXXX" "--dump" "responses" "--retries" "1"]
2021/01/07 13:50:40 DEBUG : Creating backend with remote "./test2.txt"
2021/01/07 13:50:40 DEBUG : Using config file from "/home/XXXX/.config/rclone/rclone.conf"
2021/01/07 13:50:40 DEBUG : fs cache: adding new entry for parent of "./test2.txt", "/home/XXXXX"
2021/01/07 13:50:40 DEBUG : Creating backend with remote "XXXXX"
2021/01/07 13:50:40 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/07 13:50:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/07 13:50:40 DEBUG : HTTP REQUEST (req 0xc0005fbc00)
2021/01/07 13:50:40 DEBUG : HEAD /XXXXX HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.53.3
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210107T005040Z

2021/01/07 13:50:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/07 13:50:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/07 13:50:41 DEBUG : HTTP RESPONSE (req 0xc0005fbc00)
2021/01/07 13:50:41 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 07 Jan 2021 00:50:40 GMT
Server: AmazonS3
X-Amz-Id-2: V4mAY7Kezg5y+RgwPK9R5s68/UtXF9iPauS5w7b9MkB5ZG10hr6bjsYWiVPG4DwHunsRZo3FN8Q=
X-Amz-Request-Id: FCB33A798DA488BA


2021/01/07 13:50:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/07 13:50:41 DEBUG : fs cache: renaming cache item "XXXXX/" to be canonical "XXXXXX"
2021-01-07 13:50:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-01-07 13:50:41 DEBUG : HTTP REQUEST (req 0xc0000e8400)
2021-01-07 13:50:41 DEBUG : HEAD /XXXX/test2.txt HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.53.3
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210107T005041Z
2021-01-07 13:50:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-01-07 13:50:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-01-07 13:50:41 DEBUG : HTTP RESPONSE (req 0xc0000e8400)
2021-01-07 13:50:41 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 07 Jan 2021 00:50:40 GMT
Server: AmazonS3
X-Amz-Id-2: 5DNu19IKQ0Lns+ZDiMqTs21W5TPxpGM0OO+pVVuzCQ1wyzSB/BibiaxoCAdq6hOenAIq+o+4ZTo=
X-Amz-Request-Id: 018D5C126FDFC902
2021-01-07 13:50:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-01-07 13:50:41 DEBUG : test2.txt: Need to transfer - File not found at Destination
2021-01-07 13:50:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-01-07 13:50:41 DEBUG : HTTP REQUEST (req 0xc0000e8900)
2021-01-07 13:50:41 DEBUG : PUT /XXXXX HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.53.3
Content-Length: 156
Authorization: XXXX
X-Amz-Acl: public-read
X-Amz-Content-Sha256: 270d7010e28541d025cba79779722247d464e2edd8b448b42fdf618a477e0432
X-Amz-Date: 20210107T005041Z
Accept-Encoding: gzip
2021-01-07 13:50:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-01-07 13:50:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-01-07 13:50:41 DEBUG : HTTP RESPONSE (req 0xc0000e8900)
2021-01-07 13:50:41 DEBUG : HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 07 Jan 2021 00:50:40 GMT
Server: AmazonS3
X-Amz-Id-2: YoPE5XoHAs/OXOrj76HpuzBv1EIElPEAG67I3GIYKSNhubnsuUkhJW95xJCyNQXUHG5NaF5TfXo=
X-Amz-Request-Id: 9588882026159369

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>9588882026159369</RequestId><HostId>YoPE5XoHAs/OXOrj76HpuzBv1EIElPEAG67I3GIYKSNhubnsuUkhJW95xJCyNQXUHG5NaF5TfXo=</HostId></Error>
0
2021-01-07 13:50:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-01-07 13:50:41 ERROR : test2.txt: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: 9588882026159369, host id: YoPE5XoHAs/OXOrj76HpuzBv1EIElPEAG67I3GIYKSNhubnsuUkhJW95xJCyNQXUHG5NaF5TfXo=
2021-01-07 13:50:41 ERROR : Attempt 1/1 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: 9588882026159369, host id: YoPE5XoHAs/OXOrj76HpuzBv1EIElPEAG67I3GIYKSNhubnsuUkhJW95xJCyNQXUHG5NaF5TfXo=
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.7s
2021/01/07 13:50:41 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.7s

2021/01/07 13:50:41 DEBUG : 2 go routines active
2021/01/07 13:50:41 Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: 9588882026159369, host id: YoPE5XoHAs/OXOrj76HpuzBv1EIElPEAG67I3GIYKSNhubnsuUkhJW95xJCyNQXUHG5NaF5TfXo=
#2

hello,
what makes you think this is a bug?

#3

The other similar cases resulted in code changes to fix, but if it belongs somewhere else I'm happy to move it.

#4

"File not found at Destination" uploading to s3
that was a newbie mistake, does not apply to your config file as posted.

#5

Yes, I noticed that, but was just giving examples of other posts I referenced. The fact that one was in suspected bugs did not weigh on my decision.

Happy to move it or a mod can move it if its improper.

#6

given, that we both agree that post is not related to your post,
i would edit that post and remove that item so other forum members, will take spend time reading it.

#7

Are you saying this did work, but just stopped working on that date?

There is something odd about the request you posted which may be relevant

I'd expect to see the bucket in the Host: header so Host: bucket.s3.eu.... - not using that is something amazon are planning on deprecating.

This should be enabled by this line in your config which suggests that maybe you are using a different config?

You could also try the latest beta

#8

Yes. There was a failure with the VM, and so I moved the machine over to a new VM. Don't know why this would be related, but in case it is. FWIW my google drive remote on the same machine continues to function normally.

I confirmed I'm using the same remote as the config I sent. In case the VM failure caused some type of corruption I also made a new remote with the same settings. Both exhibit the same behavior.

Upgraded to the beta, same outcome:

2021/01/08 10:33:28 DEBUG : rclone: Version "v1.54.0-beta.5040.71edc75ca" starting with parameters ["rclone" "copy" "./test2.txt" "remote:/XXXXXX/input/XXXXX/" "-vv" "--dump" "responses" "--retries" "1"]
2021/01/08 10:33:28 DEBUG : Creating backend with remote "./test2.txt"
2021/01/08 10:33:28 DEBUG : Using config file from "/home/brad/.config/rclone/rclone.conf"
2021/01/08 10:33:28 DEBUG : fs cache: adding new entry for parent of "./test2.txt", "/home/XXX/XXXXX"
2021/01/08 10:33:28 DEBUG : Creating backend with remote "remote:/XXXX/input/XXXXX/"
2021/01/08 10:33:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/08 10:33:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/08 10:33:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/08 10:33:29 DEBUG : HTTP REQUEST (req 0xc00055c400)
2021/01/08 10:33:29 DEBUG : HEAD /XXXXX/input/XXXXXX HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.0-beta.5040.71edc75ca
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210107T213328Z

2021/01/08 10:33:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/08 10:33:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/08 10:33:29 DEBUG : HTTP RESPONSE (req 0xc00055c400)
2021/01/08 10:33:29 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 07 Jan 2021 21:33:29 GMT
Server: AmazonS3
X-Amz-Id-2: tjbbgmtwsAGmMyABcnlB5b+vbvs7gjwEg9dsWUdaA9tkVxQA4h4WFMpUpWXcDuVn6eANxWqsP+0=
X-Amz-Request-Id: 90D6254DF92A08E9

2021/01/08 10:33:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/08 10:33:29 Failed to create file system for "remote:/XXXXXXXX/input/XXXXXXX/": Forbidden: Forbidden
status code: 403, request id: 90D6254DF92A08E9, host id: tjbbgmtwsAGmMyABcnlB5b+vbvs7gjwEg9dsWUdaA9tkVxQA4h4WFMpUpWXcDuVn6eANxWqsP+0=

#9

Judging by the log above, your remote is missing the provider = AWS line. Can you check it has got that?

#10

Both new and old remote have it.

[saXXXXX]
type = s3
provider = AWS
env_auth = false
access_key_id = AKIA2XXXXXXUKP
secret_access_key = 4P5AXXXXXXXXXxRze4WGFy
region = eu-central-1

[saXXXXXX2]
type = s3
provider = AWS
env_auth = false
access_key_id = AKIAXXXXXXXXXUKP
secret_access_key = 4P5AXXXXXXXXXXXWGFy
region = eu-central-1
acl = public-read

#11

Hmm..

Just a thought - can you try without the leading / on the cli, so

remote:XXXXXX/input/XXXXX

Instead of

remote:/XXXXXX/input/XXXXX
#12

Both remotes gave the same result

rclone copy -vv ./test2.txt saXXX:XXXXX/XXXX/XXXXXX/ --dump responses --retries 1
2021/01/09 11:16:28 DEBUG : rclone: Version "v1.54.0-beta.5040.71edc75ca" starting with parameters ["rclone" "copy" "-vv" "./test2.txt" "saXXX:XXXX/XXX/XXXXX/" "--dump" "responses" "--retries" "1"]
2021/01/09 11:16:28 DEBUG : Creating backend with remote "./test2.txt"
2021/01/09 11:16:28 DEBUG : Using config file from "/home/XXX/.config/rclone/rclone.conf"
2021/01/09 11:16:28 DEBUG : fs cache: adding new entry for parent of "./test2.txt", "/home/XXX/XXXXX"
2021/01/09 11:16:28 DEBUG : Creating backend with remote "saXXX:XXXXX/XXXX/XXXXX/"
2021/01/09 11:16:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 11:16:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 11:16:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/09 11:16:28 DEBUG : HTTP REQUEST (req 0xc0004b0400)
2021/01/09 11:16:28 DEBUG : HEAD /XXXXXX/XXXX/XXXXXXXX HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.0-beta.5040.71edc75ca
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210108T221628Z

2021/01/09 11:16:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/09 11:16:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/09 11:16:28 DEBUG : HTTP RESPONSE (req 0xc0004b0400)
2021/01/09 11:16:28 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 08 Jan 2021 22:16:28 GMT
Server: AmazonS3
X-Amz-Id-2: ZHxX+YAF/tMshw3vRqM2zt74CFKF1vzjKrRlzQVnAylCPmucDp6M4sigbYvgltz/anUbwcfOcgY=
X-Amz-Request-Id: BDBACC2ED8B875DC

2021/01/09 11:16:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/09 11:16:28 Failed to create file system for "saXXXX:XXX/XX/XXXXX/": Forbidden: Forbidden
status code: 403, request id: BDBACC2ED8B875DC, host id: ZHxX+YAF/tMshw3vRqM2zt74CFKF1vzjKrRlzQVnAylCPmucDp6M4sigbYvgltz/anUbwcfOcgY=

#13

It's failing at the HEAD request when rclone tries to see if the first exists.

Does the user you are using have permission to HEAD existing files?

The other possibility is the user doesn't have permission to access the bucket for some reason. Can you list the bucket with rclone?

#14

I assume so since this was working before. Its not my bucket, so I can't show the permissions file, but I will ask for it.

I assume you mean like a lsf. This is the output, works fine.

rclone lsf -vv saXXX: --dump responses
2021/01/09 12:22:24 DEBUG : rclone: Version "v1.54.0-beta.5040.71edc75ca" starting with parameters ["rclone" "lsf" "-vv" "saXXXX:" "--dump" "responses"]
2021/01/09 12:22:24 DEBUG : Using config file from "/home/XXXX/.config/rclone/rclone.conf"
2021/01/09 12:22:24 DEBUG : Creating backend with remote "saXXXX:"
2021/01/09 12:22:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 12:22:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 12:22:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/09 12:22:24 DEBUG : HTTP REQUEST (req 0xc00049c500)
2021/01/09 12:22:24 DEBUG : GET / HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.0-beta.5040.71edc75ca
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210108T232224Z
Accept-Encoding: gzip

2021/01/09 12:22:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/09 12:22:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/09 12:22:25 DEBUG : HTTP RESPONSE (req 0xc00049c500)
2021/01/09 12:22:25 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 08 Jan 2021 23:22:25 GMT
Server: AmazonS3
X-Amz-Id-2: 5PiO/HFot+xgPFfCJJjjJTvissMlgNcIUxotqeSeXoh5eFjochOA5xF4xwKlWQhwKZ7kCPjJOTE=
X-Amz-Request-Id: AE409FFE01BC4712

1616

<?xml version="1.0" encoding="UTF-8"?>

<ListAllMyBucketsResult

#15

If I try to copy a file using the AWS api, it is successful.

aws s3 cp ./test2.txt s3://XXXX/XXXX/XXXXXX/
upload: ./test2.txt to s3://XXXX/XXXX/XXXXXX/

aws s3 ls s3://XXXX/XXXX/XXXXXX/
...
...
...
2021-01-09 12:43:12 5 test2.txt

#16

I don't think the aws s3 cp does any HEAD requests to see if the file already exists or check it arrived properly. You can turn on debug to show the HTTP requests I think.

Can you GET files using rclone? So copy them from the bucket?