403 on list or mount of AWS requester pays bucket

rman0604 · March 17, 2021, 9:28pm

What is the problem you are having with rclone?

My ultimate goal is to mount this AWS open bucket: registry.opendata.aws/sentinel-2/
However I get a 403 on both an rclone listing or mount. To keep it simple I will focus on the listing.
Note this AWS command is successful: "C:\rclone>aws s3 ls sentinel-s2-l1c/tiles/ --request-payer requester"

What is your rclone version (output from `rclone version`)

v1.54.1
-->

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Win Server 2016 Datacenter

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone ls anon-eucentral1:sentinel-s2-l1c/tiles/

The rclone config contents with secrets removed.

[anon-eucentral1]
type = s3
provider = AWS
env_auth = false
region = eu-central-1
location_constraint = EU
acl = private
requester-pays = true
access_key_id = xxx
secret_access_key = xxx

A log from the command with the `-vv` flag

2021/03/17 21:16:56 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "ls" "anon-eucentral1:sentinel-s2-l1c/tiles/" "--log-level" "DEBUG"]
2021/03/17 21:16:56 DEBUG : Using config file from "C:\\Users\\xxx\\.config\\rclone\\rclone.conf"
2021/03/17 21:16:56 DEBUG : Creating backend with remote "anon-eucentral1:sentinel-s2-l1c/tiles/"
2021/03/17 21:16:57 DEBUG : fs cache: renaming cache item "anon-eucentral1:sentinel-s2-l1c/tiles/" to be canonical "anon-eucentral1:sentinel-s2-l1c/tiles"
2021/03/17 21:16:57 DEBUG : 4 go routines active
2021/03/17 21:16:57 Failed to ls: AccessDenied: Access Denied
        status code: 403, request id: XNVE6KF3KS83JHBY, host id: PiXxTkKhhgrAFNS9E7JgK4YKA9C6Vr8Uq7SQf00QetO5jHFTbW27qJxWnsb72HaaFKixAIHsJcM=

asdffdsa · March 17, 2021, 9:35pm

hello and welcome to the forum

this will provide more detail.

https://rclone.org/docs/#dump-headers

rman0604 · March 18, 2021, 1:15am

Thanks. Not seeing anything obvious here but that doesn't mean much. I have to believe my keys are valid since they work when running an ls from aws command line.

C:\rclone>rclone ls anon-eucentral1:sentinel-s2-l1c/tiles/ -vv --dump headers
2021/03/18 01:08:41 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "ls" "anon-eucentral1:sentinel-s2-l1c/tiles/" "-vv" "--dump" "headers"]
2021/03/18 01:08:41 DEBUG : Using config file from "C:\\Users\\xxx\\.config\\rclone\\rclone.conf"
2021/03/18 01:08:41 DEBUG : Creating backend with remote "anon-eucentral1:sentinel-s2-l1c/tiles/"
2021/03/18 01:08:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/18 01:08:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/18 01:08:41 DEBUG : HTTP REQUEST (req 0xc000612000)
2021/03/18 01:08:41 DEBUG : HEAD /tiles HTTP/1.1
Host: sentinel-s2-l1c.s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210318T010841Z

2021/03/18 01:08:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/18 01:08:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/18 01:08:42 DEBUG : HTTP RESPONSE (req 0xc000612000)
2021/03/18 01:08:42 DEBUG : HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/xml
Date: Thu, 18 Mar 2021 01:08:41 GMT
Server: AmazonS3
X-Amz-Id-2: 4PbLUM6BpcTuWsGLXdYh9TGrKtAvJalmfWt762C65EZnN3YRvvXWsvCaoKytXPQohXEEk9Aqsh8=
X-Amz-Request-Id: ENHF8D7YSAP0JTCR

2021/03/18 01:08:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/18 01:08:42 DEBUG : fs cache: renaming cache item "anon-eucentral1:sentinel-s2-l1c/tiles/" to be canonical "anon-eucentral1:sentinel-s2-l1c/tiles"
2021/03/18 01:08:42 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/18 01:08:42 DEBUG : HTTP REQUEST (req 0xc0002e0200)
2021/03/18 01:08:42 DEBUG : GET /?delimiter=&encoding-type=url&max-keys=1000&prefix=tiles%2F HTTP/1.1
Host: sentinel-s2-l1c.s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210318T010842Z
Accept-Encoding: gzip

2021/03/18 01:08:42 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/18 01:08:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/18 01:08:42 DEBUG : HTTP RESPONSE (req 0xc0002e0200)
2021/03/18 01:08:42 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 18 Mar 2021 01:08:42 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: AupKTkY7U7IpjA7v5bdZrYevZQcBXfAe82FNCwl9VHUpRottCJpT6EIEWi2ZlX1H+fQPDF/+92g=
X-Amz-Request-Id: ENH0SDRG59FKKJWA

2021/03/18 01:08:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/18 01:08:42 DEBUG : 4 go routines active
2021/03/18 01:08:42 Failed to ls: AccessDenied: Access Denied
        status code: 403, request id: ENH0SDRG59FKKJWA, host id: AupKTkY7U7IpjA7v5bdZrYevZQcBXfAe82FNCwl9VHUpRottCJpT6EIEWi2ZlX1H+fQPDF/+92g=

asdffdsa · March 18, 2021, 1:17pm

ok, i have never used requester pays before.
rclone ls without requester-pays = true, does that work?

now that we have those logs, an rclone expert should stop by soon.

rman0604 · March 18, 2021, 5:10pm

Same response removing "requester-pays = true". I'd like to verify if rclone is actually sending "--request-payer requester" to AWS.

asdffdsa · March 18, 2021, 5:33pm

good question,
not an expert at this, if requester-pays adds a header, it should appear in the log
if i run
rclone lsd wasabi01: --header="X-Rclone: zzzz" --dump=headers -vv

i see this

DEBUG : HTTP REQUEST (req 0xc00022c500)
DEBUG : GET / HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.54.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210318T172957Z
X-Rclone: zzzz
Accept-Encoding: gzip

test using
--header="X-Amz-Request-Payer: requester"

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ObjectsinRequesterPaysBuckets.html
"Amazon S3 can return an Access Denied error for requests that try to get objects from a Requester Pays bucket. For more information, see Error Responses in the Amazon Simple Storage Service API Reference."

" If the GET request succeeds and the requester is charged, the response includes x-amz-request-charged:requester."

rman0604 · March 19, 2021, 7:09pm

Explicitly adding the "--header="X-Amz-Request-Payer: requester" helped but am now getting:

2021/03/19 18:59:39 Failed to ls: AccessDenied: There were headers present in the request which were not signed

User-Agent: rclone/v1.54.1
Authorization: AWS4-HMAC-SHA256 Credential=AKIAV6JYPXXZ4XIIESHW/20210319/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a542aebb5d1ef192572a6ed2b38e5bcffd443a0417edd76bd155f9433d6b0204
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210319T190613Z
X-Amz-Request-Payer: requester

I'm guessing the question now is how to add the "x-amz-request-payer" to the auth header?

asdffdsa · March 19, 2021, 7:12pm

post the command with a debug log.

rman0604 · March 22, 2021, 2:25pm

rclone lsd anon-eucentral1:sentinel-s2-l1c/tiles/ -vv --dump headers --header="X-Amz-Request-Payer: requester"
2021/03/22 14:21:06 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "lsd" "anon-eucentral1:sentinel-s2-l1c/tiles/" "-vv" "--dump" "headers" "--header=X-Amz-Request-Payer: requester"]
2021/03/22 14:21:06 DEBUG : Using config file from "C:\\Users\\xxx\\.config\\rclone\\rclone.conf"
2021/03/22 14:21:06 DEBUG : Creating backend with remote "anon-eucentral1:sentinel-s2-l1c/tiles/"
2021/03/22 14:21:06 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/22 14:21:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 14:21:06 DEBUG : HTTP REQUEST (req 0xc000669700)
2021/03/22 14:21:06 DEBUG : HEAD /tiles HTTP/1.1
Host: sentinel-s2-l1c.s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210322T142106Z
X-Amz-Request-Payer: requester

2021/03/22 14:21:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 14:21:07 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 14:21:07 DEBUG : HTTP RESPONSE (req 0xc000669700)
2021/03/22 14:21:07 DEBUG : HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/xml
Date: Mon, 22 Mar 2021 14:21:06 GMT
Server: AmazonS3
X-Amz-Id-2: 9O/BDaFrrGJHPY0cwMijX4hpwaB/fLTDkzxwQwHR4i9mqGDrrqe8uLdmmqw0Iviru79fXEB033U=
X-Amz-Request-Id: 9T9AGYHM33RTMDZT

2021/03/22 14:21:07 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 14:21:07 DEBUG : fs cache: renaming cache item "anon-eucentral1:sentinel-s2-l1c/tiles/" to be canonical "anon-eucentral1:sentinel-s2-l1c/tiles"
2021/03/22 14:21:07 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 14:21:07 DEBUG : HTTP REQUEST (req 0xc00071e200)
2021/03/22 14:21:07 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=1000&prefix=tiles%2F HTTP/1.1
Host: sentinel-s2-l1c.s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210322T142107Z
X-Amz-Request-Payer: requester
Accept-Encoding: gzip

2021/03/22 14:21:07 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 14:21:07 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 14:21:07 DEBUG : HTTP RESPONSE (req 0xc00071e200)
2021/03/22 14:21:07 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 22 Mar 2021 14:21:06 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: maLXn02WNAScLqsC9EvWg5P+j9cxGhAT48/3AS0YIpF86liq72065cVRkuj64JFcOqYuvDZWnXc=
X-Amz-Request-Id: 9T93EWRPEBKJ9G7E

2021/03/22 14:21:07 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 14:21:07 ERROR : : error listing: AccessDenied: There were headers present in the request which were not signed
        status code: 403, request id: 9T93EWRPEBKJ9G7E, host id: maLXn02WNAScLqsC9EvWg5P+j9cxGhAT48/3AS0YIpF86liq72065cVRkuj64JFcOqYuvDZWnXc=
2021/03/22 14:21:07 DEBUG : 4 go routines active
2021/03/22 14:21:07 Failed to lsd with 2 errors: last error was: AccessDenied: There were headers present in the request which were not signed
        status code: 403, request id: 9T93EWRPEBKJ9G7E, host id: maLXn02WNAScLqsC9EvWg5P+j9cxGhAT48/3AS0YIpF86liq72065cVRkuj64JFcOqYuvDZWnXc=

sweh · March 22, 2021, 6:09pm

Shouldn't that be requester_pays ?

rman0604 · March 23, 2021, 1:40pm

Yes.. SMH. I am a bit ignorant on back end flags. Assume when placed in the config file the hyphens are replaced with underscores and they lose the preceding "--" ?

Looking at the mount now, thank you.

sweh · March 23, 2021, 2:46pm

Kinda. You can see the details in the docs, at Amazon S3

--s3-requester-pays

Enables requester pays option when interacting with S3 bucket.

Config: requester_pays
Env Var: RCLONE_S3_REQUESTER_PAYS
Type: bool
Default: false

system · March 26, 2021, 2:46pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.