Rclone with GCS requester pays bucket

[A-Jacobson (Austin)]

What is the problem you are having with rclone?

Can't interact with gcs requester pays bucket through rclone.

I noticed there's some code mentioning this use case here (rclone/googlecloudstorage.go at master · rclone/rclone · GitHub) so I tried adding user_project to my config file but i still get the 400 error.

Ultimately, I want to transfer the contents of this bucket to an oci bucket.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: darwin 13.3.1 (64 bit)
- os/kernel: 22.4.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

gcs

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls gcs:<bucket>

The rclone config contents with secrets removed.

[gcs]

type = google cloud storage

project_number = <removed>

user_project = <removed>  # command results in same error with and without this set.

env_auth = false

token = <removed>

A log from the command with the -vv flag

~  rclone ls gcs://<bucket>/ -vv
DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "gcs://<bucket>/" "-vv"]
DEBUG : Creating backend with remote "gcs://<bucket>/
2023/05/23 19:27:21 DEBUG : Using config file from "/Users/austin/.config/rclone/rclone.conf"
2023/05/23 19:27:21 DEBUG : fs cache: renaming cache item gcs://<bucket>/" to be canonical "gcs:<bucket>/
2023/05/23 19:27:22 DEBUG : 5 go routines active
2023/05/23 19:27:22 Failed to ls: googleapi: Error 400: Bucket is a requester pays bucket but no user project provided., required

[ncw (Nick Craig-Wood)]

That should be working...

Can you do this and post the output please? (Redact as necessary).

rclone ls gcs:bucket -vv --dump bodies

[A-Jacobson (Austin)]

Hi Nick, thanks for the quick reply.

~  rclone ls gcs://bucket/ -vv --dump bodies
2023/05/24 10:59:05 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "gcs://bucket/" "-vv" "--dump" "bodies"]
2023/05/24 10:59:05 DEBUG : Creating backend with remote "gcs://bucket/"
2023/05/24 10:59:05 DEBUG : Using config file from "/Users/austin/.config/rclone/rclone.conf"
2023/05/24 10:59:05 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/05/24 10:59:05 DEBUG : fs cache: renaming cache item "gcs://bucket/" to be canonical "gcs:bucket/"
2023/05/24 10:59:05 DEBUG : gcs: Loaded invalid token from config file - ignoring
2023/05/24 10:59:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/24 10:59:05 DEBUG : HTTP REQUEST (req 0x1400057e300)
2023/05/24 10:59:05 DEBUG : POST /token HTTP/1.1
Host: oauth2.googleapis.com
User-Agent: rclone/v1.62.2
Content-Length: 235
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip

client_id=<clientid>.apps.googleusercontent.com&client_secret=<client_secret>&grant_type=refresh_token&refresh_token=<refresh_token>
2023/05/24 10:59:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/24 10:59:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/24 10:59:06 DEBUG : HTTP RESPONSE (req 0x1400057e300)
2023/05/24 10:59:06 DEBUG : HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=utf-8
Date: Wed, 24 May 2023 17:59:06 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: scaffolding on HTTPServer2
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "access_token": "<access_token>",
  "expires_in": 3599,
  "scope": "https://www.googleapis.com/auth/devstorage.read_write",
  "token_type": "Bearer"
}
2023/05/24 10:59:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/24 10:59:06 DEBUG : Saving config "token" in section "gcs" of the config file
2023/05/24 10:59:06 DEBUG : gcs: Saved new token in config file
2023/05/24 10:59:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/24 10:59:06 DEBUG : HTTP REQUEST (req 0x140008d0400)
2023/05/24 10:59:06 DEBUG : GET /storage/v1/b/<gcs:bucket>/o?alt=json&maxResults=1000&prefix=&prettyPrint=false HTTP/1.1
Host: storage.googleapis.com
User-Agent: rclone/v1.62.2
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.20.2 gdcl/0.112.0
Accept-Encoding: gzip

2023/05/24 10:59:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/24 10:59:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/24 10:59:06 DEBUG : HTTP RESPONSE (req 0x140008d0400)
2023/05/24 10:59:06 DEBUG : HTTP/2.0 400 Bad Request
Content-Length: 225
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Wed, 24 May 2023 17:59:06 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Guploader-Uploadid: "<uploadid>"

{"error":{"code":400,"message":"Bucket is a requester pays bucket but no user project provided.","errors":[{"message":"Bucket is a requester pays bucket but no user project provided.","domain":"global","reason":"required"}]}}
2023/05/24 10:59:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/24 10:59:06 DEBUG : 6 go routines active
2023/05/24 10:59:06 Failed to ls: googleapi: Error 400: Bucket is a requester pays bucket but no user project provided., required

[ncw (Nick Craig-Wood)]

And that is with the user project in the config? It didn't seem to make it to the request.

Try passing it as a command line flag maybe?

[A-Jacobson (Austin)]

rclone ls gcs:/bucket/ -vv --dump bodies --user_project <project_name>
Error: unknown flag: --user_project
Usage:
  rclone ls remote:path [flags]

Flags:
  -h, --help   help for ls

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

is it supposed to take user_project as a cmd like flag?

There isn't a mention of the arg when i run rclone config for gcs. I just added it to my config file because i noticed it in the go code.

[ncw (Nick Craig-Wood)]

Sorry should have been more explicit.

Use the flag --gcs-user-project 123123123 . This should be the same as setting user_project in the config file, but it is worth checking as the config file may be wrong somehow.

If you use that, then you should see it in -vv --dump bodies

GET /storage/v1/b/bucket/o?...&userProject=123123123 HTTP/1.1

We didn't see that in your dump above.

[A-Jacobson (Austin)]

No worries, thanks Nick. Here's what that gets us.

rclone ls gcs:/bucket/ -vv --dump bodies --gcs-user-project <project_name>
Error: unknown flag: --gcs-user-project
Usage:
  rclone ls remote:path [flags]

Flags:
  -h, --help   help for ls

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2023/05/25 18:26:07 Fatal error: unknown flag: --gcs-user-project

--gcs-user-project also isn't listed as a flag in Global Flags. Is it meant to be?

[Animosity022]

It isn’t a global flag so it won’t be there as it’s a backend specific flag.

Google Cloud Storage

Try without the equal perhaps.

[A-Jacobson (Austin)]

--gcs-user-project isnt listed in the gcs flag docs either.

which equal are you referring to? this is the command i just ran:

rclone ls gcs:/bucket/ -vv --dump bodies --gcs-user-project <project_name>

[Animosity022]

Sorry as I mean to link directly to the flag.

Google Cloud Storage

[A-Jacobson (Austin)]

I don't believe project-number works with requester pays buckets. that's why I've been specifically trying user-project There's nothing about it in the docs but it seems supported in the source code as mentioned above.

[Animosity022]

Is there harm to trying? Easier to just see if @ncw typo'ed or that is it.

[A-Jacobson (Austin)]

My mistake, I wasn't clear. I followed the docs to use --gcs-project-number + specifying a project-number in my config before opening the this thread. The google api seems to specifically want a user_project

[ncw (Nick Craig-Wood)]

@A-Jacobson

--gcs-user-project isnt listed in the gcs flag docs either.

$ rclone help flags user-project
[snip]
Backend Flags:
      --gcs-user-project string   User project

@A-Jacobson

My mistake, I wasn't clear. I followed the docs to use --gcs-project-number + specifying a project-number in my config before opening the this thread. The google api seems to specifically want a user_project

The --gcs-project-number is needed for creating buckets.

The --gcs-user-project is needed for payer buckets.

@A-Jacobson

rclone ls gcs:/bucket/ -vv --dump bodies --gcs-user-project <project_name>
Error: unknown flag: --gcs-user-project

This works fine for me which indicates to me that you are running an old version of rclone.

What does rclone version say on the same machine that you tried the above on?

[kapitainsky]

googlecloudstorage: added gcs requester pays

has not landed in the latest rclone v1.62.2 used (as per original post)

@A-Jacobson try the latest beta v1.63.0 from:

https://beta.rclone.org/

this looks like is one ncw is using. Hence miscommunication

[ncw (Nick Craig-Wood)]

@kapitainsky

this looks like is one ncw is using. Hence miscommunication

Thanks for working that out! And sorry for confusing everyone.

[A-Jacobson (Austin)]

Works on beta! Thanks guys. Appreciate the help.