Googlecloudstorage custom endpoint subpath removed for upload

onjen · November 27, 2025, 4:00pm

Problem

In the Google cloud storage integration there is a problem when specifying a custom endpoint with a subpath, the upload ignores this subpath. For example example.org/custom/endpoint on upload the /custom/endpoint is not reflected.

There is an open issue with the underlying Google storage API go client library which is most likely related.

github.com/googleapis/google-api-go-client

Path replacement of endpoint in Drive /upload operation

opened 06:59PM - 04 Jun 24 UTC

denisazevedo

type: question priority: p3

When we create a service **with endpoint** ([option#WithEndpoint](https://pkg.go….dev/google.golang.org/api/option#WithEndpoint)) using host and path, ie: `https://myhost.com/connections/google/` ```go service, err := drive.NewService(ctx, option.WithHTTPClient(client)) ... tokenSource := option.WithTokenSource(oauth2.StaticTokenSource(&oauth2.Token{AccessToken: *token})) endpointOptions := option.WithEndpoint("https://myhost.com/connections/google/") options := []option.ClientOption{ tokenSource, endpointOptions, } ``` it works well for operations like folder creation, as the URL will replaced by: `https://myhost.com/connections/google/files` Replaced by: https://github.com/googleapis/google-api-go-client/blob/007774894a48d24634ace7b9e09def4d61433f7f/drive/v3/drive-gen.go#L5520 But for media upload operations, like file creation, it is removing our endpoint path here: https://github.com/googleapis/google-api-go-client/blob/007774894a48d24634ace7b9e09def4d61433f7f/drive/v3/drive-gen.go#L5522 Instead of calling this: `https://myhost.com/connections/google/upload/drive/v3/files` it's calling: `https://myhost.com/upload/drive/v3/files` The reason is the slash `/` at the beggining of the `relstr` during the `googleapi.ResolveRelative` func call. This is the only operation that starts with slash, all others (like _files_, _comments_, etc) are appended to our custom endpoint. Is it an issue or intentional?

As this is most likely an issue with the underlying API client, there is no way to fix this in rclone. My suggestion would be to add a hint in the googlecloudstorage docs on that endpoint parameter, that uploads won’t work if the custom endpoint contains paths.

How to reproduce

Running mitmproxy to inspect the requests by rclone.

Set proxy env variables

export HTTP_PROXY="localhost:8080"
export HTTPS_PROXY=$HTTP_PROXY

download

rclone copy -v --gcs-endpoint "example.org/custom/endpoint" --gcs-no-check-bucket --no-check-dest --gcs-bucket-policy-only :gcs:gcs-bucket-name/foo /tmp/foo

In mitmproxy the request contains the subpath

GET  example.org/custom/endpoint/storage/v1/b/gcs-bucket-name/o/foo?alt=json&prettyPrint=false

listing

rclone ls  --gcs-endpoint "example.org/custom/endpoint" :gcs:gcs-bucket-name/foo

In mitmproxy the request contains the subpath

HTTPS GET example.org/custom/endpoint/storage/v1/b/gcs-bucket-name/o?alt=json&maxResults=1000&prefix=foo%2F&prettyPrint=false

upload

rclone copy -v /tmp/foo --gcs-endpoint "example.org/custom/endpoint" --gcs-no-check-bucket --no-check-dest --gcs-bucket-policy-only :gcs:bucket-name/

In mitmproxy you see the /custom/endpoint part of the custom endpoint is removed.

HTTPS POST  example.org/upload/storage/v1/b/gcs-bucket-name/o?alt=json&name=foo%2Ffoo&prettyPrint=false&uploadType=multipart

Versions

Reproducible with different rclone versions (with 1.60.0 being the first one to introduce the endpoint parameter on googlecloudstorage)

rclone v1.60.0
- os/version: debian forky/sid (64 bit)
- os/kernel: 6.16.12+deb14+1-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.2
- go/linking: static
- go/tags: none

rclone v1.69.1
- os/version: debian forky/sid (64 bit)
- os/kernel: 6.16.12+deb14+1-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: static
- go/tags: none

rclone v1.72.0
- os/version: debian forky/sid (64 bit)
- os/kernel: 6.16.12+deb14+1-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.25.4
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (e.g. Google Drive)

Google Cloud Storage

ncw · November 27, 2025, 5:20pm

That is annoying and as it is in the generated part of the code it will be annoying to fix.

However they do take pull requests (apparently I got two merged 11 years ago now!). Not sure what the fix is, but it will probably be something quite simple in the generator.

Do you want to make a pull request for this?

The best change would likely be in the help for endpoint

github.com/rclone/rclone

backend/googlecloudstorage/googlecloudstorage.go

f72b32b47


      
          set. Normally rclone will download these files as compressed objects.
          
          If this flag is set then rclone will decompress these files with
          "Content-Encoding: gzip" as they are received. This means that rclone
          can't check the size and hash but the file contents will be decompressed.
          `,
          			Advanced: true,
          			Default:  false,
          		}, {
          			Name:     "endpoint",
          			Help:     "Endpoint for the service.\n\nLeave blank normally.",
          			Advanced: true,
          		}, {
          			Name:     config.ConfigEncoding,
          			Help:     config.ConfigEncodingHelp,
          			Advanced: true,
          			Default: (encoder.Base |
          				encoder.EncodeCrLf |
          				encoder.EncodeInvalidUtf8),
          		}, {
          			Name:    "env_auth",

(This gets put into the docs automatically)

onjen · November 27, 2025, 9:18pm

Do you want to make a pull request for this?

Sure!

github.com/rclone/rclone

googlecloudstorage: improve endpoint parameter docs

master ← onjen:gcs-endpoint-improved-docs

opened 09:17PM - 27 Nov 25 UTC

onjen

+18 -1

When specifying a custom endpoint with a subpath, there is a limitation in the G…oogle cloud storage integration that the subpath is ignored during upload operations. For example with the custom endpoint "example.org/custom/endpoint" on upload the /custom/endpoint is not reflected. As this is most likely an issue with the underlying API client, there is no way to fix this in rclone. By extending the documentation at least rclone users are made aware of this limitation. Related forum thread: https://forum.rclone.org/t/googlecloudstorage-custom-endpoint-subpath-removed-for-upload/53059 This is the rendered documentation <img width="1364" height="615" alt="image" src="https://github.com/user-attachments/assets/e8ca2551-b231-4f27-ac27-0d5354984a6f" /> - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix). - [x] I have added tests for all changes in this PR if appropriate. - [x] I have added documentation for the changes if appropriate. - [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [x] I'm done, this Pull Request is ready for review :-)

system · December 4, 2025, 7:46am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.