[GCS] Use region-specific endpoints to avoid inter-region transfer fees of $20/TB

Currently Google Cloud Storage (GCS) storage backend endpoint defaults to the global endpoint storage.googleapis.com . This is good for compatibility, but bad for inter-region transfer fees of roughly $20 / TB . For example, transfer from a VM to a bucket in the same region is still at risk of the $20/TB inter-region fee due to global routing.

GCS does support the endpoint option for manual configuration, but rclone could do a better job to reduce the potential foot-guns for lay users.

Suggestions

Here are some options that could help users avoid inter-region transfer fees

  1. Make region-specific endpoints the default. e.g. if user chooses us-west-1, endpoint is set to storage.us-west-1.googleapis.com
  2. add a region-specific-endpoint [bool] option to enable this behavior

What does the community think of this approach?

More Info

Evidence

$  rclone --dump-headers ls --verbose gcs://$BUCKET 2>&1 |grep -i host
Host: oauth2.googleapis.com
Host: storage.googleapis.com

Steps to Repro

  1. launch VM in us-west-1
  2. write 5GB to bucket in us-west-1
  3. wait 48 hours for billing
  4. check billing report "group by SKU"

Actual Results

Network Inter Region Data Transfer Out from Americas to Americas	Compute Engine	C8EA-1A86-3D28	6.29 gibibyte	$0.11	

hi, are you 1000% sure about that?, have you done any testing to confirm?, is that documented anywhere?
google must know that vm and the bucket are in the same region/data-center and internally, would route it that way.

yes.
Here was a test run from VM and bucket both within us-west-1

Network Inter Region Data Transfer Out from Americas to Americas Compute Engine C8EA-1A86-3D28 6.29 gibibyte $0.11

google must know that vm and the bucket are in the same region/data-center and internally, would route it that way.

you're right that it is meant to work this way, but there could be drain / disaster recovery test events that may alter routing, or other DNS variations.

DNS debug info showing that storage.googleapis.com != storage.us-west-1.googleapis.com

 dig +short storage.googleapis.com
74.125.197.207
173.194.202.207
142.250.107.207
173.194.203.207
108.177.98.207
74.125.20.207
74.125.135.207
74.125.142.207
142.250.99.207
172.253.117.207
74.125.195.207
antho@test-routing-1:~$ dig +short storage.us-west-1.googleapis.com
173.194.202.95
142.250.99.95
74.125.135.95
74.125.20.95
172.253.117.95
173.194.203.95
108.177.98.95
74.125.195.95
142.250.107.95
74.125.197.95
74.125.142.95

Updated with proper region-specific endpoint format : storage.us-west1.rep.googleapis.com

~$ dig +short storage.us-west1.rep.googleapis.com
34.1.67.247
antho@test-routing-1:~$ dig +short storage.googleapis.com
74.125.142.207
74.125.197.207
74.125.195.207
172.253.117.207
74.125.135.207
173.194.203.207
74.125.20.207
108.177.98.207
173.194.202.207
142.250.107.207
142.250.99.207

Traceroute

traceroute is inconclusive but seems like a shorter route to region-specific-endpoint

# storage.us-west1.rep.googleapis.com
$ traceroute -n 34.1.67.247
traceroute to 34.1.67.247 (34.1.67.247), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  34.1.67.247  0.309 ms  0.290 ms  0.189 ms
# storage.googleapis.com
antho@test-routing-1:~$ traceroute -n 74.125.142.207
traceroute to 74.125.142.207 (74.125.142.207), 30 hops max, 60 byte packets
 1  * * *
 2  209.85.143.96  1.713 ms 209.85.143.228  1.742 ms 209.85.143.94  1.293 ms
 3  142.251.252.190  3.525 ms  2.108 ms  1.493 ms
 4  172.253.78.155  1.071 ms 172.253.79.27  1.611 ms 172.253.78.155  2.058 ms
 5  66.249.94.227  4.757 ms 216.239.43.117  2.075 ms 74.125.251.249  1.990 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  74.125.142.207  0.253 ms  0.303 ms  0.298 ms

can you run the tests using a larger value, such as 5GiB?

why? we already have a compelling case

Inter-zone / Inter-Region Fees Experiment

I tested global (storage.googleapis.com) and regional (storage.us-west1.rep.googleapis.com) endpoints and waited 1 day to review bill. In this test the global endpoint was billed at $0.01/gb as "inter-zone transfer". The previous test was billed as "inter-region transfer". It seems that routing is inconsistent when using global endpoint.

Methods

  1. 1 GB transferred using region endpoint storage.us-west1.rep.googleapis.com
  2. 500 MB transferred using global endpoint storage.googleapis.com

Results & Conclusions

Using global "storage.google.com" results in data transfer fee: sometimes inter-region and sometimes inter-zone rates.
Using regional endpoint bills as "intra-zone data transfer out" at no charge.

  • 1.0 gb $0.00 / gb charge as "intra-zone data transfer out"
  • 500 mb $0.01 / gb charge as "inter-zone data transfer out"

I recommend using region-specific endpoint to avoid data transfer rates of between $10 - $20 /TB in americas, up to $110/TB international rates

Steps to Reproduce

  1. Create GCE VM & GCS bucket (single-region) in same region, e.g. us-west1
  2. Create a "regional" configuration to make it easy to swap endpoints.
  3. Confirm endpoint using --verbosity debug
  4. Copy file
# run regional test
# create regional config
gcloud config configurations create regional
gcloud config set api_endpoint_overrides/storage https://storage.us-west1.rep.googleapis.com/
dd if=/dev/zero of=regional-1gb.txt bs=1M count=1000
# confirm endpoint
gcloud --verbosity debug storage ls $BUCKET
# copy file
gcloud storage cp regional-1gb.txt  $BUCKET/test/


# run global test -- use default endpoint
gcloud config configurations activate default
gcloud --verbosity debug storage ls $BUCKET
dd if=/dev/zero of=global-500mb.txt bs=1M count=500
gcloud storage cp global-500mb.txt  $BUCKET/test/

Billing details & SKUs

Network Intra Zone Data Transfer Out	Compute Engine	14F9-7705-2FD4	0.49 gibibyte	$0.00	

Network Inter Zone Data Transfer Out	Compute Engine	DE9E-AFBC-A15A	1.02 gibibyte	$0.01

edit updated to fix typo in methods

@asdffdsa can you help me get more visibility on this case?

sorry, i do not use google products, cannot test+validate your claims.

fwiw, strange that no other rclone users are complaining about shocking unexpected charges??

I'm asking for help raising awareness with other interested users or maintainers .
Many people don't pay attention to their bill, they have auto-pay or they accept transfer fees as a side-effect of their account.

as i already wrote, i have no way validate your claims and nobody else is complaining.
you have posted your feature, time will tell if anyone else is interested.


fwiw, anybody is welcome to edit the rclone docs at github, easy to do.
maybe add two concise lines of text

  1. warning
  2. workaround

Thank you , but from my perspective you have not been helpful.

the forum has many thousands of members, i am just a volunteer forum member.
what exactly do you expect from me???

to be honest, in eight days, i am the only forum member to respond to you and have done so multiple times.
including the suggestion that "anybody is welcome to edit the rclone docs at github, easy to do"

hopefully, someone with experience with google storage will stop by soon...

@Tony_M thank you for your test results - these are very useful.

If you run through the rclone config wizard then eventually you get to choose endpoint - leaving it blank gets the SDK to choose which endpoint to use.

Endpoint for the service.
Leave blank normally.
Enter a value. Press Enter to leave empty.
endpoint> 

So I guess the SDK is making a sub-optimal choice.

A 3rd approach might be to move the endpoint setting into Standard (away from Advanced) config and give some choices for the endpoint setting, and some explanatory text talking about transfer fees for the default global region?

What do you think the explanatory text should say?

This would be very easy to implement

This would need some examples (see the other options) and the extra help and removing the Advanced: true line

I've learned a bit during testing . This transfer fee issue occurs while using GCE (compute engine VMs) . Those calling GCS from AWS or home network will naturally pay those rates (no extra transfer fees from GCS). So the audience would be GCE ↔ GCS users

A 3rd approach might be to move the endpoint setting into Standard (away from Advanced)

I'd like Rclone to remain as simple as possible. I would lean toward an "advanced" setting e.g. "use region-specific endpoint to reduce transfer fees in GCE". We could put a link to docs on the GCS backend setup to warn people about running it from GCE (see below).

This would be very easy to implement

I'm happy to prototype a PR since I did some earlier work in this space.

What do you think the explanatory text should say?

something like "The default endpoint for GCS may result in inter-region & inter-zone transfer fees when running rclone on Google Compute Engine (VMs). Setting region-specific-endpoint = true will reduce the risk of transfer fees. (i'll work on making it concise"

So I guess the SDK is making a sub-optimal choice.

GCP Billing Support explained that the GCS global endpoint routing can vary, so the region-specific endpoints are the most deterministic way to avoid the transfer fees.

That sounds like a good plan and I'd be very happy to accept a PR to fix.

When I did the original backend I assumed that it selected the best endpoint like the aws s3 SDK does.

If you need help with the PR then ping me here or probably more reliably submit the PR and ask there.

Thank you

The implementation seems to meet Google Cloud’s specs for the GCS calls. They could have done a better job highlighting the possible routing . The only docs on region-specific endpoints address compliance specs and not transfer fee issues.

I’ll share the PR around next week and thanks again for the guidance.

1 Like

i had other projects come up but it's still on the radar. i'll have updates soon.

1 Like