Currently Google Cloud Storage (GCS) storage backend endpoint defaults to the global endpoint storage.googleapis.com . This is good for compatibility, but bad for inter-region transfer fees of roughly $20 / TB . For example, transfer from a VM to a bucket in the same region is still at risk of the $20/TB inter-region fee due to global routing.
GCS does support the endpoint option for manual configuration, but rclone could do a better job to reduce the potential foot-guns for lay users.
Suggestions
Here are some options that could help users avoid inter-region transfer fees
Make region-specific endpoints the default. e.g. if user chooses us-west-1, endpoint is set to storage.us-west-1.googleapis.com
add a region-specific-endpoint [bool] option to enable this behavior
hi, are you 1000% sure about that?, have you done any testing to confirm?, is that documented anywhere?
google must know that vm and the bucket are in the same region/data-center and internally, would route it that way.
google must know that vm and the bucket are in the same region/data-center and internally, would route it that way.
you're right that it is meant to work this way, but there could be drain / disaster recovery test events that may alter routing, or other DNS variations.
I tested global (storage.googleapis.com) and regional (storage.us-west1.rep.googleapis.com) endpoints and waited 1 day to review bill. In this test the global endpoint was billed at $0.01/gb as "inter-zone transfer". The previous test was billed as "inter-region transfer". It seems that routing is inconsistent when using global endpoint.
Using global "storage.google.com" results in data transfer fee: sometimes inter-region and sometimes inter-zone rates.
Using regional endpoint bills as "intra-zone data transfer out" at no charge.
1.0 gb $0.00 / gb charge as "intra-zone data transfer out"
500 mb $0.01 / gb charge as "inter-zone data transfer out"
I recommend using region-specific endpoint to avoid data transfer rates of between $10 - $20 /TB in americas, up to $110/TB international rates
Steps to Reproduce
Create GCE VM & GCS bucket (single-region) in same region, e.g. us-west1
Create a "regional" configuration to make it easy to swap endpoints.
Confirm endpoint using --verbosity debug
Copy file
# run regional test
# create regional config
gcloud config configurations create regional
gcloud config set api_endpoint_overrides/storage https://storage.us-west1.rep.googleapis.com/
dd if=/dev/zero of=regional-1gb.txt bs=1M count=1000
# confirm endpoint
gcloud --verbosity debug storage ls $BUCKET
# copy file
gcloud storage cp regional-1gb.txt $BUCKET/test/
# run global test -- use default endpoint
gcloud config configurations activate default
gcloud --verbosity debug storage ls $BUCKET
dd if=/dev/zero of=global-500mb.txt bs=1M count=500
gcloud storage cp global-500mb.txt $BUCKET/test/
Billing details & SKUs
Network Intra Zone Data Transfer Out Compute Engine 14F9-7705-2FD4 0.49 gibibyte $0.00
Network Inter Zone Data Transfer Out Compute Engine DE9E-AFBC-A15A 1.02 gibibyte $0.01
I'm asking for help raising awareness with other interested users or maintainers .
Many people don't pay attention to their bill, they have auto-pay or they accept transfer fees as a side-effect of their account.
as i already wrote, i have no way validate your claims and nobody else is complaining.
you have posted your feature, time will tell if anyone else is interested.
fwiw, anybody is welcome to edit the rclone docs at github, easy to do.
maybe add two concise lines of text
the forum has many thousands of members, i am just a volunteer forum member.
what exactly do you expect from me???
to be honest, in eight days, i am the only forum member to respond to you and have done so multiple times.
including the suggestion that "anybody is welcome to edit the rclone docs at github, easy to do"
hopefully, someone with experience with google storage will stop by soon...
@Tony_M thank you for your test results - these are very useful.
If you run through the rclone config wizard then eventually you get to choose endpoint - leaving it blank gets the SDK to choose which endpoint to use.
Endpoint for the service.
Leave blank normally.
Enter a value. Press Enter to leave empty.
endpoint>
So I guess the SDK is making a sub-optimal choice.
A 3rd approach might be to move the endpoint setting into Standard (away from Advanced) config and give some choices for the endpoint setting, and some explanatory text talking about transfer fees for the default global region?
What do you think the explanatory text should say?
This would be very easy to implement
This would need some examples (see the other options) and the extra help and removing the Advanced: true line
I've learned a bit during testing . This transfer fee issue occurs while using GCE (compute engine VMs) . Those calling GCS from AWS or home network will naturally pay those rates (no extra transfer fees from GCS). So the audience would be GCE ↔ GCS users
A 3rd approach might be to move the endpoint setting into Standard (away from Advanced)
I'd like Rclone to remain as simple as possible. I would lean toward an "advanced" setting e.g. "use region-specific endpoint to reduce transfer fees in GCE". We could put a link to docs on the GCS backend setup to warn people about running it from GCE (see below).
This would be very easy to implement
I'm happy to prototype a PR since I did some earlier work in this space.
What do you think the explanatory text should say?
something like "The default endpoint for GCS may result in inter-region & inter-zone transfer fees when running rclone on Google Compute Engine (VMs). Setting region-specific-endpoint = true will reduce the risk of transfer fees. (i'll work on making it concise"
So I guess the SDK is making a sub-optimal choice.
GCP Billing Support explained that the GCS global endpoint routing can vary, so the region-specific endpoints are the most deterministic way to avoid the transfer fees.
The implementation seems to meet Google Cloud’s specs for the GCS calls. They could have done a better job highlighting the possible routing . The only docs on region-specific endpoints address compliance specs and not transfer fee issues.
I’ll share the PR around next week and thanks again for the guidance.