How to transfer to AWS without incurring egress charges (from within an EC2 instance)?

What is the problem you are having with rclone?

Sending OUT of aws cloud is expensive, but transfers IN are free - how to set up rclone to download directly INTO aws S3 within the "vpc" cloud and not over the internet when running rclone from within the target cloud (from an EC2 instance or "aws CloudShell").

Is it the "Endpoint for S3 API" option? If anyone knows any documentation on how to set it up, it would be really helpful.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: amazon 2 (64 bit)
- os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive ---> S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy gdrive: aws-glacier-deep:

The rclone config contents with secrets removed.

type = drive
client_id = <redacted>
client_secret = <redacted>
scope = drive
token = <redacted>
root_folder_id = <redacted>
team_drive = 

type = s3
provider = AWS
access_key_id = <redacted>
secret_access_key = <redacted>
region = eu-west-2
location_constraint = eu-west-2
acl = private
storage_class = DEEP_ARCHIVE

So in this case you would want to use a Gateway Endpoint, which will essentially create a tunnel directly to S3 so traffic is not egressed from your VPC. From the documentation, a gateway endpoint will not incur any additional costs and does not have any per GB transfer costs.

How gateway endpoints essentially work is by intercepting traffic at the VPC layer by injecting prefix lists into your specified route tables to direct traffic to S3's logical IP addresses through those gateways (so it doesn't require any type of client side configuration changes, you'd just use rclone as is with S3 API).

If you are familiar with Terraform, there are a handful of modules that will facilitate creating a VPC gateway (endpoint) that will allow you to achieve what you are looking for.

As always, you will be charged for your EC2 and S3 usage, but will not incur anything data transfer charges into S3 for these transfers.


That is clever!

I thought traffic from EC2 to S3 was free provided they are in the same region and you didn't need to do anything special to ensure that?

(Not an AWS expert!)

1 Like

Thank you, this is a fantastic advice and a really good explanation. I've dived deeper into this subject and found "Interface Endpoints" as well, which seem to be used for other AWS services (other than S3 and DynamoDB) or when connecting from a hybrid/on-premises server, but these do incur additional costs are require applications to specify their endpoints. It's nice that Gateway Endpoints like you described will "intercept" rclone automagically.

I find virtual networking in the clouds very intimidating, so your advise today saved me tons of time researching this subject :slight_smile: .

@ncw you probably hear this all the time, but thank you for your time and effort on developing rclone - it's a remarkable piece of software!

1 Like

Inherently, no. This would mostly be due to the fact that when you connect to S3, it's calling the public facing API for S3 and then uses either a NAT gateway (private instance) or Internet Gateway (public instance) to connect. It sounds like it should be free, but as explained above it's a few more steps to get it that way.

Networking egress and ingress charges in, out of, and between AWS services is a hellscape. It's a sore point for most people because there's always a "Well, not exactly..." clause to each of those avenues.

Thanks for explaining @davidjameshowell

Is there something we could be doing in rclone to make this easier? Use the internal endpoints for S3 somehow?

Not inherently within rclone as it is a more technical setup process (not difficult, but you need to know some aspects of VPC, networking, and the endpoints themselves).

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.