Sending OUT of aws cloud is expensive, but transfers IN are free - how to set up rclone to download directly INTO aws S3 within the "vpc" cloud and not over the internet when running rclone from within the target cloud (from an EC2 instance or "aws CloudShell").
Is it the "Endpoint for S3 API" option? If anyone knows any documentation on how to set it up, it would be really helpful.
Run the command 'rclone version' and share the full output of the command.
So in this case you would want to use a Gateway Endpoint, which will essentially create a tunnel directly to S3 so traffic is not egressed from your VPC. From the documentation, a gateway endpoint will not incur any additional costs and does not have any per GB transfer costs.
How gateway endpoints essentially work is by intercepting traffic at the VPC layer by injecting prefix lists into your specified route tables to direct traffic to S3's logical IP addresses through those gateways (so it doesn't require any type of client side configuration changes, you'd just use rclone as is with S3 API).
If you are familiar with Terraform, there are a handful of modules that will facilitate creating a VPC gateway (endpoint) that will allow you to achieve what you are looking for.
As always, you will be charged for your EC2 and S3 usage, but will not incur anything data transfer charges into S3 for these transfers.
Thank you, this is a fantastic advice and a really good explanation. I've dived deeper into this subject and found "Interface Endpoints" as well, which seem to be used for other AWS services (other than S3 and DynamoDB) or when connecting from a hybrid/on-premises server, but these do incur additional costs are require applications to specify their endpoints. It's nice that Gateway Endpoints like you described will "intercept" rclone automagically.
I find virtual networking in the clouds very intimidating, so your advise today saved me tons of time researching this subject .
@ncw you probably hear this all the time, but thank you for your time and effort on developing rclone - it's a remarkable piece of software!
Inherently, no. This would mostly be due to the fact that when you connect to S3, it's calling the public facing API for S3 and then uses either a NAT gateway (private instance) or Internet Gateway (public instance) to connect. It sounds like it should be free, but as explained above it's a few more steps to get it that way.
Networking egress and ingress charges in, out of, and between AWS services is a hellscape. It's a sore point for most people because there's always a "Well, not exactly..." clause to each of those avenues.
Not inherently within rclone as it is a more technical setup process (not difficult, but you need to know some aspects of VPC, networking, and the endpoints themselves).
.