IPV6 Guide & Review with Google Drive , EC2 & S3

Howto Guides
1

My regular backup of ~125GB from Google Drive to S3 via EC2 flow looks like this

  • Google Takeout (10 GB ZIP):arrow_right: Google Drive (/Takeout) :arrow_right: EC2 (IPV6-only host , used to verify contents) :arrow_right: S3 (Glacier Deep Archive).

I go this route for a few reasons

  • cloud to cloud is extremely fast . < 1 hr vs days using home ISP
  • Ec2 step let's me verify the archives
  • very easy to re-start or correct if something goes wrong.

I used IPV6 on my EC2 instance since AWS is now charging ~ $5/mo for IPV4. rclone had a few compatibility issues with IPV6.

  • oauth authorization on Google Drive assumed IPV4
  • default Amazon AWS S3 endpoint does not support IPV6

Here is a guide on how to use rclone on an IPV6-only server (no IPV4 addressing).

Authorization

rclone assumes oauth redirect URL will be IPV4 . To workaround , I ran rclone from my desktop and copied the tokens to the EC2 instance ( /.config/rclone/rclone.conf )

I tried updating the oauth server & token URL to use IPV6 but that had no effect.

The blocker was SSH hosting the port-forward IPV6 but auth URL was set by rclone to http://127.0.0.1/...

Connecting to Google Drive
Once authorized, rclone backend for Google Drive transfer supports IPV6 out of the box. No special config was needed.

rclone copy gcloud:/Takeout ./

Performance was very good, at around 60mbps (30 min for 120gb)

Connecting to S3

S3 & Amazon AWS API does not support IPV6 by default. Instead "dual-stack" endpoints must be used.

Configure rclone as usual, note your desired region (e.g. us-west-2) and update the S3 endpoint in your rclone.conf

region = us-west-2
location_constraint = us-west-2
endpoint = s3.dualstack.us-west-2.amazonaws.com

Testing The Transfer
rclone copy --progress 2023-11-30-backup s3-backup/xxx-backup/

rclone should copy to S3 as usual. If it hangs, use the AWS cli (configured above with "dual-stack" endpoints) to confirm you can connect to S3 endpoints & list buckets

[optional] Testing Using AWS CLI

~/.aws/config

[default]
s3 =
    use_dualstack_endpoint = true
    addressing_style = virtual

 aws s3api list-buckets
 # buckets should list here
3 Likes
2

Did you try the binding function to see if you could force rclone to use IPv6?

2 Likes
3

I tried --bind <IPV6 addr> but the redirect URL still used IPV4

rclone --bind 2600:1f14:90d:xxxx config

NOTICE: Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config.

4

thanks for the help. I put in a PR for the s3 endpoint support. See the "IPv6 Discussion" for some of the other points e.g.

Add S3 IPv6 "dualstack" endpoint support . Option = "dualstack_endpoint [bool]" by tonymet · Pull Request #7482 · rclone/rclone (github.com)

IPv6 Support Could Use Attention...

  1. :white_check_mark:--bind to IPv6
  2. :x:oauth using IPv6 (::1)
  3. (this PR) :x:Connect to S3 endpoint on IPv6
  4. :white_check_mark: Connect to Google Drive API endpoint on IPv6 (once authorized )
  5. :question:Connect to other backend endpoints on IPv6

Let me know if it's worthy of discussion i could open a forum post to see if this is the right approach.

1 Like
5

Note that the redirect URL is only used on the computer with the web browser. Are you running the web browser part of the oauth login on your EC2 instance?

Does it really not have a 127.0.0.1 address?

Exactly what address you can put there is quite tightly controlled by the oauth provider. Most providers will only let you use http if you use 127.0.0.1 or localhost. Using localhost would be perfect as that should cover IPv6 and IPv4 but I don't know if drive supports that. I haven't tried using http://[::]:53682/ or any of the IPv6 names for localhost

You can set your redirect URL in your client config (assuming you made a client) but rclone doesn't currently have a setting to allow you to match it in rclone.

All the ways of doing this remote setup are described here: Remote Setup

6

Does it really not have a 127.0.0.1 address?

In this case the ssh tunnel only listens on ::1 . The machine showing the oAuth screen needed redirect_uri = http://[::1]:53682 .

All the ways of doing this remote setup are described here: Remote Setup

I saw that and thanks for the great docs! But i would still call that a “workaround” rather than supporting IPv6 .

Once complete this will be seamless. That’s why I’m trying to help with some improved testing.

7

Ah, OK!

Making the redirect URL IPv6 friendly is a bit annoying as I said above, unless http://localhost will work?