Seeking advice on improving transfer rate for GDrive -> S3

What is the problem you are having with rclone?

I am transferring a large number of data from my Google Workspace tenant to AWS S3 and am not getting close to the theoretical 10TiB/day limit. The Google Drive users I'm copying have a huge number of small files, so I suspect I'm being limited by GDrive API limits, but I wanted to see if anyone had any advice on optimization tweaks I might be able to make.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.76.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive -> AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --drive-impersonate REDACTED --drive-auth-owner-only --s3-disable-http2 --s3-no-check-bucket --checksum --transfers=60 --s3-upload-concurrency=8 --s3-chunk-size=16M --stats-log-level NOTICE --stats 10000h copy gdrive: s3:gwe-backups/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

type = drive
scope = drive
client_id = XXX
client_secret = XXX
service_account_file = XXX
team_drive = 

type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-west-2
storage_class = GLACIER_IR

A log from the command that you were trying to run with the -vv flag

N/A - not seeing a specific error, just seeking advice

Best bet would be to see the log file to see if you are being limited or not being limited.

If not, you'd want to crank up things.
If yes, you'd wnat to crank down things.

Depends on what the log says.

Please pardon my ignorance - what log level is the message RE: GDrive rate limiting? I haven't seen anything of that nature, just NOTICEs about dupes in the source and occasional ERRORs about the OAuth token being expired which resolves itself.

Usually in debug.

429s are generally rate limiting and also quota messages.

Thank you. I will re-run my next transfer with debug logging enabled and take a look.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.