Any known scaling issues with Rclone and Google Drive?

Help and Support
1

What is the problem you are having with rclone?

I'm not experiencing any issues currently. I'm trying to anticipate potential problems with an upcoming Rclone project in which I will be moving about 2,000 personal drives representing about 10 TB of data from one Google tenant to another Google tenant. I plan to script this to run sequentially.

Are there any known issues to be aware of for a project of this size? Thanks!

What is your rclone version (output from rclone version)

I will use whatever version is current when this project begins in August.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Some version of Windows Server, 64-bit.

Which cloud storage system are you using? (eg Google Drive)

Google Drive source and target.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move Source: Target: --drive-server-side-across-configs

The rclone config contents with secrets removed.

I don't have this yet, as the project hasn't started. I anticipate it will be something very close to the default config.

A log from the command with the -vv flag

N/A

2
  • that is not a lot of data as it is spread over 2,000 drives.
  • you might consider renting a cheap virtual machine from google and run rclone on that.
  • have a debug log for each rclone move and have some way to parse for errors.
  • use service account files
  • can only transfer 750GB per account in a 24 hour period. rclone has ways to handle that.
3

Thank you for the information! Can you elaborate on the ways Rclone can handle the 750 GB/day/account limitation? I don't think I'm familiar with that one.

4

https://support.google.com/a/answer/172541?hl=en

imho, read the entire doc page first
https://rclone.org/drive/

1 Like
5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.