How to speed up Google Drive sync

Hi everyone,

I’d like to make a monthly backup of the entire Google Drive of my company (it’s a normal drive folder, not a G Suite / team drive). This is ~150k files, ~150GB, the large files are mostly photos and video, the small ones are google docs that get converted to .docx. I read somewhere that the API limit is 1000 files per 100 seconds, so small files should be done after 150000/10/3600 = ~4h. My internet connection typically downloads with 3-10 MB/s (~24-80Mbit/s), so 150GB would take 150*1024/3/3600 = 14h in the worst case. However, my backups tend to take around 50-200 hours.

I read Very slow sync to Google Drive but it seemed like it didn’t apply to my case.

I use the following command:
/Applications/rclone/rclone copy gdrive: /Users/…/Rclone/2019-01 --verbose --log-file=/Users/…/Rclone/2019-01.log

As you can see from the large number of checks, I had to restart the sync several times, and it’s still nowhere near done (excerpt from log file):

Transferred:   	   37.489G / 77.095 GBytes, 49%, 247.234 kBytes/s, ETA 46h39m34s
Errors:                50 (retrying may help)
Checks:            108006 / 108006, 100%
Transferred:        29768 / 39780, 75%
Elapsed time:  44h10m0.3s

I’m not pasting the log file because I don’t know how to efficiently remove all the lines with confidential information.

I’m using rclone v 1.45 64 bit on macOS Mojave 10.14.2 (18C54).

I’d really appreciate your help – thanks a lot everyone!

This is how I set up my remote (I entered advanced config but went for default values everywhere):

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> gdrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / A stackable unification remote, which can appear to merge the contents of several remotes
   \ "union"
 2 / Alias for a existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Dropbox
   \ "dropbox"
 9 / Encrypt/Decrypt a remote
   \ "crypt"
10 / FTP Connection
   \ "ftp"
11 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
12 / Google Drive
   \ "drive"
13 / Hubic
   \ "hubic"
14 / JottaCloud
   \ "jottacloud"
15 / Local Disk
   \ "local"
16 / Mega
   \ "mega"
17 / Microsoft Azure Blob Storage
   \ "azureblob"
18 / Microsoft OneDrive
   \ "onedrive"
19 / OpenDrive
   \ "opendrive"
20 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
21 / Pcloud
   \ "pcloud"
22 / QingCloud Object Storage
   \ "qingstor"
23 / SSH/SFTP Connection
   \ "sftp"
24 / Webdav
   \ "webdav"
25 / Yandex Disk
   \ "yandex"
26 / http Connection
   \ "http"
Storage> drive
** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id> 
Google Application Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret> 
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 1
ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders. (see docs).
Enter a string value. Press Enter for the default ("").
root_folder_id> 
Service Account Credentials JSON file path 
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_file> 
Edit advanced config? (y/n)
y) Yes
n) No
y/n> y
Service Account Credentials JSON blob
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_credentials> 
ID of the Team Drive
Enter a string value. Press Enter for the default ("").
team_drive> 
Only consider files owned by the authenticated user.
Enter a boolean value (true or false). Press Enter for the default ("false").
auth_owner_only> 
Send files to the trash instead of deleting permanently.
Defaults to true, namely sending files to the trash.
Use `--drive-use-trash=false` to delete files permanently instead.
Enter a boolean value (true or false). Press Enter for the default ("true").
use_trash> 
Skip google documents in all listings.
If given, gdocs practically become invisible to rclone.
Enter a boolean value (true or false). Press Enter for the default ("false").
skip_gdocs> 
Only show files that are shared with me.

Instructs rclone to operate on your "Shared with me" folder (where
Google Drive lets you access the files and folders others have shared
with you).

This works both with the "list" (lsd, lsl, etc) and the "copy"
commands (copy, sync, etc), and with all other commands too.
Enter a boolean value (true or false). Press Enter for the default ("false").
shared_with_me> 
Only show files that are in the trash.
This will show trashed files in their original directory structure.
Enter a boolean value (true or false). Press Enter for the default ("false").
trashed_only> 
Deprecated: see export_formats
Enter a string value. Press Enter for the default ("").
formats> 
Comma separated list of preferred formats for downloading Google docs.
Enter a string value. Press Enter for the default ("docx,xlsx,pptx,svg").
export_formats> 
Comma separated list of preferred formats for uploading Google docs.
Enter a string value. Press Enter for the default ("").
import_formats> 
Allow the filetype to change when uploading Google docs (e.g. file.doc to file.docx). This will confuse sync and reupload every time.
Enter a boolean value (true or false). Press Enter for the default ("false").
allow_import_name_change> 
Use file created date instead of modified date.,

Useful when downloading data and you want the creation date used in
place of the last modified date.

**WARNING**: This flag may have some unexpected consequences.

When uploading to your drive all files will be overwritten unless they
haven't been modified since their creation. And the inverse will occur
while downloading.  This side effect can be avoided by using the
"--checksum" flag.

This feature was implemented to retain photos capture date as recorded
by google photos. You will first need to check the "Create a Google
Photos folder" option in your google drive settings. You can then copy
or move the photos locally and use the date the image was taken
(created) set as the modification date.
Enter a boolean value (true or false). Press Enter for the default ("false").
use_created_date> 
Size of listing chunk 100-1000. 0 to disable.
Enter a signed integer. Press Enter for the default ("1000").
list_chunk> 
Impersonate this user when using a service account.
Enter a string value. Press Enter for the default ("").
impersonate> 
Use alternate export URLs for google documents export.,

If this option is set this instructs rclone to use an alternate set of
export URLs for drive documents.  Users have reported that the
official export URLs can't export large documents, whereas these
unofficial ones can.

See rclone issue [#2243](https://github.com/ncw/rclone/issues/2243) for background,
[this google drive issue](https://issuetracker.google.com/issues/36761333) and
[this helpful post](https://www.labnol.org/internet/direct-links-for-google-drive/28356/).
Enter a boolean value (true or false). Press Enter for the default ("false").
alternate_export> 
Cutoff for switching to chunked upload
Enter a size with suffix k,M,G,T. Press Enter for the default ("8M").
upload_cutoff> 
Upload chunk size. Must a power of 2 >= 256k.

Making this larger will improve performance, but note that each chunk
is buffered in memory one per transfer.

Reducing this will reduce memory usage but decrease performance.
Enter a size with suffix k,M,G,T. Press Enter for the default ("8M").
chunk_size> 
Set to allow files which return cannotDownloadAbusiveFile to be downloaded.

If downloading a file returns the error "This file has been identified
as malware or spam and cannot be downloaded" with the error code
"cannotDownloadAbusiveFile" then supply this flag to rclone to
indicate you acknowledge the risks of downloading the file and rclone
will download it anyway.
Enter a boolean value (true or false). Press Enter for the default ("false").
acknowledge_abuse> 
Keep new head revision of each file forever.
Enter a boolean value (true or false). Press Enter for the default ("false").
keep_revision_forever> 
If Object's are greater, use drive v2 API to download.
Enter a size with suffix k,M,G,T. Press Enter for the default ("off").
v2_download_min_size> 
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine or Y didn't work
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
Configure this as a team drive?
y) Yes
n) No
y/n> y
Fetching team drive list...
No team drives found in your account--------------------
[gdrive]
type = drive
scope = drive
token = {"access_token":"<removed>","token_type":"Bearer","refresh_token":"<removed>","expiry":"2019-01-16T15:32:57.235595+01:00"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
gdrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

try
rclone copy gdrive: /Users/…/Rclone/2019-01 --transfers=40 --checkers=40 --tpslimit=10 --drive-chunk-size=1M --verbose --log-file=/Users/…/Rclone/2019-01.log

try add --tpslimit 2 --bwlimit 9M to your command

I see you didn’t use your own client id - that is recommended for speeding things up.

Also try --fast-list if you’ve got lots of memory - that will speed up the initial recursion through the file system enormously.

Also you might like to increase this to > the number of files you have which will use more memory, but you’ll get accurate time to completions

  --max-backlog int   Maximum number of objects in sync or check backlog. (default 10000)

I am not sure if he could use your command. Isn’t Google limiting the concurrent connections to 10?
So checkers 40 and transfer 40 are not possible, right ? @ncw

I don’t know what the limit is. However it often seems to make things slower increasing the number of checkers/transfers.

He does not need bwlimit since whole storage is only 150GB and tpslimit 2 will severely limit transfer if there is tons of small files.

For example, Iam using 60 transfer’s from my local drive to cloude (rclone move), this is example with 42 ( since i dont have a log when i had more files on local before moved to cloud )

BEGINNING OF LOG

Transferred:   	    4.467G / 95.194 GBytes, 5%, 73.761 MBytes/s, ETA 20m59s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 43, 0%
Elapsed time:        1m2s
Transferring:
 *     movies/12 Gifts of C…15) - WEBDL-720p.mp4:  5% /1.381G, 1.271M/s, 17m27s
 *     movies/Angry Angel (…17) - WEBDL-720p.mkv:  3% /2.022G, 1.156M/s, 28m47s
 *     movies/Christmas in …16) - WEBDL-720p.mp4:  6% /1.158G, 1.146M/s, 16m6s
 *     movies/Christmas in …17) - WEBDL-720p.mp4:  7% /1.081G, 1.326M/s, 12m50s
 *     movies/Christmas on …18) - WEBDL-720p.mkv:  3% /2.316G, 1.342M/s, 28m22s
 *     movies/Crown for Chr…15) - WEBDL-720p.mp4:  6% /1.528G, 1.391M/s, 17m37s
 *     movies/Every Christm…16) - WEBDL-720p.mp4:  7% /1.071G, 1.304M/s, 12m58s
 *     movies/Falling for Y…18) - HDTV-1080p.mkv:  3% /3.918G, 2.124M/s, 30m18s
 *     movies/Ghost Stories…) - Bluray-1080p.mkv:  1% /8.276G, 1.892M/s, 1h13m18s
 *     movies/Honey (2003)/…) - Bluray-1080p.mkv:  0% /8.455G, 1.055M/s, 2h15m39s
 *     movies/My Christmas …16) - WEBDL-720p.mp4:  7% /1.149G, 1.408M/s, 12m50s
 *     movies/Plug Love (20…7) - WEBDL-1080p.mkv:  0% /11.176G, 1.406M/s, 2h14m32s
 *     movies/Steve Martin …8) - WEBDL-1080p.mkv:  7% /1.914G, 2.154M/s, 14m0s
 *     movies/The Mistletoe…17) - WEBDL-720p.mp4:  7% /1.073G, 1.235M/s, 13m41s
 *     movies/The Perfect C…17) - HDTV-1080p.mkv:  1% /3.978G, 1.124M/s, 59m14s
 *     series/Ancient Alien…E14 - WEBDL-720p.mkv:  5% /1.547G, 1.212M/s, 20m30s
 *     series/Ancient Alien…E15 - WEBDL-720p.mkv:  6% /1.548G, 1.232M/s, 19m58s
 *     series/Berlin Statio…E05 - WEBDL-720p.mkv: 14% /1.234G, 1.635M/s, 11m2s
 *     series/Black-ish/Sea…5E09 - HDTV-720p.mkv: 14% /723.833M, 1.149M/s, 9m0s
 *     series/Catastrophe (…E01 - HDTV-1080p.mkv:  5% /1.680G, 1.094M/s, 24m40s
 *     series/Graves/Season…03 - WEBDL-1080p.mkv:  9% /1.248G, 1.545M/s, 12m29s
 *     series/Graves/Season…04 - WEBDL-1080p.mkv:  6% /1.247G, 1.095M/s, 18m4s
 *     series/Graves/Season…05 - WEBDL-1080p.mkv:  8% /1.228G, 1.317M/s, 14m36s
 *     series/Graves/Season…06 - WEBDL-1080p.mkv:  7% /1.180G, 1.099M/s, 16m58s
 *     series/Graves/Season…07 - WEBDL-1080p.mkv:  9% /1.232G, 1.355M/s, 13m59s
 *     series/Graves/Season…08 - WEBDL-1080p.mkv: 10% /1.232G, 1.466M/s, 12m53s
 *     series/Graves/Season…09 - WEBDL-1080p.mkv:  8% /1.245G, 1.470M/s, 13m16s
 *     series/Graves/Season…10 - WEBDL-1080p.mkv: 11% /1.243G, 1.467M/s, 12m45s
 *     series/Late Night wi…07 - WEBDL-1080p.mkv:  7% /1.220G, 1.212M/s, 15m51s
 *     series/Lethal Weapon…11 - WEBDL-1080p.mkv:  6% /1.534G, 1.181M/s, 20m49s
 *     series/Living the Dr…E01 - HDTV-1080p.mkv: 10% /1.841G, 1.965M/s, 14m19s
 *     series/Manifest/Seas…10 - WEBDL-1080p.mkv: 14% /1.566G, 2.033M/s, 11m10s
 *     series/Modern Family…10 - WEBDL-1080p.mkv: 11% /870.647M, 1.330M/s, 9m37s
 *     series/NCIS- New Orl…5E11 - HDTV-720p.mkv: 11% /1.032G, 1.537M/s, 10m9s
 *     series/Project Blue …1E01 - HDTV-720p.mkv: 18% /751.362M, 1.591M/s, 6m22s
 *     series/Schitt's Cree…01 - WEBDL-1080p.mkv: 15% /890.006M, 1.439M/s, 8m41s
 *     series/The Gifted/Se…11 - WEBDL-1080p.mkv:  7% /1.529G, 1.590M/s, 15m7s
 *     series/The Late Show…07 - WEBDL-1080p.mkv: 11% /1.316G, 1.008M/s, 19m48s
 *     series/The Rookie/Se…1E09 - HDTV-720p.mkv: 12% /911.231M, 1.439M/s, 9m13s
 *     series/The Tonight S…07 - WEBDL-1080p.mkv:  9% /1.223G, 1.392M/s, 13m36s

END OF LOG

2019/01/09 07:09:19 INFO  : 
Transferred:   	   95.217G / 95.217 GBytes, 100%, 56.682 MBytes/s, ETA 0s
Errors:                 0
Checks:                43 / 43, 100%
Transferred:           43 / 43, 100%
Elapsed time:    28m40.1s

So try
rclone copy gdrive: /Users/…/Rclone/2019-01 --fast-list --transfers=40 --checkers=40 --tpslimit=10 --drive-chunk-size=1M --verbose --log-file=/Users/…/Rclone/2019-01.log

I strongly believe you can achieve like 90% of your internet limits if you take time and play around with settings above, eg you should finish the transfer within 14h 15h

If you wanna tweak it even more you could use experimental beta with additional flag --driver-pacer-burst
https://beta.rclone.org/branch/v1.45-103-gc9e50b32-fix-2880-drive-pacer-beta/
Read more about it here: https://github.com/ncw/rclone/issues/2880

Assuming you have gsuite and unlimited space it would not be bad idea to include versioning, so when you copy from local and overwrite the files on cloud you could keep previous version. To do that look into https://rclone.org/docs/#backup-dir-dir ( set --backup-dir gdrive:oldversion and add suffix with timestamp )

You could basically setup a system that will run 3, 4 times daily and upload only changes while preserving old versions in cloud.

Amazing, thanks so much everyone, really appreciate it! I’ll try these things and report back! :slight_smile:

Thanks a lot everyone, the entire backup now completes within ~3 hours at ~10MB/s! I’ve tested a few things and I think the main speed increase was due to setting up my own remote. The additional hassle seems clearly worth the effort.

I’m now using the following command:

rclone copy gdrive: /Users/…/Rclone/2019-02 --fast-list --transfers=40 --checkers=40 --tpslimit=10 --drive-chunk-size=1M --max-backlog 200000 --verbose --log-file=/Users/…/Rclone/2019-02.1.log

Also try --fast-list if you’ve got lots of memory - that will speed up the initial recursion through the file system enormously.

It seems that this doesn’t use a lot of memory, just up to ~150MB.

It works so much better now, thanks a lot! I’ll buy your wife some flowers :wink:

2 Likes

Hello, we cant' get it faster than 15KB/s with GSuite Drive. We created our own ID and using it.
All possible tweaks mentioned above didn't work. We have a connection which typically works on 50MB/s.
Here is what we get with RClone (v1.48.0):

Transferred: 834.834k / 3.681 GBytes, 0%, 13.705 kBytes/s, ETA 78h12m27s
Errors: 0
Checks: 0 / 0, -
Transferred: 78 / 10538, 1%
Elapsed time: 1m0.9s
Transferring:

This speed makes it unpractical to use. Anybody has any other suggestion in order to increase performance?

Here is also the googleapis stats

-- UPDATE --
The speed is as expected if I transfer big files however if I send a directory with small files (like a git repo) the speed makes crawl down to unusable levels
I also notice the %errors on oauth and lists

Please make a new post and do not bump old posts.