Sync between team drives - hitting rate limits

What is the problem you are having with rclone?

Tryiing to sync data between 2 drives and after approx 3-3.5 TB I get rate limit errors and can't do anything until the next day.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0

  • os/version: ubuntu 18.04 (64 bit)
  • os/kernel: 5.6.0-050600-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: static
  • go/tags: none
No 1 minor release back

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 /usr/bin/rclone sync "tv:Media/TV" "saltbox347323tv:Media/tv3" --drive-service-account-file  /opt/sa/vaulted/7.json  -vvvP --transfers=16 --drive-server-side-across-configs

The rclone config contents with secrets removed.

[movie]
type = drive
scope = drive
team_drive = 0AJdYVATZ8eOQUk9PVA
service_account_file = /opt/sa/sasync/93.json

[google]
type = union
upstreams = movie: tv: saltbox347323tv:
token = {"access_token":"ya29.A0ARrdaM8j4idv-KoVB8jQVU07pEgR1w4DIOxqrnCssRzkBd5CzUad-dL31rxIWZ1zDXUcq26gSlQnN3rx-bdbPmcEaIf5mHc4X98I9dv-12etLaeA4Pe6vEwAKR6jcEEVbrjvOqJ0UarM7lxsaZo7X44JlWUO2UA","token_type":"Bearer","refresh_token":"1//03RpZx-BgVtasCgYIARAAGAMSNwF-L9IrzBztXPwSyj1m3lBkKmjCMekynsPaIJFvMV5Nda8hZUAZ16Q_EAoeO4qTAt-YqRKT8PQ","expiry":"2022-03-30T00:49:04.77086765+02:00"}

[saltbox347323tv]
type = drive
scope = drive
service_account_file = /opt/sa/saltbox/500.json
team_drive = 0AIM9nXoo2tiLUk9PVA
root_folder_id =

[tv]
type = drive
scope = drive
service_account_file = /opt/sa/vaulted/501.json
team_drive = 0ANs4mfxvSpnKUk9PVA
root_folder_id =

A log from the command with the -vv flag

2022/05/04 23:24:39 DEBUG : rclone: Version "v1.58.0" starting with parameters ["/usr/bin/rclone" "sync" "tv:Media/TV" "saltbox347323tv:Media/tv3" "--drive-service-account-file" "/opt/sa/vaulted/7.json" "-vvvP" "--transfers=16" "--drive-server-side-across-configs"]
2022/05/04 23:24:39 DEBUG : Creating backend with remote "tv:Media/TV"
2022/05/04 23:24:39 DEBUG : Using config file from "/home/jleiss/.config/rclone/rclone.conf"
2022/05/04 23:24:39 DEBUG : tv: detected overridden config - adding "{H34wN}" suffix to name
2022/05/04 23:24:39 DEBUG : fs cache: renaming cache item "tv:Media/TV" to be canonical "tv{H34wN}:Media/TV"
2022/05/04 23:24:39 DEBUG : Creating backend with remote "saltbox347323tv:Media/tv3"
2022/05/04 23:24:39 DEBUG : saltbox347323tv: detected overridden config - adding "{H34wN}" suffix to name
2022/05/04 23:24:40 DEBUG : fs cache: renaming cache item "saltbox347323tv:Media/tv3" to be canonical "saltbox347323tv{H34wN}:Media/tv3"
2022-05-04 23:24:52 NOTICE: Holey Moley (AU) (2021): Duplicate directory found in destination - ignoring
2022-05-04 23:24:52 NOTICE: I Think You Should Leave with Tim Robinson (2019): Duplicate directory found in destination - ignoring
2022-05-04 23:24:52 NOTICE: LEGO Masters (UK) (2017): Duplicate directory found in destination - ignoring
2022-05-04 23:24:56 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E01_We Are Family.mp4: md5 = 042928b2feb5db3bd9de19fb27f04ff9 OK
2022-05-04 23:24:56 INFO  : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E01_We Are Family.mp4: Copied (server-side copy)
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 45.237026ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 128.658504ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 174.777392ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 274.432421ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 372.084526ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 460.403822ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 521.886274ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 483.407466ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 581.994273ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 638.221931ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 559.562141ms
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 159.371144ms
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E04_It Takes a Village, to Buy a Dress.mp4: md5 = 53ed8b992fd22c42cc8c02e9c3e27163 OK
2022-05-04 23:24:59 INFO  : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E04_It Takes a Village, to Buy a Dress.mp4: Copied (server-side copy)
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E02_Budget Big, Budget Small.mp4: md5 = 4abaa9e2fca6159ec940ebe0a02ed93b OK
2022-05-04 23:24:59 INFO  : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E02_Budget Big, Budget Small.mp4: Copied (server-side copy)
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E06_It's All A Puzzle.mp4: md5 = 7faeb8f61347af70033554d65116de4f OK
2022-05-04 23:24:59 INFO  : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E06_It's All A Puzzle.mp4: Copied (server-side copy)
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 77.111135ms
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 127.639571ms
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E03_Unconventional Couples.mp4: md5 = 31489ed0e3093d0be5b0b9d7159ac46f OK
2022-05-04 23:24:59 INFO  : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E03_Unconventional Couples.mp4: Copied (server-side copy)
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 86.802509ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 139.403662ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 232.391867ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 331.232535ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 421.269745ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 504.19068ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 556.39603ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 602.97548ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 623.649075ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 594.057207ms
2022-05-04 23:25:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 1.965816834s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 170.751351ms
2022-05-04 23:25:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 2.284404073s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E05_Second Time Round.mp4: md5 = 070d858fffa31be5ef466fc2ff02022c OK
2022-05-04 23:25:01 INFO  : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E05_Second Time Round.mp4: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 1.002215118s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E03.Hell.Hath.No.Fury.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = 17d6fc82accc8c88cf6161b86888b9b3 OK
2022-05-04 23:25:01 INFO  : Scandal (2012)/Season 01/Scandal.2012.S01E03.Hell.Hath.No.Fury.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E01.Sweet.Baby.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = 8f812cb481406f5da6dba4e6d67efe08 OK
2022-05-04 23:25:01 INFO  : Scandal (2012)/Season 01/Scandal.2012.S01E01.Sweet.Baby.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E02.Dirty.Little.Secrets.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = 0e9b2cba1019eb8bacb3f25002a621a0 OK
2022-05-04 23:25:01 INFO  : Scandal (2012)/Season 01/Scandal.2012.S01E02.Dirty.Little.Secrets.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 1.259567954s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E05.Crash.and.Burn.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = c34c4497929190d48030e93924d7303d OK
2022-05-04 23:25:01 INFO  : Scandal (2012)/Season 01/Scandal.2012.S01E05.Crash.and.Burn.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 11.405463ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 83.450505ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 147.601941ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 95.419045ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 184.582223ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 267.944468ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 341.221512ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 424.338907ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 522.949464ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 556.376306ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 651.854887ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 562.609744ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 371.6537ms
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:03 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:03 DEBUG : pacer: Rate limited, increasing sleep to 1.268971898s
2022-05-04 23:25:03 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:03 DEBUG : pacer: Rate limited, increasing sleep to 2.239851242s
2022-05-04 23:25:03 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:03 DEBUG : pacer: Rate limited, increasing sleep to 4.074724094s
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 47.233675ms
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 146.998194ms
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 232.817075ms
2022-05-04 23:25:04 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:04 DEBUG : pacer: Rate limited, increasing sleep to 1.069324465s
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 120.706215ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 208.607579ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 303.730889ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 400.695313ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 471.606473ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 554.413486ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 613.762032ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 525.828543ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 123.817922ms
2022-05-04 23:25:05 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:05 DEBUG : pacer: Rate limited, increasing sleep to 2.33606546s
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 6.070149ms
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 76.258677ms
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 171.069687ms
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 174.963867ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 166.64059ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 258.915125ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 346.782921ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 434.680247ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 513.745503ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 608.797203ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 643.196125ms
2022-05-04 23:25:06 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:06 DEBUG : pacer: Rate limited, increasing sleep to 4.144454502s
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 558.523405ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 228.278794ms
Transferred:       19.747 GiB / 2.268 TiB, 1%, 1.518 GiB/s, ETA 25m16s
Transferred:           10 / 1584, 1%
Elapsed time:        27.8s
Transferring:
 * Scandal (2012)/Season ….1.H.264-CasStudio.mkv:  0% /3.846Gi, 0/s, -
 * Scandal (2012)/Season …5.1.x264-CasStudio.mkv:  0% /3.498Gi, 0/s, -
 * Scandal (2012)/Season …5.1.x264-CasStudio.mkv:  0% /3.713Gi, 0/s, -
 * Say Yes To The Dress -…ilemma WEBDL-1080p.mkv:  0% /2.003Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.258Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.112Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.421Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.644Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.750Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.623Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.x265-SIGMA.mkv:  0% /2.386Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.592Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.652Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv:  0% /2.755Gi, 0/s, -
 * Scandal (2012)/Season …L-1080p.x265-SIGMA.mkv:  0% /2.449Gi, 0/s, --^C

Those are Google daily limits you are you hitting.

Nothing to do but wait.

any adjustments that can be made to avoid the limits?

Switch to Dropbox?

Nope as they are all on the Google side and nothing you can do. More users get more per day but that’s about it.