What is the problem you are having with rclone?
Tryiing to sync data between 2 drives and after approx 3-3.5 TB I get rate limit errors and can't do anything until the next day.
Run the command 'rclone version' and share the full output of the command.
rclone v1.58.0
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 5.6.0-050600-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.8
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
/usr/bin/rclone sync "tv:Media/TV" "saltbox347323tv:Media/tv3" --drive-service-account-file /opt/sa/vaulted/7.json -vvvP --transfers=16 --drive-server-side-across-configs
The rclone config contents with secrets removed.
[movie]
type = drive
scope = drive
team_drive = 0AJdYVATZ8eOQUk9PVA
service_account_file = /opt/sa/sasync/93.json
[google]
type = union
upstreams = movie: tv: saltbox347323tv:
token = {"access_token":"ya29.A0ARrdaM8j4idv-KoVB8jQVU07pEgR1w4DIOxqrnCssRzkBd5CzUad-dL31rxIWZ1zDXUcq26gSlQnN3rx-bdbPmcEaIf5mHc4X98I9dv-12etLaeA4Pe6vEwAKR6jcEEVbrjvOqJ0UarM7lxsaZo7X44JlWUO2UA","token_type":"Bearer","refresh_token":"1//03RpZx-BgVtasCgYIARAAGAMSNwF-L9IrzBztXPwSyj1m3lBkKmjCMekynsPaIJFvMV5Nda8hZUAZ16Q_EAoeO4qTAt-YqRKT8PQ","expiry":"2022-03-30T00:49:04.77086765+02:00"}
[saltbox347323tv]
type = drive
scope = drive
service_account_file = /opt/sa/saltbox/500.json
team_drive = 0AIM9nXoo2tiLUk9PVA
root_folder_id =
[tv]
type = drive
scope = drive
service_account_file = /opt/sa/vaulted/501.json
team_drive = 0ANs4mfxvSpnKUk9PVA
root_folder_id =
A log from the command with the -vv flag
2022/05/04 23:24:39 DEBUG : rclone: Version "v1.58.0" starting with parameters ["/usr/bin/rclone" "sync" "tv:Media/TV" "saltbox347323tv:Media/tv3" "--drive-service-account-file" "/opt/sa/vaulted/7.json" "-vvvP" "--transfers=16" "--drive-server-side-across-configs"]
2022/05/04 23:24:39 DEBUG : Creating backend with remote "tv:Media/TV"
2022/05/04 23:24:39 DEBUG : Using config file from "/home/jleiss/.config/rclone/rclone.conf"
2022/05/04 23:24:39 DEBUG : tv: detected overridden config - adding "{H34wN}" suffix to name
2022/05/04 23:24:39 DEBUG : fs cache: renaming cache item "tv:Media/TV" to be canonical "tv{H34wN}:Media/TV"
2022/05/04 23:24:39 DEBUG : Creating backend with remote "saltbox347323tv:Media/tv3"
2022/05/04 23:24:39 DEBUG : saltbox347323tv: detected overridden config - adding "{H34wN}" suffix to name
2022/05/04 23:24:40 DEBUG : fs cache: renaming cache item "saltbox347323tv:Media/tv3" to be canonical "saltbox347323tv{H34wN}:Media/tv3"
2022-05-04 23:24:52 NOTICE: Holey Moley (AU) (2021): Duplicate directory found in destination - ignoring
2022-05-04 23:24:52 NOTICE: I Think You Should Leave with Tim Robinson (2019): Duplicate directory found in destination - ignoring
2022-05-04 23:24:52 NOTICE: LEGO Masters (UK) (2017): Duplicate directory found in destination - ignoring
2022-05-04 23:24:56 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E01_We Are Family.mp4: md5 = 042928b2feb5db3bd9de19fb27f04ff9 OK
2022-05-04 23:24:56 INFO : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E01_We Are Family.mp4: Copied (server-side copy)
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 45.237026ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 128.658504ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 174.777392ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 274.432421ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 372.084526ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 460.403822ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 521.886274ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 483.407466ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 581.994273ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 638.221931ms
2022-05-04 23:24:58 DEBUG : pacer: Reducing sleep to 559.562141ms
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 159.371144ms
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E04_It Takes a Village, to Buy a Dress.mp4: md5 = 53ed8b992fd22c42cc8c02e9c3e27163 OK
2022-05-04 23:24:59 INFO : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E04_It Takes a Village, to Buy a Dress.mp4: Copied (server-side copy)
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E02_Budget Big, Budget Small.mp4: md5 = 4abaa9e2fca6159ec940ebe0a02ed93b OK
2022-05-04 23:24:59 INFO : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E02_Budget Big, Budget Small.mp4: Copied (server-side copy)
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E06_It's All A Puzzle.mp4: md5 = 7faeb8f61347af70033554d65116de4f OK
2022-05-04 23:24:59 INFO : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E06_It's All A Puzzle.mp4: Copied (server-side copy)
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 77.111135ms
2022-05-04 23:24:59 DEBUG : pacer: Reducing sleep to 127.639571ms
2022-05-04 23:24:59 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E03_Unconventional Couples.mp4: md5 = 31489ed0e3093d0be5b0b9d7159ac46f OK
2022-05-04 23:24:59 INFO : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E03_Unconventional Couples.mp4: Copied (server-side copy)
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 86.802509ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 139.403662ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 232.391867ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 331.232535ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 421.269745ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 504.19068ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 556.39603ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 602.97548ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 623.649075ms
2022-05-04 23:25:00 DEBUG : pacer: Reducing sleep to 594.057207ms
2022-05-04 23:25:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 1.965816834s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 170.751351ms
2022-05-04 23:25:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 2.284404073s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E05_Second Time Round.mp4: md5 = 070d858fffa31be5ef466fc2ff02022c OK
2022-05-04 23:25:01 INFO : Say Yes To The Dress Australia (2016)/Season 01/Say Yes to the Dress_ Australia_S01E05_Second Time Round.mp4: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 1.002215118s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E03.Hell.Hath.No.Fury.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = 17d6fc82accc8c88cf6161b86888b9b3 OK
2022-05-04 23:25:01 INFO : Scandal (2012)/Season 01/Scandal.2012.S01E03.Hell.Hath.No.Fury.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E01.Sweet.Baby.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = 8f812cb481406f5da6dba4e6d67efe08 OK
2022-05-04 23:25:01 INFO : Scandal (2012)/Season 01/Scandal.2012.S01E01.Sweet.Baby.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E02.Dirty.Little.Secrets.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = 0e9b2cba1019eb8bacb3f25002a621a0 OK
2022-05-04 23:25:01 INFO : Scandal (2012)/Season 01/Scandal.2012.S01E02.Dirty.Little.Secrets.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:01 DEBUG : pacer: Rate limited, increasing sleep to 1.259567954s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:01 DEBUG : Scandal (2012)/Season 01/Scandal.2012.S01E05.Crash.and.Burn.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: md5 = c34c4497929190d48030e93924d7303d OK
2022-05-04 23:25:01 INFO : Scandal (2012)/Season 01/Scandal.2012.S01E05.Crash.and.Burn.1080p.AMZN.WEBRip.DD5.1.x264-CasStudio.mkv: Copied (server-side copy)
2022-05-04 23:25:01 DEBUG : pacer: Reducing sleep to 11.405463ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 83.450505ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 147.601941ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 95.419045ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 184.582223ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 267.944468ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 341.221512ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 424.338907ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 522.949464ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 556.376306ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 651.854887ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 562.609744ms
2022-05-04 23:25:02 DEBUG : pacer: Reducing sleep to 371.6537ms
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:03 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:03 DEBUG : pacer: Rate limited, increasing sleep to 1.268971898s
2022-05-04 23:25:03 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:03 DEBUG : pacer: Rate limited, increasing sleep to 2.239851242s
2022-05-04 23:25:03 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:03 DEBUG : pacer: Rate limited, increasing sleep to 4.074724094s
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 47.233675ms
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 146.998194ms
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:03 DEBUG : pacer: Reducing sleep to 232.817075ms
2022-05-04 23:25:04 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:04 DEBUG : pacer: Rate limited, increasing sleep to 1.069324465s
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 120.706215ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 208.607579ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 303.730889ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 400.695313ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 471.606473ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 554.413486ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 613.762032ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 525.828543ms
2022-05-04 23:25:04 DEBUG : pacer: Reducing sleep to 123.817922ms
2022-05-04 23:25:05 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:05 DEBUG : pacer: Rate limited, increasing sleep to 2.33606546s
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 0s
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 6.070149ms
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 76.258677ms
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 171.069687ms
2022-05-04 23:25:05 DEBUG : pacer: Reducing sleep to 174.963867ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 166.64059ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 258.915125ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 346.782921ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 434.680247ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 513.745503ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 608.797203ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 643.196125ms
2022-05-04 23:25:06 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022-05-04 23:25:06 DEBUG : pacer: Rate limited, increasing sleep to 4.144454502s
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 558.523405ms
2022-05-04 23:25:06 DEBUG : pacer: Reducing sleep to 228.278794ms
Transferred: 19.747 GiB / 2.268 TiB, 1%, 1.518 GiB/s, ETA 25m16s
Transferred: 10 / 1584, 1%
Elapsed time: 27.8s
Transferring:
* Scandal (2012)/Season ….1.H.264-CasStudio.mkv: 0% /3.846Gi, 0/s, -
* Scandal (2012)/Season …5.1.x264-CasStudio.mkv: 0% /3.498Gi, 0/s, -
* Scandal (2012)/Season …5.1.x264-CasStudio.mkv: 0% /3.713Gi, 0/s, -
* Say Yes To The Dress -…ilemma WEBDL-1080p.mkv: 0% /2.003Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.258Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.112Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.421Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.644Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.750Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.623Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.x265-SIGMA.mkv: 0% /2.386Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.592Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.652Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.h265-SIGMA.mkv: 0% /2.755Gi, 0/s, -
* Scandal (2012)/Season …L-1080p.x265-SIGMA.mkv: 0% /2.449Gi, 0/s, --^C