What is the problem you are having with rclone?
Increase speed of server-side copy on google drive? Ideally I'd like to download the full VFS/List and compare them locally and then only run api commands on the ones I need to server side copy. Is the command below the best way? Would be nice if I could pause/resume it also since I can only do 750GB a day. So if I could do 750 and then just wait until tomorrow and pick back up. Just seems to get stuck in the api rate limit.
As an aside for another use I already have both drives mounted and fully setup with the VFS cached... if somehow I could use that to do a sync.
What is your rclone version (output from rclone version
)
rclone v1.56.0-beta.5530.df60e6323
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-73-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.4
- go/linking: static
- go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-73-generic (x86_64)
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync --drive-server-side-across-configs gddrivea:folder gddriveb:folder --max-transfer 730G -vv --fast-list 2>&1 | tee SYNC.LOG
The rclone config contents with secrets removed.
[gddrivea]
type = drive
client_id = CLIENTID
client_secret = SECRET
scope = drive
token = TOKEN
team_drive = DRIVEA
root_folder_id =
[gddriveb]
type = drive
client_id = CLIENTID
client_secret = SECRET
scope = drive
token = TOKEN
team_drive = DRIVEB
root_folder_id =
2021/06/06 01:06:09 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:09 DEBUG : pacer: Rate limited, increasing sleep to 16.02279528s
2021/06/06 01:06:26 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:26 DEBUG : pacer: Rate limited, increasing sleep to 16.300382049s
2021/06/06 01:06:42 DEBUG : pacer: Reducing sleep to 0s
2021/06/06 01:06:53 INFO :
Transferred: 0 / 1.748 TiByte, 0%, 0 Byte/s, ETA -
Checks: 17330 / 17330, 100%
Transferred: 0 / 274, 0%
Elapsed time: 11m0.8s
Transferring:
* FILEA: 0% /8.008Gi, 0/s, -
* FILEB: 0% /14.573Gi, 0/s, -
* FILEC: 0% /3.529Gi, 0/s, -
* FILED: 0% /1.682Gi, 0/s, -
2021/06/06 01:06:59 DEBUG : pacer: low level retry 10/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 1.971218263s
2021/06/06 01:06:59 DEBUG FILEA: Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 1/10
2021/06/06 01:06:59 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 2.519573432s
2021/06/06 01:06:59 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 4.880814947s
2021/06/06 01:06:59 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 8.823586082s
2021/06/06 01:06:59 DEBUG : pacer: Reducing sleep to 0s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 1.183129342s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 2.26079317s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 4.778253103s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 8.982230336s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 16.470163768s
2021/06/06 01:07:03 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:03 DEBUG : pacer: Rate limited, increasing sleep to 16.300987225s
2021/06/06 01:07:19 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:19 DEBUG : pacer: Rate limited, increasing sleep to 16.066246535s
2021/06/06 01:07:35 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:35 DEBUG : pacer: Rate limited, increasing sleep to 16.599384851s
2021/06/06 01:07:51 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:51 DEBUG : pacer: Rate limited, increasing sleep to 16.434871865s
2021/06/06 01:07:53 INFO :