Server Side Sync on Google Drive

What is the problem you are having with rclone?

Increase speed of server-side copy on google drive? Ideally I'd like to download the full VFS/List and compare them locally and then only run api commands on the ones I need to server side copy. Is the command below the best way? Would be nice if I could pause/resume it also since I can only do 750GB a day. So if I could do 750 and then just wait until tomorrow and pick back up. Just seems to get stuck in the api rate limit.

As an aside for another use I already have both drives mounted and fully setup with the VFS cached... if somehow I could use that to do a sync.

What is your rclone version (output from rclone version)

rclone v1.56.0-beta.5530.df60e6323

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-73-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.4
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-73-generic (x86_64)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --drive-server-side-across-configs gddrivea:folder gddriveb:folder --max-transfer 730G -vv --fast-list  2>&1 | tee SYNC.LOG

The rclone config contents with secrets removed.

[gddrivea]
type = drive
client_id = CLIENTID
client_secret = SECRET
scope = drive
token = TOKEN
team_drive = DRIVEA
root_folder_id = 

[gddriveb]
type = drive
client_id = CLIENTID
client_secret = SECRET
scope = drive
token = TOKEN
team_drive = DRIVEB
root_folder_id = 
2021/06/06 01:06:09 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:09 DEBUG : pacer: Rate limited, increasing sleep to 16.02279528s
2021/06/06 01:06:26 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:26 DEBUG : pacer: Rate limited, increasing sleep to 16.300382049s
2021/06/06 01:06:42 DEBUG : pacer: Reducing sleep to 0s
2021/06/06 01:06:53 INFO  :
Transferred:              0 / 1.748 TiByte, 0%, 0 Byte/s, ETA -
Checks:             17330 / 17330, 100%
Transferred:            0 / 274, 0%
Elapsed time:      11m0.8s
Transferring:
 * FILEA:  0% /8.008Gi, 0/s, -
 * FILEB:  0% /14.573Gi, 0/s, -
 * FILEC:  0% /3.529Gi, 0/s, -
 * FILED:  0% /1.682Gi, 0/s, -

2021/06/06 01:06:59 DEBUG : pacer: low level retry 10/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 1.971218263s
2021/06/06 01:06:59 DEBUG FILEA: Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 1/10
2021/06/06 01:06:59 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 2.519573432s
2021/06/06 01:06:59 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 4.880814947s
2021/06/06 01:06:59 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:06:59 DEBUG : pacer: Rate limited, increasing sleep to 8.823586082s
2021/06/06 01:06:59 DEBUG : pacer: Reducing sleep to 0s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 1.183129342s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 2.26079317s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 4.778253103s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 8.982230336s
2021/06/06 01:07:01 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:01 DEBUG : pacer: Rate limited, increasing sleep to 16.470163768s
2021/06/06 01:07:03 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:03 DEBUG : pacer: Rate limited, increasing sleep to 16.300987225s
2021/06/06 01:07:19 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:19 DEBUG : pacer: Rate limited, increasing sleep to 16.066246535s
2021/06/06 01:07:35 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:35 DEBUG : pacer: Rate limited, increasing sleep to 16.599384851s
2021/06/06 01:07:51 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2021/06/06 01:07:51 DEBUG : pacer: Rate limited, increasing sleep to 16.434871865s
2021/06/06 01:07:53 INFO  :

You can just use the stop upload limit flag and run it daily until you are caught up:

https://rclone.org/drive/#drive-stop-on-upload-limit

Max transfer does nothing on server side since nothing is really transferred since it all happens server side.

Yep. That was it. Thanks!

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.