Best practices for handling aggressive google drive rate limiting

Help and Support
1

What is the problem you are having with rclone?

I am trying to upload 200TB of backups to google drive using my institutions unlimited for the time being storage. Every day a cron job kicks off an rclone copy and runs until rate limiting kicks in. Google has been aggressively rate limiting my account from the start and usually only upload a few hundred GBs before failing, but lately they have been getting shut down after single digit GBs. The upload will never finish at this rate as the size of the backups grows faster than this.

I am of course already using my own API key, but would like to seek further input from the community on optimal rclone args for this scenario and general thoughts.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.92.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

google drive

The command you were trying to run

rclone sync $OAK/backups/restic gdrive:/oak/restic \
       --log-file=$OAK/backups/logs/rclone/snapshot_${SUBMIT_DATE} \
       --cache-dir=$SCRATCH/rclone_cache \
       --fast-list \
       --checkers=64 \
       --transfers=64 \
       --tpslimit=10 \
       --drive-chunk-size=128M \
       --max-backlog=200000 \
       --drive-stop-on-upload-limit \
       --stats=5m \
       --stats-log-level=NOTICE

The rclone config contents with secrets removed.

type = dropbox
token = {"access_token":XXXXXXXXXXXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

[gdrive_shared]
type = drive
client_id = XXXXXXXXXXX
client_secret = XXXXXXXXX
scope = drive
token = {"access_token":"XXXXXXXXXX","token_type":"Bearer","refresh_token":"1XXXXXXXXXXX","expiry":"2023-03-16T12:48:36.136251787-07:00"}
team_drive = XXXXXXXXXX
root_folder_id =

[gdrive]
type = drive
client_id = XXXXXXXX
client_secret = XXXXXXXXXX
scope = drive
token = {"access_token":"XXXXXXXXXXXX","expiry":"2023-08-03T18:43:44.896432637-07:00"}
team_drive =

A log from the command with the -vv flag

standby on the verbose log, i'd have to take my services down for a day which i am reticent to do

I have abridged the logs to meet the character limit

2023/08/02 09:49:54 NOTICE:
Transferred:   	   19.584 MiB / 3.199 TiB, 0%, 41.127 KiB/s, ETA 2y33w5d13h41m28s
Checks:              4068 / 4068, 100%
Transferred:            8 / 200129, 0%
Elapsed time:       5m1.3s
Transferring:
 * data/49/490004b51208bc…7dde25fbfc8a9efe0bc28b:  0% /16.140Mi, 0/s, -
 * data/49/49000cbc0dec6c…8335db885cdbee914ccdd7:  0% /16.464Mi, 0/s, -
 * data/49/490017ee93c7c6…b052c27fe86acefbae5c98:  0% /16.033Mi, 0/s, -
 * data/49/49001936897623…c47134859b19790332a0dc:  0% /16.843Mi, 0/s, -
 * data/49/49001cfc4140b4…8c283d221b39a3e29a9022:  0% /16.371Mi, 0/s, -
 * data/49/49001e0183f637…4fd3f2013c54bf3931a400:  0% /16.270Mi, 0/s, -
 * data/49/49001f305dad7e…816c05a76cde79ffa2b1fb:  0% /19.433Mi, 0/s, -
 * data/49/490020272d1036…5ba75b4c1ab4b6a41d849d:  0% /16.175Mi, 0/s, -
 * data/49/490020a467535b…a597271566f728b3ede187:  0% /16.206Mi, 0/s, -
 * data/49/4900247241fd38…ed591ea085a1a37cfbd0d1:  0% /17.043Mi, 0/s, -
 * data/49/4900258d590fea…c2a988ab6d99d083213787:  0% /16.720Mi, 0/s, -
 * data/49/490027c15dcd39…2a15db226ca565c15ea568:  0% /16.434Mi, 0/s, -
 * data/49/490028b7410d67…a7c75d0c2e4f83e3ae8767:  0% /16.476Mi, 0/s, -
 * data/49/49002a8a5818d5…094651d070be3a3296cc76:  0% /16.206Mi, 0/s, -
 * data/49/49002c0c6c5f1c…2b3cccd53c0f74f64f50ce:  0% /16.064Mi, 0/s, -
 * data/49/49002c622c4337…98ce06dce58f8a362a7a84:  0% /16.155Mi, 0/s, -
 * data/49/49002fbdfd861a…692c23b5c3930a5e2e650e:  0% /16.101Mi, 0/s, -
 * data/49/490030cc351f10…f3e86f52fef8e0c78b3fab:  0% /16.166Mi, 0/s, -
 * data/49/490030eac6254c…a748846cb35c3b23c9d575:  0% /16.333Mi, 0/s, -
 * data/49/4900338b2aa3f3…8ed61be7f3fdb6b043a254:  0% /16.614Mi, 0/s, -
 * data/49/4900371c083475…b04a326b51d294e0460673:  0% /16.106Mi, 0/s, -
 * data/49/49003974d6cfd8…41cbf4b17d9196508449ec:  0% /18.757Mi, 0/s, -
 * data/49/49003b19c50ff2…c826bfe614303580b09f71:  0% /16.308Mi, 0/s, -
 * data/49/49003bac38727f…75fb04f0a8d4fb22d76e44:  0% /16.759Mi, 0/s, -
 * data/49/49003c7150f623…7da17c71f9c53a0ad67b16:  0% /16.205Mi, 0/s, -
 * data/49/49003dc71e8f9d…98182b87de3b79dc0cc6d2:  0% /16.568Mi, 0/s, -
 * data/49/49003e9e459b4c…1d0eb902548636213abe64:  0% /17.453Mi, 0/s, -
 * data/49/49003ec9c07491…bbea12e0594ee506ad1fd1:  0% /16.251Mi, 0/s, -
 * data/49/49003fdbb48a7b…03cc208ac2dc9690740385:  0% /16.487Mi, 0/s, -
 * data/49/490040ab268d38…d2afc7eb45883eac5f8284:  0% /16.330Mi, 0/s, -
 * data/49/490040c78beac5…c0180e218b9d616b3546b6:  0% /16.101Mi, 0/s, -
 * data/49/490000286ea43e…3224fd353ee7a9d8ffe70d:  0% /17.439Mi, 0/s, -
 * data/49/490040ee3972db…80b85cca6a4a127960b256:  0% /16.537Mi, 0/s, -
 * data/49/49004173587feb…ef3d103f34c4633f6bbf41:  0% /16.207Mi, 0/s, -
 * data/49/490042b5e3023b…8e3c913b38aa23045ff178:  0% /17.103Mi, 0/s, -
 * data/49/490047ca0272e9…4d24fce00c5fd6876a8db6:  0% /17.219Mi, 0/s, -
 * data/49/49004a6d3f8a71…1e7cc525b434ec70d6c195:  0% /16.280Mi, 0/s, -
 * data/49/49004aa90e93ad…d3b3d5cc5ed21f77b2e47f:  0% /16.220Mi, 0/s, -
 * data/49/49004eac85a48c…84b548f62527d62f854bed:  0% /16.521Mi, 0/s, -
 * data/49/490050250ad89d…6a27c291346505930fb3e0:  0% /16.094Mi, 0/s, -
 * data/49/49005341b5099d…8e3443d92a512e6a2be3da:  0% /16.151Mi, 0/s, -
 * data/49/49005429ddf0ab…0914746a852be957e3c487:  0% /16.510Mi, 0/s, -
 * data/49/49005653ada658…09043d79b6e207990bcf03:  0% /16.094Mi, 0/s, -
 * data/49/49005697bb5cd0…a81b5ee80b5735203ef278:  0% /16.698Mi, 0/s, -
 * data/49/490058d7eacc44…738331452cd4164b25fb1b:  0% /16.190Mi, 0/s, -
 * data/49/49005af80ed458…49ac5d2d3e015d14d9fbf4:  0% /16.353Mi, 0/s, -
 * data/49/49005bfede395a…2bc0842eba223b9737142c:  0% /16.222Mi, 0/s, -
 * data/49/49005d305dd6bc…1ba05b75b5ea5555cdd05d:  0% /17.204Mi, 0/s, -
 * data/49/49005e222dc5a3…85f485a4d5b6beef7c07f3:  0% /16.601Mi, 0/s, -
 * data/49/49005ed2b38034…f3b5ea30a4f2661858f4ed:  0% /17.508Mi, 0/s, -
 * data/49/49006020c68265…d529b5531806bdc3bd8ee3:  0% /16.037Mi, 0/s, -
 * data/49/4900646a36458f…0bacb8b05997e0da8fe5c2:  0% /16.078Mi, 0/s, -
 * data/49/490066fc1ca89d…319e280d78c1b06f001d36:  0% /16.547Mi, 0/s, -
 * data/49/49006700a94a78…74009407be1af2bd7a21ae:  0% /17.570Mi, 0/s, -
 * data/49/4900671aae6f60…689287f83554e7ae07b536:  0% /17.672Mi, 0/s, -
 * data/49/4900691d15e962…e2310dffdff97eb54e217e:  0% /16.035Mi, 0/s, -
 * data/49/490069cc363a80…b6d9e862b8a5c3514c8b34:  0% /16.272Mi, 0/s, -
 * data/49/49006b22ec1975…e5bee955902b0d321aad38:  0% /16.158Mi, 0/s, -
 * data/49/49006c8ae466ea…53f063189e301f440260e0:  0% /16.265Mi, 0/s, -
 * data/49/49006ebcb11cf9…1b1787d63036dbd9909866:  0% /16.907Mi, 0/s, -
 * data/49/490070524ea3c5…9bacbe1f4ce350546ffa89:  0% /17.620Mi, 0/s, -
 * data/49/4900719f18185e…433e07bbf0ac24dc49999e:  0% /16.237Mi, 0/s, -
 * data/49/490073a2d32b6c…3192ec7967b52e6179c186:  0% /16.252Mi, 0/s, -
 * data/49/490074383ae111…4f11f5f9730ceab1b1349e:  0% /16.092Mi, 0/s, -

2023/08/02 09:51:06 ERROR : Google drive root 'oak/restic': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2023/08/02 09:51:06 ERROR : data/49/49014a0cceebad6b33d683fcadbae65535af8d8f8220472067f7733aff905fdf: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2023/08/02 09:51:06 ERROR : Cancelling sync due to fatal error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2023/08/02 09:51:06 ERROR : data/49/490182d490b0126d5c9f6cafa88c7495fdafe8f09a6977126f523565a69d1b48: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdshJV5xKDFfwMBq42WjzdmACkqIO1d5KCxh4gcIPEs5TQA7sJKloFM0wkC7HuE3n-H2Z7d5t7u3voG4GdqdsoU3Bw": context canceled


==============
abridged, continues similarly for many lines
==============

2023/08/02 09:51:06 ERROR : data/49/49015d0559e18c90b75343d99b941e703dab043f4d2a7ce0e62d6119eaddfb93: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdv2CkPbVSyVrEUxDwmPHst67BV3S75GUqCaH0CsZ7_ccptrLoXw7HCt9ZiMhf-0qc0f0HQpj10jXrpJxndSw_t8rg": context canceled
2023/08/02 09:51:06 ERROR : data/49/4901208b7f3d501165eb2a3b5ca4f33eadcb0d3ab58328faed4a78e001c0a226: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdsZmL_z1_lOrintWxxw3zsEVgP6oZk0SXv2aMcosiiZ9O171K3bQ_jTRCDGoxUDvoBdQOB7DjyKz5nVzg5qZ-1j": context canceled
2023/08/02 09:52:15 ERROR : Google drive root 'oak/restic': not deleting files as there were IO errors
2023/08/02 09:52:15 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/08/02 09:52:15 ERROR : Fatal error received - not attempting retries
2023/08/02 09:52:15 NOTICE:
Transferred:   	    3.418 GiB / 3.201 TiB, 0%, 41.127 KiB/s, ETA 2y33w5d7h6m33s
Errors:                64 (fatal error encountered)
Checks:              4068 / 4068, 100%
Transferred:          152 / 200216, 0%
Elapsed time:      7m22.2s

2023/08/02 09:52:15 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2

Summary of daily logs showing successful upload volume decreasing over time

(exceeded character limit for the main post)

==> snapshot_2023-04-27 <==
2023/04/28 04:29:28 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/04/28 04:29:28 ERROR : Fatal error received - not attempting retries
2023/04/28 04:29:28 NOTICE:
Transferred:   	  749.459 GiB / 3.930 TiB, 19%, 1 B/s, ETA -188y16w3d14h56m976ms
Errors:                64 (fatal error encountered)
Checks:              2891 / 2891, 100%
Transferred:        45534 / 245598, 19%
Elapsed time:   4h27m42.7s

2023/04/28 04:29:28 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-04-28 <==
2023/05/01 19:48:57 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/01 19:48:57 ERROR : Fatal error received - not attempting retries
2023/05/01 19:48:57 NOTICE:
Transferred:   	  293.279 GiB / 3.484 TiB, 8%, 611.590 KiB/s, ETA 9w1d23h22m45s
Errors:                64 (fatal error encountered)
Checks:              2892 / 2892, 100%
Transferred:        17885 / 217949, 8%
Elapsed time:    2h4m35.9s

2023/05/01 19:48:57 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-02 <==
2023/05/03 00:07:07 ERROR : Google drive root 'oak/restic': not deleting files as there were IO errors
2023/05/03 00:07:07 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/03 00:07:07 ERROR : Fatal error received - not attempting retries
2023/05/03 00:07:07 NOTICE:
Transferred:   	    1.710 GiB / 3.168 TiB, 0%, 191.726 KiB/s, ETA 29w2d6h10m14s
Errors:                64 (fatal error encountered)
Transferred:          543 / 200601, 0%
Elapsed time:      4m17.4s

2023/05/03 00:07:07 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-03 <==
2023/05/05 00:27:39 ERROR : Google drive root 'oak/restic': not deleting files as there were IO errors
2023/05/05 00:27:39 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/05 00:27:39 ERROR : Fatal error received - not attempting retries
2023/05/05 00:27:39 NOTICE:
Transferred:   	  184.536 GiB / 3.378 TiB, 5%, 8.403 MiB/s, ETA 4d14h51m23s
Errors:                64 (fatal error encountered)
Transferred:        13673 / 213737, 6%
Elapsed time:    1h51m9.2s

2023/05/05 00:27:39 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-04 <==
2023/05/06 10:29:35 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/06 10:29:35 ERROR : Fatal error received - not attempting retries
2023/05/06 10:29:35 NOTICE:
Transferred:   	  749.262 GiB / 3.929 TiB, 19%, 323.876 KiB/s, ETA 17w3d16h49m58s
Errors:                64 (fatal error encountered)
Checks:              2978 / 2978, 100%
Transferred:        45462 / 245526, 19%
Elapsed time:     5h5m8.5s

2023/05/06 10:29:35 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-06 <==
2023/05/07 23:38:43 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/07 23:38:43 ERROR : Fatal error received - not attempting retries
2023/05/07 23:38:43 NOTICE:
Transferred:   	  268.263 GiB / 3.460 TiB, 8%, 61.742 KiB/s, ETA 1y39w5d17h49m29s
Errors:                64 (fatal error encountered)
Checks:              2986 / 2986, 100%
Transferred:        16151 / 216215, 7%
Elapsed time:   1h39m32.7s

2023/05/07 23:38:43 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-07 <==
2023/05/08 20:21:22 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/08 20:21:22 ERROR : Fatal error received - not attempting retries
2023/05/08 20:21:22 NOTICE:
Transferred:   	  124.696 GiB / 3.320 TiB, 4%, 501.522 KiB/s, ETA 11w2d5h53m17s
Errors:                64 (fatal error encountered)
Checks:              2991 / 2991, 100%
Transferred:         7587 / 207651, 4%
Elapsed time:     50m32.2s

2023/05/08 20:21:22 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-08 <==
2023/05/09 10:29:10 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/09 10:29:10 ERROR : Fatal error received - not attempting retries
2023/05/09 10:29:10 NOTICE:
Transferred:   	   51.329 GiB / 3.249 TiB, 2%, 32.365 KiB/s, ETA 3y19w2h26m50s
Errors:                64 (fatal error encountered)
Checks:              3011 / 3011, 100%
Transferred:         3083 / 203147, 2%
Elapsed time:     29m28.6s

2023/05/09 10:29:10 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-09 <==
2023/05/11 11:18:26 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/11 11:18:26 ERROR : Fatal error received - not attempting retries
2023/05/11 11:18:26 NOTICE:
Transferred:   	  236.320 GiB / 3.429 TiB, 7%, 0 B/s, ETA -121y20w6d19h39m52s200ms
Errors:                64 (fatal error encountered)
Checks:              3020 / 3020, 100%
Transferred:        14406 / 214470, 7%
Elapsed time:   1h53m28.8s

2023/05/11 11:18:26 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-11 <==
2023/05/13 10:17:50 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/13 10:17:50 ERROR : Fatal error received - not attempting retries
2023/05/13 10:17:50 NOTICE:
Transferred:   	  438.502 GiB / 3.626 TiB, 12%, 1.408 MiB/s, ETA 3w6d13h33m42s
Errors:                64 (fatal error encountered)
Checks:              3053 / 3053, 100%
Transferred:        26635 / 226699, 12%
Elapsed time:   2h54m13.9s

2023/05/13 10:17:50 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-13 <==
2023/05/14 17:06:05 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/14 17:06:05 ERROR : Fatal error received - not attempting retries
2023/05/14 17:06:05 NOTICE:
Transferred:   	  232.988 GiB / 3.426 TiB, 7%, 982.195 KiB/s, ETA 5w5d11h7m49s
Errors:                64 (fatal error encountered)
Checks:              3104 / 3104, 100%
Transferred:        14157 / 214221, 7%
Elapsed time:   1h20m10.5s

2023/05/14 17:06:05 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-14 <==
2023/05/15 11:44:24 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/15 11:44:24 ERROR : Fatal error received - not attempting retries
2023/05/15 11:44:24 NOTICE:
Transferred:   	  233.379 GiB / 3.425 TiB, 7%, 191.822 KiB/s, ETA 29w4d3h36m15s
Errors:                64 (fatal error encountered)
Checks:              3127 / 3127, 100%
Transferred:        14204 / 214268, 7%
Elapsed time:    2h2m13.3s

2023/05/15 11:44:24 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-15 <==
2023/05/16 14:35:24 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/16 14:35:24 ERROR : Fatal error received - not attempting retries
2023/05/16 14:35:24 NOTICE:
Transferred:   	  168.422 GiB / 3.363 TiB, 5%, 104.574 KiB/s, ETA 1y2w1d2h28m7s
Errors:                64 (fatal error encountered)
Checks:              3133 / 3133, 100%
Transferred:        10061 / 210125, 5%
Elapsed time:    1h19m9.7s

2023/05/16 14:35:24 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-16 <==
2023/05/17 15:06:22 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/17 15:06:22 ERROR : Fatal error received - not attempting retries
2023/05/17 15:06:22 NOTICE:
Transferred:   	  425.743 GiB / 3.613 TiB, 12%, 286.589 KiB/s, ETA 19w5d15h51m43s
Errors:                64 (fatal error encountered)
Checks:              3137 / 3137, 100%
Transferred:        25719 / 225783, 11%
Elapsed time:   2h48m59.7s

2023/05/17 15:06:22 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-17 <==
2023/05/18 22:27:22 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/18 22:27:22 ERROR : Fatal error received - not attempting retries
2023/05/18 22:27:22 NOTICE:
Transferred:   	  378.533 GiB / 3.567 TiB, 10%, 355.098 KiB/s, ETA 15w6d21h43m22s
Errors:                64 (fatal error encountered)
Checks:              3148 / 3148, 100%
Transferred:        22986 / 223050, 10%
Elapsed time:   2h47m34.2s

2023/05/18 22:27:22 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-18 <==
2023/05/19 09:47:31 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/19 09:47:31 ERROR : Fatal error received - not attempting retries
2023/05/19 09:47:31 NOTICE:
Transferred:   	  369.621 GiB / 3.559 TiB, 10%, 47 B/s, ETA -1y5d19h8m30s838ms
Errors:                64 (fatal error encountered)
Checks:              3161 / 3161, 100%
Transferred:        22517 / 222581, 10%
Elapsed time:   2h17m40.4s

2023/05/19 09:47:31 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-19 <==
2023/05/20 15:59:29 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/20 15:59:29 ERROR : Fatal error received - not attempting retries
2023/05/20 15:59:29 NOTICE:
Transferred:   	  748.439 GiB / 3.929 TiB, 19%, 1.172 MiB/s, ETA 4w5d2h29m56s
Errors:                64 (fatal error encountered)
Checks:              3163 / 3163, 100%
Transferred:        45683 / 245747, 19%
Elapsed time:     5h3m5.3s

2023/05/20 15:59:29 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-20 <==
2023/05/21 16:34:14 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/21 16:34:14 ERROR : Fatal error received - not attempting retries
2023/05/21 16:34:14 NOTICE:
Transferred:   	  545.122 GiB / 3.730 TiB, 14%, 338.185 KiB/s, ETA 16w5d12h25m42s
Errors:                64 (fatal error encountered)
Checks:              3187 / 3187, 100%
Transferred:        33094 / 233158, 14%
Elapsed time:   3h36m33.1s

2023/05/21 16:34:14 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-21 <==
2023/05/22 22:54:38 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/22 22:54:38 ERROR : Fatal error received - not attempting retries
2023/05/22 22:54:38 NOTICE:
Transferred:   	  750.117 GiB / 3.930 TiB, 19%, 5.144 KiB/s, ETA 21y8w4d9h21m17s
Errors:                64 (fatal error encountered)
Checks:              3194 / 3194, 100%
Transferred:        45726 / 245790, 19%
Elapsed time:   5h46m54.2s

2023/05/22 22:54:38 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-22 <==
2023/05/24 03:27:42 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/24 03:27:42 ERROR : Fatal error received - not attempting retries
2023/05/24 03:27:42 NOTICE:
Transferred:   	  163.629 GiB / 3.357 TiB, 5%, 258.319 KiB/s, ETA 21w6d20h1m28s
Errors:                64 (fatal error encountered)
Checks:              3198 / 3198, 100%
Transferred:         9857 / 209921, 5%
Elapsed time:    1h6m32.2s

2023/05/24 03:27:42 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-24 <==
2023/05/25 18:37:50 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/25 18:37:50 ERROR : Fatal error received - not attempting retries
2023/05/25 18:37:50 NOTICE:
Transferred:   	  375.630 GiB / 3.564 TiB, 10%, 547.027 KiB/s, ETA 10w2d15h27m51s
Errors:                64 (fatal error encountered)
Checks:              3206 / 3206, 100%
Transferred:        22901 / 222965, 10%
Elapsed time:   2h44m13.8s

2023/05/25 18:37:50 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-25 <==
2023/05/27 03:27:12 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/27 03:27:12 ERROR : Fatal error received - not attempting retries
2023/05/27 03:27:12 NOTICE:
Transferred:   	  747.786 GiB / 3.929 TiB, 19%, 1.030 MiB/s, ETA 5w2d16h34m2s
Errors:                64 (fatal error encountered)
Checks:              3227 / 3227, 100%
Transferred:        45217 / 245281, 18%
Elapsed time:   5h43m58.6s

2023/05/27 03:27:12 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-26 <==
2023/05/28 03:12:51 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/28 03:12:51 ERROR : Fatal error received - not attempting retries
2023/05/28 03:12:51 NOTICE:
Transferred:   	  303.884 GiB / 3.495 TiB, 8%, 4.099 MiB/s, ETA 1w2d11h15m8s
Errors:                64 (fatal error encountered)
Checks:              3264 / 3264, 100%
Transferred:        18478 / 218542, 8%
Elapsed time:    2h5m39.3s

2023/05/28 03:12:51 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-28 <==
2023/05/31 03:27:21 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/05/31 03:27:21 ERROR : Fatal error received - not attempting retries
2023/05/31 03:27:21 NOTICE:
Transferred:   	  450.691 GiB / 3.638 TiB, 12%, 4.658 MiB/s, ETA 1w1d7h58m28s
Errors:                64 (fatal error encountered)
Checks:              3302 / 3302, 100%
Transferred:        27466 / 227530, 12%
Elapsed time:   3h22m44.7s

2023/05/31 03:27:21 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-05-31 <==
2023/06/02 02:52:37 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/02 02:52:37 ERROR : Fatal error received - not attempting retries
2023/06/02 02:52:37 NOTICE:
Transferred:   	  260.836 GiB / 3.453 TiB, 7%, 148.169 KiB/s, ETA 38w2d5h20m49s
Errors:                64 (fatal error encountered)
Checks:              3386 / 3386, 100%
Transferred:        15881 / 215945, 7%
Elapsed time:    2h6m32.7s

2023/06/02 02:52:37 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-02 <==
2023/06/04 05:16:44 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/04 05:16:44 ERROR : Fatal error received - not attempting retries
2023/06/04 05:16:44 NOTICE:
Transferred:   	  253.294 GiB / 3.445 TiB, 7%, 588.631 KiB/s, ETA 9w4d12h11m11s
Errors:                64 (fatal error encountered)
Checks:              3394 / 3394, 100%
Transferred:        15416 / 215480, 7%
Elapsed time:   1h57m51.4s

2023/06/04 05:16:44 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-04 <==
2023/06/06 17:16:35 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/06 17:16:35 ERROR : Fatal error received - not attempting retries
2023/06/06 17:16:35 NOTICE:
Transferred:   	  603.872 GiB / 3.787 TiB, 16%, 4.416 MiB/s, ETA 1w1d18h56m14s
Errors:                64 (fatal error encountered)
Checks:              3408 / 3408, 100%
Transferred:        36808 / 236872, 16%
Elapsed time:    4h15m1.7s

2023/06/06 17:16:35 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-06 <==
2023/06/08 20:23:13 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/08 20:23:13 ERROR : Fatal error received - not attempting retries
2023/06/08 20:23:13 NOTICE:
Transferred:   	   69.472 GiB / 3.266 TiB, 2%, 7.783 MiB/s, ETA 4d23h39m57s
Errors:                64 (fatal error encountered)
Checks:              3433 / 3433, 100%
Transferred:         4245 / 204309, 2%
Elapsed time:     37m43.3s

2023/06/08 20:23:13 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-08 <==
2023/06/10 19:18:02 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/10 19:18:02 ERROR : Fatal error received - not attempting retries
2023/06/10 19:18:02 NOTICE:
Transferred:   	  146.954 GiB / 3.341 TiB, 4%, 386.670 KiB/s, ETA 14w4d18h43m30s
Errors:                64 (fatal error encountered)
Checks:              3523 / 3523, 100%
Transferred:         8933 / 208997, 4%
Elapsed time:     1h6m7.3s

2023/06/10 19:18:02 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-10 <==
2023/06/13 11:44:03 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/13 11:44:03 ERROR : Fatal error received - not attempting retries
2023/06/13 11:44:03 NOTICE:
Transferred:   	  176.222 GiB / 3.369 TiB, 5%, 27.239 KiB/s, ETA 3y51w6d14h50m24s
Errors:                64 (fatal error encountered)
Checks:              3532 / 3532, 100%
Transferred:        10547 / 210611, 5%
Elapsed time:   1h29m50.2s

2023/06/13 11:44:03 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-13 <==
2023/06/15 16:10:37 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/15 16:10:37 ERROR : Fatal error received - not attempting retries
2023/06/15 16:10:37 NOTICE:
Transferred:   	  491.288 GiB / 3.677 TiB, 13%, 373.971 KiB/s, ETA 15w1d6h11m31s
Errors:                64 (fatal error encountered)
Checks:              3537 / 3537, 100%
Transferred:        29860 / 229924, 13%
Elapsed time:   3h39m23.4s

2023/06/15 16:10:37 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-15 <==
2023/06/16 21:51:52 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/06/16 21:51:52 ERROR : Fatal error received - not attempting retries
2023/06/16 21:51:52 NOTICE:
Transferred:   	   50.712 GiB / 3.247 TiB, 2%, 498.505 KiB/s, ETA 11w2d17h14m56s
Errors:                64 (fatal error encountered)
Checks:              3568 / 3568, 100%
Transferred:         3002 / 203066, 1%
Elapsed time:      41m4.9s

2023/06/16 21:51:52 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-06-16 <==
2023/07/09 03:14:19 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/09 03:14:19 ERROR : Fatal error received - not attempting retries
2023/07/09 03:14:19 NOTICE:
Transferred:   	   22.003 GiB / 3.220 TiB, 1%, 1.501 MiB/s, ETA 3w4d20h24m36s
Errors:                64 (fatal error encountered)
Checks:              3585 / 3585, 100%
Transferred:         1538 / 201602, 1%
Elapsed time:      13m1.9s

2023/07/09 03:14:19 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-09 <==
2023/07/11 19:53:35 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/11 19:53:35 ERROR : Fatal error received - not attempting retries
2023/07/11 19:53:35 NOTICE:
Transferred:   	    3.699 GiB / 3.201 TiB, 0%, 81.957 KiB/s, ETA 1y17w19h51m45s
Errors:                64 (fatal error encountered)
Checks:              3846 / 3846, 100%
Transferred:          172 / 200236, 0%
Elapsed time:      6m44.5s

2023/07/11 19:53:35 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-11 <==
2023/07/12 22:03:52 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/12 22:03:52 ERROR : Fatal error received - not attempting retries
2023/07/12 22:03:52 NOTICE:
Transferred:   	    7.210 GiB / 3.204 TiB, 0%, 82.688 KiB/s, ETA 1y16w3d11h32m16s
Errors:                64 (fatal error encountered)
Checks:              3854 / 3854, 100%
Transferred:          391 / 200455, 0%
Elapsed time:      7m19.9s

2023/07/12 22:03:52 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-12 <==
2023/07/14 03:37:57 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/14 03:37:57 ERROR : Fatal error received - not attempting retries
2023/07/14 03:37:57 NOTICE:
Transferred:   	    3.567 GiB / 3.201 TiB, 0%, 230.805 KiB/s, ETA 24w4d4h3m30s
Errors:                64 (fatal error encountered)
Checks:              3869 / 3869, 100%
Transferred:          170 / 200234, 0%
Elapsed time:      5m43.6s

2023/07/14 03:37:57 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-14 <==
2023/07/15 21:25:40 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/15 21:25:40 ERROR : Fatal error received - not attempting retries
2023/07/15 21:25:40 NOTICE:
Transferred:   	   13.507 GiB / 3.211 TiB, 0%, 485.290 KiB/s, ETA 11w4d21h11m17s
Errors:                64 (fatal error encountered)
Checks:              3887 / 3887, 100%
Transferred:          794 / 200858, 0%
Elapsed time:     10m52.6s

2023/07/15 21:25:40 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-15 <==
2023/07/16 11:39:51 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/16 11:39:51 ERROR : Fatal error received - not attempting retries
2023/07/16 11:39:51 NOTICE:
Transferred:   	   10.378 GiB / 3.208 TiB, 0%, 18.994 KiB/s, ETA 5y38w1d1h16m46s
Errors:                64 (fatal error encountered)
Checks:              3921 / 3921, 100%
Transferred:          574 / 200638, 0%
Elapsed time:      8m41.3s

2023/07/16 11:39:51 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-16 <==
2023/07/17 11:10:01 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/17 11:10:01 ERROR : Fatal error received - not attempting retries
2023/07/17 11:10:01 NOTICE:
Transferred:   	    4.364 GiB / 3.202 TiB, 0%, 60 B/s, ETA 78y1w2d9h58m52s871ms
Errors:                64 (fatal error encountered)
Checks:              3924 / 3924, 100%
Transferred:          205 / 200269, 0%
Elapsed time:      5m43.6s

2023/07/17 11:10:01 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-17 <==
2023/07/18 06:43:18 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/18 06:43:18 ERROR : Fatal error received - not attempting retries
2023/07/18 06:43:18 NOTICE:
Transferred:   	    6.973 GiB / 3.205 TiB, 0%, 1011.812 KiB/s, ETA 5w4d6h40m7s
Errors:                64 (fatal error encountered)
Checks:              3927 / 3927, 100%
Transferred:          366 / 200430, 0%
Elapsed time:      7m24.8s

2023/07/18 06:43:18 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-18 <==
2023/07/19 15:46:25 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/19 15:46:25 ERROR : Fatal error received - not attempting retries
2023/07/19 15:46:25 NOTICE:
Transferred:   	    6.156 GiB / 3.204 TiB, 0%, 51.056 KiB/s, ETA 2y6w6d8h53m47s
Errors:                64 (fatal error encountered)
Checks:              3931 / 3931, 100%
Transferred:          326 / 200390, 0%
Elapsed time:      7m57.3s

2023/07/19 15:46:25 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-19 <==
2023/07/20 08:51:54 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/20 08:51:54 ERROR : Fatal error received - not attempting retries
2023/07/20 08:51:54 NOTICE:
Transferred:   	    7.709 GiB / 3.205 TiB, 0%, 99.810 KiB/s, ETA 1y4w5d4h51s
Errors:                64 (fatal error encountered)
Checks:              3947 / 3947, 100%
Transferred:          422 / 200486, 0%
Elapsed time:      8m42.2s

2023/07/20 08:51:54 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-20 <==
2023/07/22 18:47:18 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/22 18:47:18 ERROR : Fatal error received - not attempting retries
2023/07/22 18:47:18 NOTICE:
Transferred:   	    7.943 GiB / 3.206 TiB, 0%, 0 B/s, ETA 150y11w4d23h20m10s799ms
Errors:                64 (fatal error encountered)
Checks:              3961 / 3961, 100%
Transferred:          435 / 200499, 0%
Elapsed time:      8m21.1s

2023/07/22 18:47:18 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-22 <==
2023/07/24 17:07:13 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/24 17:07:13 ERROR : Fatal error received - not attempting retries
2023/07/24 17:07:13 NOTICE:
Transferred:   	    4.579 GiB / 3.202 TiB, 0%, 379.906 KiB/s, ETA 14w6d14h3m12s
Errors:                64 (fatal error encountered)
Checks:              3977 / 3977, 100%
Transferred:          243 / 200307, 0%
Elapsed time:      6m42.8s

2023/07/24 17:07:13 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-24 <==
2023/07/25 20:17:34 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/25 20:17:34 ERROR : Fatal error received - not attempting retries
2023/07/25 20:17:34 NOTICE:
Transferred:   	    3.851 GiB / 3.201 TiB, 0%, 8.164 KiB/s, ETA 13y17w3d1h57m15s
Errors:                64 (fatal error encountered)
Checks:              4004 / 4004, 100%
Transferred:          178 / 200242, 0%
Elapsed time:      6m12.3s

2023/07/25 20:17:34 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-25 <==
2023/07/26 16:39:20 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/26 16:39:20 ERROR : Fatal error received - not attempting retries
2023/07/26 16:39:20 NOTICE:
Transferred:   	    6.868 GiB / 3.204 TiB, 0%, 70.663 KiB/s, ETA 1y28w1d6h21m25s
Errors:                64 (fatal error encountered)
Checks:              4009 / 4009, 100%
Transferred:          375 / 200439, 0%
Elapsed time:      5m51.0s

2023/07/26 16:39:20 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-26 <==
2023/07/28 00:34:14 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/28 00:34:14 ERROR : Fatal error received - not attempting retries
2023/07/28 00:34:14 NOTICE:
Transferred:   	    3.558 GiB / 3.201 TiB, 0%, 1.224 MiB/s, ETA 4w3d17h13m31s
Errors:                64 (fatal error encountered)
Checks:              4025 / 4025, 100%
Transferred:          170 / 200228, 0%
Elapsed time:      10m5.2s

2023/07/28 00:34:14 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-28 <==
2023/07/31 05:36:22 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/07/31 05:36:22 ERROR : Fatal error received - not attempting retries
2023/07/31 05:36:22 NOTICE:
Transferred:   	    6.720 GiB / 3.204 TiB, 0%, 778.103 KiB/s, ETA 7w2d1h38m47s
Errors:                64 (fatal error encountered)
Checks:              4044 / 4044, 100%
Transferred:          368 / 200432, 0%
Elapsed time:      7m58.5s

2023/07/31 05:36:22 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-07-31 <==
2023/08/02 09:52:15 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/08/02 09:52:15 ERROR : Fatal error received - not attempting retries
2023/08/02 09:52:15 NOTICE:
Transferred:   	    3.418 GiB / 3.201 TiB, 0%, 41.127 KiB/s, ETA 2y33w5d7h6m33s
Errors:                64 (fatal error encountered)
Checks:              4068 / 4068, 100%
Transferred:          152 / 200216, 0%
Elapsed time:      7m22.2s

2023/08/02 09:52:15 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

==> snapshot_2023-08-02 <==
2023/08/03 17:49:18 ERROR : Google drive root 'oak/restic': not deleting directories as there were IO errors
2023/08/03 17:49:18 ERROR : Fatal error received - not attempting retries
2023/08/03 17:49:18 NOTICE:
Transferred:   	    2.885 GiB / 3.200 TiB, 0%, 93.616 KiB/s, ETA 1y8w3d11h32m15s
Errors:                64 (fatal error encountered)
Checks:              4075 / 4075, 100%
Transferred:          130 / 200194, 0%
Elapsed time:      5m33.3s

2023/08/03 17:49:18 Failed to sync with 64 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
3

I think your values for transfers and checkers are too high.
I suggest you set both of them to 6.

4

When you hit quotas you have to wait 24h and then try again.

@jaimedelano is right. I would leave it to default. 6 might be too much

You can only upload 750GB per 24h to gdrive so limit your daily transfer by using --max-transfer flag.

5

Thank you for the recommendations.

When you hit quotas you have to wait 24h and then try again.

The logs show several occasions where more than 24h elapsed between jobs and the subsequent job was still immediately rate limited so this appears to not be a factor. Which is fortunate because that would be annoying to implement.

6

The 24 hours really isn’t documented that well as Google has many restrictions so waiting 24 is generally a good start.

Other than try later, there is not much else to do.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.