Union multi service account with same Team Drive for Restic backup

Unknown_Name · October 11, 2023, 12:53pm

I'm trying to resolve the error userRateLimitExceeded while using rclone/GoogleTeamDrive as backend for restic backup.

My plan is:

Create multiple service accounts on Google
In rclone, create multiple configurations with the same Team Drive but different service accounts.

In rclone, create a union config use all configs above, example

[TeamDrive01]
type = drive
client_id = client-id-1
scope = drive
service_account_file = D:\rclone-key\service-account-1.json
team_drive = 123123123123
root_folder_id =

[TeamDrive02]
type = drive
client_id = client-id-2
scope = drive
service_account_file = D:\rclone-key\service-account-2.json
team_drive = 123123123123
root_folder_id =

...
[TeamDriveAll]
type = union
upstreams = TeamDrive01: TeamDrive02: TeamDrive03: TeamDrive04: TeamDrive05:
action_policy = eprand
create_policy = eprand
search_policy = eprand

Then, in restic: restic backup -r rclone:TeamDriveAll:RESTIC D:\SomeDatabase --no-scan --iexclude-file D:\restic-exclude.txt

Do you think this approach will work? Any advice is very appreciated.

What is the problem you are having with rclone?

Restic cannot backup with error userRateLimitExceeded

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: Microsoft Windows Server 2012 R2 Datacenter (64 bit)
- os/kernel: 6.3.9600.19023 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive (Shared Drive for Team)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

restic backup -r rclone:TeamDriveXXX:RESTIC D:\SomeDatabase --no-scan --iexclude-file D:\restic-exclude.txt

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[TeamDriveXXX]
type = drive
client_id = XXX
scope = drive
service_account_file = D:\rclone-key\service_key_rclonexxx.json
team_drive = XXX

A log from the command that you were trying to run with the `-vv` flag

This log from restic

rclone: 2023/10/11 16:45:54 ERROR : locks/fd610a29e15669dfadcc901a1e20c0910efeb2a08c2c1830c212ad6d9bd8549c: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
unable to create lock in backend: server response unexpected: 500 Internal Server Error (500)

asdffdsa · October 11, 2023, 1:39pm

welcome to the forum,

not sure the exact issue as i do not use team_drive
but i would remove client_id = XXX, as it is already specified in the service account file.

this is what i use

[gdrive.saf]
type = drive
scope = drive
service_account_file = c:\data\rclone\rr\other\gdrive.sa\it.org.json

[gdrive.sac]
type = drive
scope = drive
service_account_credentials = {"type": "service_account", "project_id": "redacted",   "private_key_id": "redacted",   "private_key": "redacted",   "client_email": "redacted",   "client_id": "redacted",   "auth_uri": "https://accounts.google.com/o/oauth2/auth",   "token_uri": "https://oauth2.googleapis.com/token",   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/redacted.iam.gserviceaccount.com" }

kapitainsky · October 11, 2023, 2:50pm

It is interesting idea - but no clue if it will work or not. I am curious myself what your results will be.

But the most straightforward way of dealing with rate quotas is to tweak rclone flags using restic flag -o rclone.args (refer to their docs) and limit number of API calls (less concurrent transfers, less transactions per second) - you will find plenty of examples on this forum.

Animosity022 · October 11, 2023, 3:00pm

That setup actually makes things far, far worse.

What the OP probably wants is a round robin config, which doesn't exist.

By having more remotes on the same union, you query each remote in the union for everything.

So that's quite an amount of overhead when compared to using 1 remote.

What's the reason you want to use a bunch of service accounts over one remote?

Unknown_Name · October 11, 2023, 3:09pm

What's the reason you want to use a bunch of service accounts over one remote?

I want to avoid error userRateLimitExceeded by using multi service accounts, my local storage have a lot of small files, with more than 2.000.000 files/folders (total: ~4TB)

By having more remotes on the same union, you query each remote in the union for everything

Sadly

kapitainsky · October 11, 2023, 3:13pm

You are using restic though - so does not matter how many files you have locally. Restic by default uses 16MB pack size. On top of tweaking rclone flags you can also increase restic pack size to 128MB - as a result you will have 8 times less files on your remote.

Unknown_Name · October 11, 2023, 3:14pm

I has config restic packs to RESTIC_PACK_SIZE=100 (100MB) but still hit this error.

kapitainsky · October 11, 2023, 3:16pm

Yes Google drive is not something designed for intensive workloads like restic backup. You have to limit number of API calls or use another cloud provider.

Animosity022 · October 11, 2023, 3:17pm

That's not how it works though. You have a 1 billion API hits per day. You are not going to hit that. I'd advise to try your test with a single remote and post the complete debug log.

Animosity022 · October 11, 2023, 3:19pm

Google has quite more API hits than almost any other provider I've used and their rate limiting is somewhat documented and easy to tune.

Generally folks mix API rate limits and upload/download limits as they are confused to be the same thing, which they are not.

Unknown_Name · October 11, 2023, 3:22pm

As I see in Google Cloud Console, the Google Drive API limit only 20000 per days for service account

kapitainsky · October 11, 2023, 3:26pm

I was testing Google for my own backup and sometimes even saw rate limits errors when running simple content listing... Having other options available I did not investigate in details. My limited usage impression was that it is quite unpredictable service for heavy workloads (many files, heavy mixed read and writes ops).

But seeing many google speed related posts on this forum I think that with right flags it is tweakable - even if not fast but can be made stable.

kapitainsky · October 11, 2023, 3:28pm

Hard to say without debug log what exactly limit you are hitting. Maybe nothing about daily quotas but rather much shorted period of time limits.

Animosity022 · October 11, 2023, 3:32pm

You get roughly 200 API hits per second with a standard per user quota on Google these days.

Rclone is generally configured around that and does well.

Those are generally not API related though and more so on peering/wrong flags/wonky setups/bad internet/etc.

Unknown_Name · October 13, 2023, 9:45am

I update the debug logs, rate limit seem apply to all service account of same Google Account / Project

C:\Users\Administrator>SET RCLONE_VERBOSE=2

C:\Users\Administrator>restic backup -r rclone:DemoDrive:RESTIC D:\Database\DemoDB --no-scan --iexclude-file D:\Scripts\restic-exclude.txt
rclone: 2023/10/13 16:36:17 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "DemoDrive:RESTIC"]
rclone: 2023/10/13 16:36:17 DEBUG : Creating backend with remote "DemoDrive:RESTIC"
rclone: 2023/10/13 16:36:17 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
rclone: 2023/10/13 16:36:17 DEBUG : Setting drive_pacer_min_sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
rclone: 2023/10/13 16:36:17 DEBUG : Setting drive_pacer_burst="200" from environment variable RCLONE_DRIVE_PACER_BURST
rclone: 2023/10/13 16:36:17 DEBUG : DemoDrive: detected overridden config - adding "{cHldw}" suffix to name
rclone: 2023/10/13 16:36:17 DEBUG : Setting drive_pacer_min_sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
rclone: 2023/10/13 16:36:17 DEBUG : Setting drive_pacer_burst="200" from environment variable RCLONE_DRIVE_PACER_BURST
rclone: 2023/10/13 16:36:18 DEBUG : fs cache: renaming cache item "DemoDrive:RESTIC" to be canonical "DemoDrive{cHldw}:RESTIC"
rclone: 2023/10/13 16:36:18 DEBUG : file-7931991868572922989: GET request error: object not found
rclone: 2023/10/13 16:36:19 DEBUG : keys: list request
repository d94a78f5 opened (version 2, compression level auto)
rclone: 2023/10/13 16:36:22 DEBUG : locks: list request
rclone: 2023/10/13 16:36:24 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:24 DEBUG : pacer: Rate limited, increasing sleep to 1.136305597s
rclone: 2023/10/13 16:36:24 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:24 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:25 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 271.880647ms: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:27 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:27 DEBUG : pacer: Rate limited, increasing sleep to 1.87923677s
rclone: 2023/10/13 16:36:27 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:27 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:27 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 1.080042061s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:29 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:29 DEBUG : pacer: Rate limited, increasing sleep to 1.97833867s
rclone: 2023/10/13 16:36:29 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:29 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:30 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 627.108891ms: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:32 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:32 DEBUG : pacer: Rate limited, increasing sleep to 1.084889597s
rclone: 2023/10/13 16:36:32 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:32 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:33 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 1.913874595s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:36 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:36 DEBUG : pacer: Rate limited, increasing sleep to 1.496964676s
rclone: 2023/10/13 16:36:36 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:36 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:36 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 3.501716316s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:41 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:41 DEBUG : pacer: Rate limited, increasing sleep to 1.082219298s
rclone: 2023/10/13 16:36:41 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:41 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:41 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 3.662423413s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:46 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:46 DEBUG : pacer: Rate limited, increasing sleep to 1.187066952s
rclone: 2023/10/13 16:36:46 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:46 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:46 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 8.255680315s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:36:55 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:36:55 DEBUG : pacer: Rate limited, increasing sleep to 1.669166271s
rclone: 2023/10/13 16:36:55 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:55 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:36:56 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 6.402950199s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:37:03 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:37:03 DEBUG : pacer: Rate limited, increasing sleep to 1.216514674s
rclone: 2023/10/13 16:37:03 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:37:03 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:37:03 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 17.810737405s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:37:18 INFO  :
rclone: Transferred:        2.125 KiB / 2.125 KiB, 100%, 15 B/s, ETA 0s
rclone: Errors:                18 (retrying may help)
rclone: Transferred:            3 / 3, 100%
rclone: Elapsed time:       1m5.3s
rclone:
rclone: 2023/10/13 16:37:22 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:37:22 DEBUG : pacer: Rate limited, increasing sleep to 1.455943789s
rclone: 2023/10/13 16:37:22 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:37:22 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:37:22 DEBUG : pacer: Reducing sleep to 0s

st error: object not found
Save(<lock/18c0bd3527>) returned error, retrying after 9.946527965s: server response unexpected: 500 Internal Server Error (500)
rclone: 2023/10/13 16:37:33 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
rclone: 2023/10/13 16:37:33 DEBUG : pacer: Rate limited, increasing sleep to 1.441577489s
rclone: 2023/10/13 16:37:33 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 put error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:37:33 ERROR : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Post request
 rcat error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
rclone: 2023/10/13 16:37:34 DEBUG : pacer: Reducing sleep to 0s
rclone: 2023/10/13 16:37:34 DEBUG : locks/18c0bd3527753ea8cf56d5f3298584327c56456931747725777e609c75b539bf: Delete request error: object not found
unable to create lock in backend: server response unexpected: 500 Internal Server Error (500)

C:\Users\Administrator>

system · November 12, 2023, 9:45am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.