Google Drive limits reached with multiple service accounts?

[cdemi (Christopher Demicoli)]

What is the problem you are having with rclone?

Hi everyone, I am a new rclone user. I am trying to upload 5TB to Google Suite Google Drive. As I have understood the documentation, in order to upload more than 750GB a day, you have to create multiple service accounts.

I have created 4 service accounts and the actual user account where I want to upload to (so I have 5 potential keys), however I am noticing that once I hit the limit for one service account and I switch to another service account, I am still blocked from Google Drive and no matter which other account I try, I cannot continue to upload.

What is your rclone version (output from rclone version)

rclone v1.52.3

• os/arch: windows/amd64
• go version: go1.14.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --progress --transfers=15 --fast-list --drive-stop-on-upload-limit --max-transfer 745G hetzner-local:z:/angel/Angel_1.hbk/Pool gdriverclone2:Backups/Angel/Angel_1.hbk/Pool

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = xxxx
client_secret = xxxx
scope = drive
token = xxxx

[hetzner-local]
type = local
nounc = false

[gdriverclone1]
type = drive
service_account_file = rclone1.json
impersonate = xxx@xxx.xxx

[gdriverclone2]
type = drive
service_account_file = rclone2.json
impersonate = xxx@xxx.xxx

[gdriverclone3]
type = drive
service_account_file = rclone3.json
impersonate = xxx@xxx.xxx

[gdriverclone4]
type = drive
service_account_file = rclone4.json
impersonate = xxx@xxx.xxx

[gdriverclone5]
type = drive
service_account_file = rclone5.json
impersonate = xxx@xxx.xxx

[gdriverclone6]
type = drive
service_account_file = rclone6.json
impersonate = xxx@xxx.xxx

A log from the command with the -vv flag

2020-08-28 14:28:14 ERROR : 0/29/1227.bucket.67: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails&supportsAllDrives=true&uploadType=resumable&upload_id=ABg5-UyH8EFJ3la8ablei-6N_frH969anctxeLAJ5C7Q7sRhyMIKptiUAN9Ub2FAZu4hqh9ZbgtCl2aqV5TzhcXhL0c": context canceled
2020-08-28 14:28:14 ERROR : 0/29/1231.bucket.67: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails&supportsAllDrives=true&uploadType=resumable&upload_id=ABg5-UzErUaR_z4_HXtnRhjT1fyU5Sk2-AYmDdCZg2iKPxf1JyBOjcnO-Zx4cV1qf4JM1yYkxO9qhtDzZDAl395_ag": context canceled
2020-08-28 14:28:14 ERROR : 0/29/1223.bucket.67: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2020-08-28 14:28:14 ERROR : 0/29/1226.bucket.67: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails&supportsAllDrives=true&uploadType=resumable&upload_id=ABg5-UxF6Je-twDOd9leKvN1EmedViRiSYBs0TBJgYC65KOa2uEiFVgZDetpiVQmQZ1hTdtvW-fUAXT1iBpR0du7aEaEZjzwIg": context canceled

Am I doing something wrong? Or have I understood the documentation incorrectly?

[cdemi (Christopher Demicoli)]

Actually, I think I figured it out. It's because I'm using impersonate. I have shared the folder from the user to the service account and am using root_folder_id instead. So far it looks like it's working!

Edit: It didn't work. Now I'm getting:

2020-08-28 14:46:16 ERROR : chunk_index/1828.idx.154: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded

I assume this is the quota of the service account

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

you read more about that specific error and solutions.
https://developers.google.com/drive/api/v3/handle-errors#resolve_a_403_error_user_rate_limit_exceeded

[asdffdsa (jojothehumanmonkey)]

you can search the forum, for example,

Gdrive storageQuotaExceeded error

I’ve been receiving the following error for the past few days googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded I realized I had setup my rclone config incorrectly for working with PLEX. I had it setup as drive1 -> crypt1 and accoring to this guide it should be drive -> cache -> crypt. So I created a new config on the same gdrive account: drive2 -> cache -> crypt2 and then tried to copy from crypt1 to crypt2, it seemed to work for a day or so before …

[darthShadow (Anagh Kumar Baranwal)]

Without impersonate, it will use the service_account's storage quota which is a measly 15 GB. You need to set shared_with_me to True to copy to the shared folder but even that is not always foolproof for normal drive.

Docs: https://rclone.org/drive/#drive-shared-with-me

Easiest & Fastest way is to create a Team/Shared Drive, share it with all the service accounts, upload into that and when the full upload is done, move it into your main drive.

[Vikral (Vikral Thakur)]

did you find any solution for this? I used shared_with_me when I upload a file, using a service account, in a shared folder, I get Drive storage quota has been exceeded

[Berzerk90er]

Same Problem for me. Anyone have a Idea?