Google Drive pacer issue

[nle]

Ok, I think I've come a bit further debugging/explanation, but first the backstory.

I'm trying to migrate from a aging local file server to a cloud solution. I allready have GSuite, but unfortunately, Google Drive File Stream is alot worse to work on than Dropbox (GDFS ie. don't let you update modified PSDs in InDesign). So Dropbox it was, but I'm still keeping GSuite for mail, documents, etc. (and as a external backup – don't put all eggs in one basket etc.). The server also backed up to an cryptfs on GSuite.

Fast forward a bit, and I have successfully copied (and verified via rclone check) all the TB of files from the server to Dropbox.

I then bought a VPS, and set up my sync script there (all the same except from copying from a local drive, it now copies from dropbox). I ran the script manually.

This worked fine, but then I got hit with the rate limit (I think), not totally sure if it was both the rate limit (first) and then the StorageQuotaExceeded. Trying to bypass this and get my backup done, I created three service accounts with edit rights, added them to a group, and added the group to my "backup" folder in my personal gdrive.

Running the sync script again, seemed to work for a while, but then I got hit by the googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded, and I also noticed it uploading files that allready was there (duplicates). (I get the storageQuotaExceeded error now from both my regular user and service accounts. I currently have an open issue with google on this.)

one important note: I do not backup to a Team Drive due to the 400.000 files limit, but to a dedicated folder in my personal drive. And this is one of the reasons for the issues

What I realized:

• It was after I introduced service accounts the duplicate issue happened. Not sure why, since they should see all files and have edit permissions.
• (Can I bypass this, with using the impersonate setting?)

What I don't know is:

• Why – if the service account has edit rights and can see the files – they started uploading some duplicates (while other files got left alone because they excisted)
• Why I got hit by googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded since I have unlimited space.

The plan is to:

• Get rid of duplicates by running "rclone delete" from the service account. I get permission errors on files that the service account doesn't own, and it will delete the rest that it do own. (Currently doing this)
• Do a test with the impersonate flag to see if I can use that with the service account. If yes, try to continue with the sync command.
• See if I still get the googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded and if yes, wait for Google to com back to me.

Thanks for all the input so far.

[random404]

Just use rclone dedupe with your standard account dude, no need to use rclone delete

[nle]

I tried that, but that did not work. I don't have permission to dedupe (delete) the files created by the service account. So the service accounts can't delete mye users files, and the user can't delte the service accounts files. T

The story would be different if they were on a Team Drive.

[nle]

Doh. I can just delete the service accounts, and voila that removes the files they own (i.e all the dupes). I wish I thought about that a while ago.

[random404]

Create a teamdrive, add your accounts and just do server side moves to it with all your accounts. After everything is on it, you should be able to just run rclone dedupe

[nle]

Can't do that due to Team Drives 400000 file limit.

rclone size gdrive: --fast-list -v
Total objects: 716098
Total size: 7.970 TBytes (8762697034083 Bytes)