Google Photos - Persistent Quota Issue

Help and Support
1

What is the problem you are having with rclone?

Have been unable to use rclone in several days because of a quota issue.
2021/05/05 09:37:46 ERROR : FOLDER/SUBFOLDER/FILENAME.mp4: Failed to copy: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027photoslibrary.googleapis.com\u0027 for consumer \u0027project_number:202264815644\u0027."

*Yes, I am on the general rClone API/App ID config for this project- I did not have my own setup when I started and ran into the issue of rClone being unable to access folders once I switched to my own, so I switched back to generic to complete this sync rather than completely start over. That decision might be biting me now...

What is your rclone version (output from rclone version)

I am NOT on the latest version of rclone - wanted to finish this project before updating.
rclone v1.54.1

  • os/arch: windows/amd64
  • go version: go1.15.8

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 x64

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

rclone copy F:\TSMA tsma:album/ --exclude .thumbs/** --log-file=C:\Users\User\Desktop\rclone\logs\rclone4KFB.txt --log-level NOTICE

Can provide additional logs/config info if needed, but doubting that it's my setup and rather a more broad Google Photos issue.
Either Google Photos has clamped down on rClone's general API project cap or someone is burning through the quota every day.
I tried waiting a full 24 hours before trying again and wasn't able to get a single photo to upload.

Others able to use Google Photos right now?

2

Waited another 48 hours and tried again
Worked for about 5 hours and was able to upload ~3GB of photos.

Checking logs I got an unusually high number of 409 errors.
I have usually disregarded those in reasonable quantities assuming they'd work on the next pass.

But then

Failed to copy: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027photoslibrary.googleapis.com\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)

Started again.

Not sure if note worthy, but I do pay for Google One (100GB) and am well under quota (<20% use? I frequently "recover space" by having Goolge Photos downsize my uploads after the fact)

Anyone able to use rClone for Google Photos on the generic/general use AppID?

3

I have updated to the newest version of rClone:
rclone-v1.55.1-windows-amd64

& renewed my google access token via config

Tried again, same error on first file. Ended the process (I always reboot just to be sure..not sure closing the command prompt window in Windows10 does it?)

Will try again tomorrow.

4
  1. update rclone.
  2. post config file.
  3. make sure to use your own client id.
  4. use --log-level DEBUG for more details.
  5. no need to reboot between runs.
    on the command line
    press ctrl+c to kill an instance of rclone running.
    press the up arrow to recall the last command and edit it or run it again.
1 Like
5

Thanks for the reply

6

sure
also, very useful on the command line with rclone is --progress

7

Let me know if I included anything dangerous :x
I have never posted full log to pastebin

Thanks for the --progress tip also, might add that to my command line.
Only discovered rclone about a month ago and I'm racing against Google's June 1st unlimited storage cutoff deadline to get all my photos uploaded

8

i do not use google products myself.
now that we have a debug log, we have experts who can help and hopefully one will stop by soon.

if you need help with windows command line, i can help.

9

I also have this problem.

10

According to the google console about 1/2 of the uploads are failing!

image
image748×562 84.8 KB

However also according to the google console, the client_id is within its quotas.

So I'd guess you are hitting some other kind of rate limit.

Suggestions

  • --tpslimit 5 - to slow rclone down
  • --transfers 1 to only transfer one file at once
  • --checkers 1 to only read one directory at once

You will have to tweak those numbers.

1 Like
11

Wondering if I'm hitting the same issue that was discussed her:
https://forum.rclone.org/t/google-photos-upload-issue/15092. I'll definitely use my own ClientID on future projects to avoid this issue.

I will try the limit commands you suggested Nick thanks, going to wait until tomorrow in hopes everything resets. Do you happen to know what google's time zone is for resetting?

I take it there is no Google Photos API equivalent of rclone about remote: for me to check the quota usage nor '--drive-stop-on-upload-limit' like with google drive API? Any other way for me to provide more info to forum on what type of quota I might be hitting?

12

Google are quite opaque about API limits in general, and even more so with Google Photos API.

The Google Photos API seems to be the unloved step child in the API family - it has so many things missing and it is generally frustrating!

1 Like
13

I am stumped on what quota I would be hitting.
Retried after midnight and it started ok (very slowly with the limits Nick suggested and I also added an exclude file to make sure no large non-media photos were causing the problem).
But it died after only a couple hours:

rclone copy F:\TSMA tsma:album/ --log-file=C:\Users\User\Desktop\rclone-v1.55.1-windows-amd64\logs\rclone4KFB.txt --log-level DEBUG --progress --exclude-from exclude-file.txt --tpslimit 5 --transfers 1 --checkers 1
Transferred: 651.210M / 1.329 GBytes, 48%, 18.258 kBytes/s, ETA 11h3m5s
Errors: 108 (retrying may help)
Transferred: 8017 / 18019, 44%
Elapsed time: 10h16m47.4s
Transferring:
FBA/Liuzzi/Photo…9274_32845300_5415.jpg: 0% /38.100k, 0/s, -^C

I only got 8000 files (600MB) transferred in <3 hours before starting to get u0027 errors.

Killed the process after checking the log file and seeing many u0027 errors. (Doesn't seem to show clearly on the progress screen, so I occasionally copy/paste a copy of the active log file to check on it).
I have pasted an excerpt of the final ~5 minutes into pastebin:

More of the same re u0027 but also some info on "low level retry" and "pacer"

I have also resorted to removing "completed" folders from the source folder (anything alphabetically before FBA/L* I have temporarily moved to another folder to limit rclone's need to check prior folders and waste calls/quota). But this seemingly didn't help much as I had already done that prior to last night's attempt.

Any other suggestions?

14

:frowning:

You can try lowering --tpslimit - this will slow things down further but may enable it to continue.

That isn't a great deal of data

And that isn't a huge number of files :frowning:

15

Tried again last night with similar result. <1.5 hours in and hit the u0027 issue.

--tpslimit 1 --transfers 1 --checkers 1
Transferred: 288.963M / 1.080 GBytes, 26%, 8.539 kBytes/s, ETA 1d3h12m36s
Errors: 180 (retrying may help)
Transferred: 1865 / 11867, 16%
Elapsed time: 9h45m44.3s

Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027photoslibrary.googleapis.com\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)

Think I better look into other methods for getting my photos backed up.
Thanks for everyone's input.

16

@ressof were you ever able to find a solution/work around?

17

I maked my own client-id. That solved my problem.

1 Like
18

You should be able to move to your own client id and continue where you were. Just make sure the google account that you used stays the same.

19

Are you sure about this?

It was my understanding that the rclone clientID was what mattered for access privileges.
I ran into this issue a month ago when first getting into rclone.
100% sure I was using the same google account because that's the only place I am uploading my photos to :slight_smile: ... only thing that had changed was switching over to my own clientID.

If there's a work around, I'd love to learn it.
I had the issue where all albums (ie folders) that were created by the rclone general clientID were inaccessible to rclone when acting as my personal clientID :\

What API limits do you have on your personal clientID?

If you meant that I could continue forward with m personal clientID (ie upload albums L->Z), then that is probably true. But my clientID will never be able to check/update albums A-K, and rclone general use clientID will never be able to check/update albums L->Z...making all future syncs/updates cumbersome.

20

I'm using rclone sync and rclone mount.

Both of them initially used the default client id. Later on switched to my own client id using this manual: Google drive
Then edited the rclone.conf file manually to add the new client id to it.
Did not need to reupload any data an could continue as if nothing happened.

These are my limits according to google and they work just fine.
image