Google Photos upload issue

elliott · March 22, 2020, 9:45pm

What is the problem you are having with rclone?

Getting 429s ("size of uploads" limit) when trying to upload a ~1.5 MB image to Google Photos. I've tried with a couple different files, including one I downloaded from Google Photos.

What is your rclone version (output from `rclone version`)

rclone v1.51.0

os/arch: linux/amd64

go version: go1.14

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Arch Linux, 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy rclone-test.jpg "eeshugerman-gphotos:album/rclone-test"

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

2020/03/22 15:17:57 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "copy" "-vv" "--retries" "1" "rclone-test.jpg" "eeshugerman-gphotos:album/rclone-test"]
2020/03/22 15:17:57 DEBUG : Using config file from "/home/elliott/.config/rclone/rclone.conf"
2020/03/22 15:17:57 DEBUG : Google Photos path "album/rclone-test": NewObject: remote="rclone-test.jpg"
2020/03/22 15:17:57 DEBUG : Google Photos path "album/rclone-test": List: dir=""
2020/03/22 15:18:08 DEBUG : Google Photos path "album/rclone-test": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/03/22 15:18:08 DEBUG : Google Photos path "album/rclone-test": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": >List: err=
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": >NewObject:
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: Need to transfer - File not found at Destination
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:09 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027photoslibrary.googleapis.com\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:09 DEBUG : pacer: Rate limited, increasing sleep to 1.736407133s
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 1/10
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:09 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:09 DEBUG : pacer: Rate limited, increasing sleep to 2.572993218s
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 2/10
2020/03/22 15:18:09 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:09 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:11 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:11 DEBUG : pacer: Rate limited, increasing sleep to 4.723416159s
2020/03/22 15:18:11 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:11 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:11 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 3/10
2020/03/22 15:18:11 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:11 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:13 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:13 DEBUG : pacer: Rate limited, increasing sleep to 8.161982225s
2020/03/22 15:18:13 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:13 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:13 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 4/10
2020/03/22 15:18:13 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:13 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:18 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:18 DEBUG : pacer: Rate limited, increasing sleep to 16.437296871s
2020/03/22 15:18:18 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:18 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:18 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 5/10
2020/03/22 15:18:18 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:18 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:26 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:26 DEBUG : pacer: Rate limited, increasing sleep to 16.485556293s
2020/03/22 15:18:26 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:26 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:26 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 6/10
2020/03/22 15:18:26 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:26 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:43 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:43 DEBUG : pacer: Rate limited, increasing sleep to 16.345996536s
2020/03/22 15:18:43 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:43 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:43 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 7/10
2020/03/22 15:18:43 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:43 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:18:57 INFO :
Transferred: 9.560M / 10.926 MBytes, 88%, 201.707 kBytes/s, ETA 6s
Transferred: 0 / 1, 0%
Elapsed time: 48.5s
Transferring:
                          rclone-test.jpg:  0% /1.366M, 0/s, -
2020/03/22 15:18:59 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:18:59 DEBUG : pacer: Rate limited, increasing sleep to 16.205685132s
2020/03/22 15:18:59 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:18:59 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:18:59 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 8/10
2020/03/22 15:18:59 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:18:59 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:19:16 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:19:16 DEBUG : pacer: Rate limited, increasing sleep to 16.097443423s
2020/03/22 15:19:16 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:19:16 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:19:16 DEBUG : rclone-test.jpg: Received error: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests) - low level retry 9/10
2020/03/22 15:19:16 DEBUG : Google Photos path "album/rclone-test": Put: src=rclone-test.jpg
2020/03/22 15:19:16 DEBUG : rclone-test.jpg: Update: src=rclone-test.jpg
2020/03/22 15:19:32 DEBUG : pacer: low level retry 1/1 (error {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests))
2020/03/22 15:19:32 DEBUG : pacer: Rate limited, increasing sleep to 16.960667254s
2020/03/22 15:19:32 DEBUG : rclone-test.jpg: >Update: err=couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:19:32 DEBUG : Google Photos path "album/rclone-test": >Put:
2020/03/22 15:19:32 ERROR : rclone-test.jpg: Failed to copy: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:19:32 ERROR : Attempt 1/1 failed with 1 errors and: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)
2020/03/22 15:19:32 Failed to copy: couldn't upload file: {
"code": 8,
"message": "Quota exceeded for quota metric \u0027Size of uploads\u0027 and limit \u0027Size of uploads per day\u0027 of service \u0027\u0027 for consumer \u0027project_number:202264815644\u0027."
} (429 429 Too Many Requests)

asdffdsa · March 22, 2020, 10:06pm

hello and welcome to the forum,

this is a not rclone bug.

google has limits on uploads in a given time period

given the errors in the log, you have exceeded certain upload limits by google.
you are getting the same messages from google, over and over.

"Quota exceeded Size of uploads per day

https://developers.google.com/photos/library/guides/api-limits-quotas
"If the quota of requests to the Library API is exceeded, the API returns an error code 429 and a message that the project has exceeded the quota"

@thestigma...

thestigma · March 22, 2020, 11:00pm

I'm not very experienced with the Gphotos actually, but I afree with asdffdsa that this is almost certainly an error from hitting a quota (I mean - the error from the server is pretty clear on that part). Most likely triggered due to a very high amount of uploads in a 24hr period. It should reset on it's own tomorrow and you can resume then.

asdffdsa · March 22, 2020, 11:02pm

not sure what that means, but i like it.

thestigma · March 22, 2020, 11:04pm

It's something tasty you eat with your covfefe I'm sure...

elliott · March 22, 2020, 11:17pm

I read the error message and am familiar with the concept of API limits. I have a hard time believing that uploading a single 1.5MB image exceeds any of Google's quotas.

I tried with --tpslimit 8 to avoid the documented 10,000 requests per day limit (though this issue apparently has to do with file size which, would be a different quota) -- same result.

I should say, I did try uploading several thousand photos with Rclone this morning and then killed the process after a couple minutes. HOWEVER, I also tried to upload the single 1.5MB test image to a different Google account, from a new IP, and got the same error. I'll try again tomorrow to confirm, but considering a different account and IP didn't change anything, I'm not optimistic.

Also, this may be a topic for another thread, but shouldn't Rclone accommodate daily limits? That is, if I use a single rclone copy command to upload 100K photos, it should spread out the requests over a couple days if need-be.

asdffdsa · March 22, 2020, 11:31pm

killed the process,
perhaps you did not kill that process.

google and all of its too many, never ending limits and usage secrets.

we cannot expect rclone,
free,
open source to deal with all of them.

just wait a few hours and try again.

you can use wasabi, that does not have those limits, but it will cost you $.

elliott · March 22, 2020, 11:38pm

OK, I'll try again in a couple days with the single 1.5MB file -- if I get the same error, would you then agree that this is a bug (if not in Rclone then in the Google Photos API)?

In the meantime I'll check out Wasabi.

asdffdsa · March 22, 2020, 11:43pm

would you then agree that this is a bug
not bugs, but limits with free google services.

no need to wait days.
at some point soon, that upload will not fail. keep trying.

google storage, which has a free tier, given its many limits, can be a very good, but limited solution.

calisro · March 23, 2020, 2:45am

I'm pretty sure the API in use is a shared one. Isn't it @ncw?

ncw · March 23, 2020, 11:01am

Yes that is the shared google photos project.

So making your own project ID should help, although note that the limits are really low

These are what rclone's quotas look like at the moment

So not too bad...

It might just be worth slowing things down a bit with --tpslimit 3 say and see if that helps.

elliott · March 23, 2020, 4:51pm

Ohh that explains it! Thanks!

I was wondering why I didn't see any activity in that dashboard for my account.

It was working earlier today, but I'm gonna need a couple days of interrupted requests to upload everything, so yeah I'll try using my own project ID.

Is there any documentation on this that I should know about? I'm thinking I'll need to create a project in the Google Cloud dashboard with OAuth credentials and then pass the client ID and secret to rclone config -- is that correct?

elliott · March 24, 2020, 12:57am

Quick update: It's working splendidly now that I've created my own project ID! It's actually a good deal faster than I was anticipating (based on my past experience with the official Backup & Sync app) -- I just did a test run of ~2 GB in 30 minutes!

ncw · March 24, 2020, 7:50am

Interesting... so my project ID must be overloaded somehow. It looks like it is being rate limited according to the graphs but not in a way that is recorded in the cloud console.

elliott · March 25, 2020, 1:16am

Yep seems there are limits on the (total?) size of uploads that are neither documented nor tracked in the console .

en3r0 · April 8, 2020, 8:05pm

Would like to confirm I ran into this same situation today. Creating my own project ID fixed it and made the uploads much much faster.

system · April 11, 2020, 8:05pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.