Uploading already existing photos to Google photo

I plan to upload my complete photo/video archive to google photo. Some (the most recent) have already been uploaded (directly from my phone) and is not placed in any album.
My question is; what happends when i upload the same photo/video again from my computer (now it is placed in an folder).

My hope is that the file is uploaded and placed in the corresponding folder/album in google photo.

It did say something about duplicates in the doc, but it was unclear if this was only valid for duplicates that had been uploaded by rclone both times.

Wasn't sure how to interpret what was written on the google API-page either... (couldn't add link, but its under this topic: manage-albums#adding-items-to-album)

/Anton

I think as long as the file has the name name then everything will work out.

Google deduplicates images so if you upload the same image again, it gives you the previous one.

This works fine in general, except for the filename - this will be the filename of the original upload rather than the new upload.

I am currently in the process of uploading my photo library to Google Photos (rclone copy source:sourcepath remote:album). However its becoming a never ending story because of throttling (I use tpslimit 5 and transfers 4).

If I would let the throttling cool down by aborting the copy, I imagine Rclone needs to check Google Photos if files already exist when I start the copy again. If so, would those checks count towards the Google Photos quota's?

Yes that will count against the listing quota which is more generous than the upload quota.

What throttling error are you getting?

I am curious how many files & MBs you are getting uploaded in a day before getting errors.
If you use "--progress" command it should give you a pretty decent read on how many files/much data goes through

In my experience you're probably best to stick with a tpslimit 2. Its slower yea but without all the retries which will count against your quota. I've uploaded in the past my entire photo archive (250G 42,000 pics/vids) to photos so I can use the app/web as a front end. Anything more than that and I see aggressive throttling. It took like 6 hours but not terrible considering.

Thank you for confirming. I since have found this is indeed how it works.

I am not sure what it all means, but this reflects my experience during the last couple of days;
As you can see I aborted the process a couple of times, so I can't say for sure, but I would say I have currently uploaded about 250G and an educated guess would be that I uploaded around 14500 files. Probably not very useful as it doesn't say much about the "throttle trigger". Seeing as this is the fourth try, I will try to keep an I on it this time.

Thanks for the tip! I will try a tpslimit of 2 next time I get thrott. Your stats are very impressive. My archive is about 484G with 70213 files. But I am already more than 3 days underway...

1 Like

Looking at the graphs, it seems you are calling the API less than once every second, so you could even reduce the tplimit to 1 without any change in overall speed (probably the bandwidth is the bottleneck, not the tplimit).
Since Google has a limit on 10000 API calls, even with tplimit set to 1, you will reach that in about 3hours (10000/60/60=2,8h).

Really? How did you managed this without hitting the 10000 limit Google has? Did you use your own ID?

It's been a while but I believe in the beginning I used the default and switched later on to my own. It's a fair point though about the 10k limit on your own ID. I'm sure the default has gotten busier over time. It could have been done coincidentally over the reset period as well.

I just ran an upload today and this is the throughput:
Transferred: 23.041Gi / 23.041 GiByte, 100%, 6.934 MiByte/s, ETA 0s
Transferred: 1744 / 1744, 100%
Elapsed time: 56m45.2s

So about an hour for 2000 files but these files are larger and many move videos. At that rate it wold have taken a bit over 35 hours. Probably closer to 40. Maybe my old age isn't recalling correctly either. :expressionless:

So these are the stats for the last session;

2021-05-14 rclone progress

Looking at the log it seems the elapsed time before the "Failed to copy: failed to create media item: Quota exceeded for quota metric 'All requests' and limit 'All requests per day' of service 'photoslibrary.googleapis.c0m' for consumer", was about 15hours.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.