The reason why is that Google Photos fetches every photo which you have uploaded. File ID, description, date modified (drive), date created, title, filename, albums, some exif data.
If I'm not wrong, album covers are preloaded even if you never see them because like 1000 albums in an extreme case.
Basically it loads all that stuff and keep them. So, I have 2.234.328 photos and it literally eats my entire storage. Also images which you looked at are being cached and not deleted after a specific time.
Overall. I suggest to clear App Data and let it reload the library. It will take a long time depending how many photos you got.
I'm not sure but I think Google photos even preload all images in a "thumbnail" size. I haven't looked for it.
Fun fact: Apple Music app does the same. Just for music.
If anyone wants to have a go, here is a prototype google photos backend. It is read only for the time being. I implemented a few different ways of reading the media.
It seems rclone can't set files into the folders. The structure is great and correct. As i wrote, folders are there but the images not. I tried to run ncdu (just to see if there even anywhere images displayed. Overkill kinda becuase files are being displayed multiple times but just to see if there something displayed). Well took some time after my images got scanned but overall it seems no images are displayed at all.
On the Google API Console i notice quite high latency. I think this will fix itself with the Error i mentioned. As i wrote, great structure!
I would help but i'm not a programmer. I understand most code and what they to etc but coding/helping is not possible for me.
Small thing: rclone already linking to https://rclone.org/googlephotos/ which isn't avaible yet because not existing (404) but i guess this comes when Google Photos is ready on rclone.
Yep. Files are being listed. I have done some stress testing. Ncdu the entire libary (Yes. I want to overkill). Well, let's say it took time because rclone sees files within multiple directory as another files (can be good or bad. Depends the need of the user). Overall i have hit the quata limit quite fast.
2019/06/18 15:58:21 ERROR : media/by-day/2002/2002-02-12: error listing: couldn't list files: Quota exceeded for quota metric 'photoslibrary.googleapis.com/read_requests' and limit 'ReadsPerMinutePerUser' of service 'photoslibrary.googleapis.com' for consumer 'project_number:12055929533'. (429 RESOURCE_EXHAUSTED)
2019/06/18 15:58:28 ERROR : media/by-day/2018/2018-01-29: error listing: couldn't list files: Quota exceeded for quota metric 'photoslibrary.googleapis.com/read_requests' and limit 'ReadsPerMinutePerUser' of service 'photoslibrary.googleapis.com' for consumer 'project_number:12055929533'. (429 RESOURCE_EXHAUSTED)
2019/06/18 15:58:29 ERROR : media/by-day/2017/2017-01-30: error listing: couldn't list files: Quota exceeded for quota metric 'photoslibrary.googleapis.com/read_requests' and limit 'ReadsPerMinutePerUser' of service 'photoslibrary.googleapis.com' for consumer 'project_number:12055929533'. (429 RESOURCE_EXHAUSTED)
I think the limits are ok because who takes a look at 75.000 pics a day?
But still the erros isn't correct. It showing up way to early before you reach your actual quota limit.
I think rclone could decrease the request quite a lot. In the folder structure, rclone request for each "by-date" the files over and over again in multiple folders. Could rclone just call all item (if not defined) and use the data to build up files in the folders?
Example
image.jpg has been requested and date/time has been fetched.
insert file into folder structure without doing another request. It's like a cache i would say. I hope you know what i mean.
Overall it is working how it should. You can read/view all images.
yes, the beta appears to be listing and downloading photos just fine! However video files, .mp4, are unplayable after downloading. No errors in the log. They are playable through the photos web interface and if they are downloaded from the web interface.
I think that is because the rclone app isn't approved yet. And I don't think I can ask for more quota until it is, so I'd better work out how to do that!
rclone could do that yet, though I was thinking that if you are doing a backup the by-month folders might be better and the by-day folders would be for if you were looking for something specific.
Overall it is fine. I mean the overall folders which are sorted by-year, by-month and by-date. Let's say you view images in an album. Images which has been loaded, there should be some kinda "cache". Rclone is not caching the image more the meta data of the files.
Basically if you saw images in an album which has been taken at like 25.03.2003, rclone could use the "cached" meta and set those files in the by-date (by-date, by-month, by-year) folders.
What rclone could think of:
Ohh. You saw a image of Paris.jpg. Ohh this images has been taken at 25.03.2003. I'm gonna help and set those images already into the folders which are date based. Now for those files I don't need to do a second request. I only need to request the rest.
Another thing. Would it be possible that the by-date based folders only create folder to the specific time if an image in that time exist. Example: If I wouldn't have an image of the year 2000 the folder 2000 shouldn't appear. Same for months or day.
Yes caching could be useful. I don't want to be caching 1,000,000 photos though so it probably needs to be a bit clever.
One problem with fetching everything is that you can only fetch 100 photo metadatas at a time which makes bit listings really slow!
That would be nice, however it would need an query for each directory in the parent to see if there was anything there. Perhaps that wouldn't slow it down too much.
It is. I'm considering doing my own sync to drive from my devices with rclone. and then possibly using this backend to potentially 'load' (sync) the pictures into photos so they are browsable in the app rather than using the app as the backup tool. (Unless these restrictive issues can be worked out of course).
No, all my files are uploaded in original quality and like I said if I download them through the Photos Web UI they are correct. Just appears to be an issue with the API... which is really disappointing and makes it unusable for me.
