I have been syncing my folder of photos to Google Photos, up to now without issue. Unfortunately, even with verbose logging and debug log level, I am not seeing any errors to help diagnose the issue.
2021/10/26 15:18:36 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "sync" "/Photos/2021" "gphotos:album/2021" "--exclude" "Thumbs.db" "--exclude" "*.iso" "--exclude" "*.psd" "--retries" "1" "--low-level-retries" "1" "--log-level" "DEBUG"]
2021/10/26 15:18:36 DEBUG : Creating backend with remote "/Photos/2021"
2021/10/26 15:18:36 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2021/10/26 15:18:36 DEBUG : Creating backend with remote "gphotos:album/2021"
2021/10/26 15:18:36 DEBUG : Google Photos path "album/2021": List: dir=""
2021/10/26 15:18:36 DEBUG : Thumbs.db: Excluded
2021/10/26 15:19:36 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time: 1m0.0s
2021/10/26 15:20:36 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time: 2m0.0s
Config:
[gphotos]
type = google photos
token = {"access_token":"token","token_type":"Bearer","refresh_token":"token","expiry":"2021-10-26T16:09:07.3177541+10:00"}
I am running version 1.56.2 from within unRaid, but have also experienced the exact same issue running the same rclone command in Windows so it doesn't appear to be OS specific.
If I leave rclone running in the state above, the elapsed time will display every minute with 0/0 transferred. I haven't allowed it to run for more than 15 minutes. Something I've also noticed is that during this time, a lot of api calls are still being made. In the example log above, the google api page reported that 122 calls were made. I am using my own client/secret/token.
Correct, working for days and now rclone will not sync even using the exact same command (literally copying and pasting it from a notepad of regularly used commands). I had the same problem when I first started using rclone. I deleted everything from google photos manually, and rclone was happy again. It has now synced up almost 30,000 files with no issue, until now.
I tried adding the client_id and client_secret to the conf file but it didn't appear to make a difference. Also, when the sync was working fine these lines weren't in the conf file.
I added dump=headers. It appears a get request is being sent every couple of seconds. There is a http 200 OK response, but nothing additional occurs. Then another request is sent a couple of seconds later. I've removed the token information but the log (until I cancelled the process) is here:
hi, definitely not one time glitch. this is now the only activity I get out of rclone with google photos. don't really need to paste a new log as it's exactly the same as the original post and the pastebin header dump above. i even let the command run overnight last night. I ended up with a log following the same pattern of 0/0 byte/s transferred every minute. However, after an elapsed time of 2 hours, the process ended with "429 resource exhausted" errors, meaning that for 2 hours nothing got synced, but a day's worth of api calls were used anyway.
well, i see what you mean, seems the same api call over and over again.
i did notice a few things, might mean nothing
how many photos do you have in gphotos:album/2004?
seems that rclone is reading perhaps 50 at a time.
might try --fast-list to reduce api calls, tho not sure that works with gphotos.
one command seems to be running on windows, another command looks like linux???
what operating sytstems are you using?
i am going to guess that the linux os is unraid.
you did not post the output of rclone version
the command you posted does not match the command in the full debug log?
in the full debug log, the first api call is GET /v1/albums?pageSize=50 HTTP/1.1
but all the rest are GET /v1/albums?pageSize=50&pageToken=secretToken HTTP/1.1
in the debug log, the source path is "t:\\2004"
but rclone uses "//?/t:/Bob_GDrive/Photos/2004"
for the linux command you posted, has several excludes but the debug log has only one exclude.
hi again, sorry for the confusion! the main system that will do the syncing is rclone via unraid. however, for ease of creating logs etc I'm running rclone on Windows. both systems are resulting in the exact same issue so figured it would be ok. For simplicity I'll continue troubleshooting on Windows. There are 372 files in the 2004 folder. However, the same issue occurs if I try to sync a folder with 2 images.
I was obfuscating some folder names, hence not matching up with the log. I haven't hidden anything this time as I think even the tokens are useless without the clientid/secret etc.
Here is a command I've just run on Windows against a folder with 2 images (including fast-list as suggested):
If I try rclone ls gphotos:album I get no response.
If I try rclone ls gphotos:album -i --dump=headers I get the same stream of API calls per the pastebin log above in this post
I don't touch or use Google Photos at all as it's generally just very finicky with quotas/rates each day so it's not stable enough for me to ever use it.
I hear you and I have experienced issues with quotas etc. I'd be happy if that was the issue because I could just stop and start the process. But I've waited the full 24 hours until the quota reset and same thing. Plus, there are no error messages indicating any issues with quotas/rate.
So, being almost impossible to replicate, I've deleted everything from google photos including albums. Once photos is basically reset, rclone functions start working again. One strange thing is that getting a list of albums still returns a list, some that seem quite old, but maybe that is more to do with google creating auto albums, or previous sync from other software.
I also ran an rclone purge on the albums folder which did remove some of the albums. With was is as close to a completely reset google photos folder, I run rclone against a single image folder with a single image. Sync occurs fast and successfully. I run the same command again, and back into the loop of requests per above. The folder name is 1938, and the filename has no special characters. I'll keep resetting and testing other folders to see if I can find any consistency.
However, any opinion on whether these "orphaned" album folders could be causing an issue? Photos with EXIF dates saying 1938 shouldn't cause an issue I imagine? Also, anyone know if there is some way to purge google photos of absolutely everything... including these orphan albums?
Okay, so I've finally narrowed down a potential cause, but not a reason. If I upload any photos if exif date information prior to 1970, it will complete the upload but no further uploads are possible. If I delete the images prior to 1970, I can run rclone sync again. Any reason why photos with dates prior to 1970 would cause this issue? I'm thinking something to do with epoch starting in 1970? Strangely, though, the initial upload creates the files, and google reads the dates correctly. Same occurred when using the Windows google photos sync tool. For now I'm happy to just sync > 1970 photos, but would love to resolve the issue.
