Fix google photos

Feature
21

I got good performance with --transfers 3. I just noticed that the google photos proxy is using a lot of ram ! 30% in my 16 gb system doing nothing.

Transferred: 19.841 GiB / 19.841 GiB, 100%, 4.173 MiB/s, ETA 0s
Transferred: 2522 / 2522, 100%
Elapsed time: 2h59m49.1s

So far all good. I just noticed that also trying to use checksum to compare the uploads between google photos and dropbox appears to do nothing, it only checks using size

EDIT: After downloading + 1200 files now I got rate limited again and can't download more. I did ~40 gb in a single day. If there is more documentation about the limits I would like to know. I tried to change my IP but no success

EDIT2: I'm still downloading as fast I can, sometimes i need to stop both rclone copy and the proxy and restart it a few times and then it works, it was glitching but no meaningful messages even with --debug on proxy so I thought i was being ratelimited but I actually think it's just a bug in the proxy itself that probably happens after too many files are downloaded? idk

22

Trying to speed up the process I tried to run another instance of gphotosdl on a different port with the argument --addr but it did not let me run it saying:

2024/09/27 07:20:15 DEBUG [852962:852962:0927/072015.415876:ERROR:chrome_main_delegate.cc(592)] Failed to create a ProcessSingleton for your profile directory. This means that running multiple instances would start multiple browser processes rather than opening a new window in the existing process. Aborting now to avoid profile corruption.
2024/09/27 07:20:16 ERROR Failed to make browser err="browser launch: [launcher] Failed to get the debug url: [852962:852962:0927/072015.415099:ERROR:process_singleton_posix.cc(335)] Failed to create /home/x/.config/gphotosdl/browser/SingletonLock: File exists (17)\n[852962:852962:0927/072015.415876:ERROR:chrome_main_delegate.cc(592)] Failed to create a ProcessSingleton for your profile directory. This means that running multiple instances would start multiple browser processes rather than opening a new window in the existing process. Aborting now to avoid profile corruption.\n"

So I tried to run it in a headless server, I copied the settings folder to my headless server, installed chrome deb and all the dependencies, discovered that gphotosdl will refuse to run as root, so made a user for it, copied the settings folder again, but when running it asks to login again which obviously I can't do in a headless server so I was expecting that copying the data from my desktop would allow me to bypass the login and run another instace

23

I think I finished moving all my data from google photos from dropbox. I did with

rclone copy -vvP --transfers 2 --gphotos-proxy "http://localhost:8282" gphotos:media/by-year/ mydropbox:Photos/ --fast-list

So far all I can is thank you @ncw for making this possible. I believe this will help a lot of people.

If there is any other way I can help let me know and also if there is any way to double check everything is ok (with photos is never enough right? ) thanks

24

Ha! Yes I noticed that too and have fixed in the latest release of gphotosdl

Thank you very much for all your testing and I'm glad you got your photos moved.

There are some unknowns around rate limiting.

Copying the entire browser directory should let you run it elsewhere. This needs to be more explicit in the docs.

Also we are missing a -profile flag which would let you run two at once.

Being able to download multiple at once would be a good idea.

Interesting that you are having problems with the gphotos API rate limiting you. You could try --check-first so it does all the API work at the start.

25

hey, this is awesome work! I can't seem to get it to work though (i'm also new to rclone)

  1. I've installed the latest beta:
rclone v1.69.0-beta.8363.4db09331c

  1. And set up google photos with the config function. I tested with "rclone lsd remote:album" and it works

  2. I downloaded the latest gphotosdl release and extracted it is some place. I then ran the command "gphotosdl -login" and i get an error that the browser is not found. I tried to delete the config folders that it created, but the error repeats itself. Here is the -debug output

mano@mano:~/Downloads/gphotosdl$ ./gphotosdl -debug
2024/10/07 11:28:43 DEBUG gphotosdl version 0.0.1, commit 8d9e2562e0745b9b91a4e124798259608042ace7, built at 2024-09-26T11:51:50Z
2024/10/07 11:28:43 DEBUG Configured config config_root=/home/mano/.config/gphotosdl browser_config=/home/mano/.config/gphotosdl/browser
2024/10/07 11:28:43 DEBUG Created download directory download_directory=/tmp/gphotosdl806395460
2024/10/07 11:28:43 ERROR Configuration failed err="browser not found"

Maybe I need to manually do some configuration? Or do i need to link gphotosdl with the rclone installation somehow? If so, I don't think I saw this step in the instructions..

thanks a lot in advance! Super excited if this works!

26

welcome to the forum,

can you please post the output of:

  • rclone version
  • rclone config redacted remote:
27 
rclone v1.69.0-beta.8363.4db09331c
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.11.0-061100-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.2
- go/linking: static
- go/tags: none


[gfotos]
type = google photos
token = XXX
client_id = XXX
client_secret = XXX
### Double check the config for sensitive info before posting publicly

I named it gfotos, not remote

28

what is the default browser on your system?

29

works fine on windows but there are a few few minor issues.

  1. i get the following error and need to whitelist.
gphotosdl.exe
2024/10/07 09:39:59 ERROR Failed to make browser err="browser launch: fork/exec C:\\Users\\user01\\AppData\\Local\\Temp\\leakless-amd64-adb80298fa6a3af7ced8b1c9b5f18007\\leakless.exe: Operation did not complete successfully because the file contains a virus or potentially unwanted software."
  1. rclone should run the default browser, which on my system, is firefox. instead rclone runs ms-edge
  2. the webpage that is opened is https://www.google.com/photos/about/
    maybe it should be https://accounts.google.com/
30

This only works with a chrome derived browser, so chrome, chromium, edge. If you haven't got one of those you will need to download one.

The library I used can do that for you but I didn't enable that.

If there /about page is opened it means you are not logged in.

31

so i installed chromium.

$ chromium --version
Chromium 129.0.6668.89 snap

tried running gphotosdl -login -debug

 ./gphotosdl -login -debug
2024/10/09 21:18:57 DEBUG gphotosdl version 0.0.1, commit 8d9e2562e0745b9b91a4e124798259608042ace7, built at 2024-09-26T11:51:50Z
2024/10/09 21:18:57 DEBUG Configured config config_root=/home/mano/.config/gphotosdl browser_config=/home/mano/.config/gphotosdl/browser
2024/10/09 21:18:57 DEBUG Created download directory download_directory=/tmp/gphotosdl1739549468
2024/10/09 21:18:57 DEBUG Found browser browser_path=/snap/bin/chromium
2024/10/09 21:18:57 DEBUG made browser preferences prefs="{\"download\":{\"default_directory\":\"/tmp/gphotos\"}}"
2024/10/09 21:18:57 INFO Log in to google with the browser that pops up, close it, then re-run this without the -login flag
2024/10/09 21:18:57 INFO Waiting for browser to be closed
2024/10/09 21:18:57 ERROR Browser run failed err="exit status 21"

then i tried without -login just for completeness

 ./gphotosdl -debug
2024/10/09 21:19:35 DEBUG gphotosdl version 0.0.1, commit 8d9e2562e0745b9b91a4e124798259608042ace7, built at 2024-09-26T11:51:50Z
2024/10/09 21:19:35 DEBUG Configured config config_root=/home/mano/.config/gphotosdl browser_config=/home/mano/.config/gphotosdl/browser
2024/10/09 21:19:35 DEBUG Created download directory download_directory=/tmp/gphotosdl322002728
2024/10/09 21:19:35 DEBUG Found browser browser_path=/snap/bin/chromium
2024/10/09 21:19:35 DEBUG made browser preferences prefs="{\"download\":{\"default_directory\":\"/tmp/gphotos\"}}"
2024/10/09 21:19:35 DEBUG [434765:434765:1009/211935.922768:ERROR:process_singleton_posix.cc(340)] Failed to create /home/mano/.config/gphotosdl/browser/SingletonLock: Permission denied (13)
2024/10/09 21:19:35 DEBUG [434765:434765:1009/211935.922864:ERROR:chrome_main_delegate.cc(597)] Failed to create a ProcessSingleton for your profile directory. This means that running multiple instances would start multiple browser processes rather than opening a new window in the existing process. Aborting now to avoid profile corruption.
2024/10/09 21:19:36 ERROR Failed to make browser err="browser launch: [launcher] Failed to get the debug url: [434765:434765:1009/211935.922768:ERROR:process_singleton_posix.cc(340)] Failed to create /home/mano/.config/gphotosdl/browser/SingletonLock: Permission denied (13)\n[434765:434765:1009/211935.922864:ERROR:chrome_main_delegate.cc(597)] Failed to create a ProcessSingleton for your profile directory. This means that running multiple instances would start multiple browser processes rather than opening a new window in the existing process. Aborting now to avoid profile corruption.\n"

I also made my own api client id, activated the google photos api and google drive api etc, basically all the recommendations.

tested with rclone lsd gfotos:album and it works fine

tried to do a

 rclone copy -vvP --transfers 2 --gphotos-proxy "http://localhost:8282" gfotos:media/by-year/ /home/mano/someFolder/ --fast-list

but i can't get a single file through, with the output something like

...
2024/10/09 21:39:45 DEBUG : pacer: low level retry 2/10 (error Get "http://localhost:8282/id/xxxxxxxxxx": dial tcp 127.0.0.1:8282: connect: connection refused)
2024/10/09 21:39:45 DEBUG : pacer: Rate limited, increasing sleep to 8.372009929s
2024/10/09 21:39:49 DEBUG : pacer: low level retry 3/10 (error Get "http://localhost:8282/id/xxxxxxx": dial tcp 127.0.0.1:8282: connect: connection refused)
2024/10/09 21:39:49 DEBUG : pacer: Rate limited, increasing sleep to 16.202656641s
2024/10/09 21:39:58 DEBUG : pacer: low level retry 3/10 (error Get "http://localhost:8282/id/xxxxxxxxx": dial tcp 127.0.0.1:8282: connect: connection refused)
2024/10/09 21:39:58 DEBUG : pacer: Rate limited, increasing sleep to 16.001196045s
...

I'm guessing this is because of the unsuccessful gphotosdl -login step, but I don't know how to fix it. I also tried to wipe the config folder after installing chromium. This doesn't help though :frowning:

32

ok so i tried everything, purged apt chrome, removed snap chromium, removed purged apt chromium, basically cleaned up my entire system of any traces of old shitty installations (as i was learning to use ubuntu)

I still couldnt get gphotosdl to use chromium. So i deleted it, and now installed google chrome. Everything works perfectly fine

33

You can use Google Takeout to export a copy of your Google Photos library. The docs say it does this in full resolution and you also get album metadata and favorites metadata. I've never done Takeout with Photos.