Full Google Photos clone

Help and Support
1

What is the problem you are having with rclone?

My Google account was touching the storage limit so I created a new Google account just for storing Photos.

I used rclone to copy Photos from the old to the new account.

I'm pretty sure I saw all photos in the destination before deleting them from the origin. But now I see only 50 photos in the new account. And the old account is empty.

Run the command 'rclone version' and share the full output of the command.

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.6

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

These are all the commands I executed, some of them I redacted since they did not cause any change (like running ---help, etc.):

 1152  rclone config
 1164  rclone move "Google Photos":Takeout "Google Photos 2":Takeout
 1165  rclone moveto "Google Photos":Takeout "Google Photos 2":Takeout
 1166  rclone moveto "Google Photos": "Google Photos 2":
 1167  rclone moveto "Google Photos":* "Google Photos 2":*
 1168  rclone move "Google Photos":* "Google Photos 2":*
 1169  rclone move "Google Photos": "Google Photos 2":
 1173  rclone move "Google Photos": "Google Photos 2":
 1176  rclone sync "Google Photos" "Google Photos 2"
 **1179  rclone sync -P "Google Photos" "Google Photos 2"**
 1191  rclone sync -P "Google Photos" "Google Photos 2"
 1195  rclone delete "Google Photos":Takeout
 1197  rclone -P delete "Google Photos":
 1200  rclone -P delete "Google Photos"
 1201  rclone -P delete "Google Photos"|less
 1202  rclone delete "Google Photos":
 1203  rclone delete "Google Photos"
 1216  rclone delete "Google Photos":/
 1223  rclone delete "Google Photos":album
 1224  rclone delete "Google Photos":media
 1225  rclone delete "Google Photos":album
 1226  rclone delete "Google Photos":albums
 1228  rclone delete "Google Photos":album/*
 1229  rclone delete "Google Photos":album/

The command in bold is the one I'm pretty sure copied all files from one account to the other.

But I'm not sure what the delete commands did...

The rclone config contents with secrets removed.

[Google Photos]
type = google photos
token = {"access_token":---,"token_type":"Bearer","refresh_token":---,"expiry":"2022-07-22T11:19:18.593692877+01:00"}

[Google Photos 2]
type = google photos
token = {"access_token":---,"token_type":"Bearer","refresh_token":---,"expiry":"2022-07-19T00:47:15.84263213+01:00"}

In my local filesystem I have two directories: "Google Photos" and "Google Photos 2".

$ du -csh Google\ Photos*
540K	Google Photos
31G	Google Photos 2

I had manually downloaded all photos from the old account to the "Google Photos" directory as a takeout, but now this is empty. I don't remember having created "Google Photos 2".

Although I would not expect rclone to modify the local filesystem, I have the impression that it did, because I don't remember having manually removed any files from the local filesystem.

Any help would be appreciated so that I don't risk loosing all my photos :\

2

hello and welcome to the forum,

--- i assume that you have looked in the gdrive recycle bins?

--- the latest stable version of rclone is v1.59.0, really need to update before running rclone again.

for remotes, the name must has a colon character

in your case, the source dir and the dest dir are both local, relative to the current working dir.
rclone sync -P "Google Photos" "Google Photos 2"

fwiw, rclone debug output would show that.

echo %cd%
C:\data\rclone

rclone lsd remote -vv
DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "lsd" "remote" "-vv"]
DEBUG : Creating backend with remote "remote"
DEBUG : fs cache: renaming cache item "remote" to be canonical "C:/data/rclone/remote"
ERROR : : error listing: directory not found
Failed to lsd with 2 errors: last error was: directory not found

rclone lsd remote: -vv
DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "lsd" "remote:" "-vv"]
DEBUG : Creating backend with remote "remote:"
          -1 2022-07-25 15:35:02        -1 files
          -1 2021-03-31 15:23:01        -1 watch
3

Thanks for the help.

So I probably synced local folders GP and GP2, then deleted all photos in local folder GP1.

How do I now copy all photos from local GP2 to the remote GP2 account?

I can't make sense of the directory structure:

In local/source I have:

$ ls Google\ Photos\ 2/Takeout/
archive_browser.html 'Google Photos'

In remote/destination I have:

$ rclone lsd Google\ Photos\ 2:
-1 2022-07-30 01:47:37 -1 album
-1 2022-07-30 01:47:37 -1 feature
-1 2022-07-30 01:47:37 -1 media
-1 2022-07-30 01:47:37 -1 shared-album
-1 2022-07-30 01:47:37 -1 upload

4

Are they in albums? If not, just upload all the photos to /upload. If they are in albums then you'll need to upload them to /album/

5

Yes, I believe they are in albums, since they are in directories under: Google\ Photos\ 2/Takeout/Google\ Photos/
which I believe correspond to album names.

So I'm now running:

$ rclone copy Google\ Photos\ 2/Takeout/Google\ Photos/ Google\ Photos\ 2:/album/

and it's failing while trying to copy .json files, but that may be expected.

6

Yes you can't upload the json files, I suggest you --exclude "*.json"

7

Similar to rsync, I see.

There's something I still don't understand with the way GP are organized.

I downloaded a Takeout, which is what Google calls "retrieving all information" they have in the servers in the form of a download.

I did all this because my Google account was touching the 15 GB limit.

So I created this 2nd Google account just for storing photos, and removed the photos from my first account.

But when I was performing the rclone copy command, I got out of space in the new account. This leads me to conclude that the Takeout has much more info in some way, so that when I uploaded the albums with rclone the storage wasn't enough.

I still need to investigate this...

8

Its possible you had some photos stored at a lower resolution which weren't counting towards your limit. See Choose the upload size of your photos and videos - Computer - Google Photos Help and note in particular:

Important: Any photos or videos that you’ve backed up in High quality before 1 June 2021 won't count toward your Google Account storage.

Is that possible?

9

Yes, that's probably it... Didn't think of that.

Files are locally occupying 31 GB...

10

From my analysis it seems that instead of creating 3 Google accounts to hold the 31 GB of photos it compensates to pay for the Google's 100 GB with 20 EUR annually. Any other cloud ideas?

11

I opted for the 100 GB upgrade.

I uploaded all files but it seems some of them are repeatedly copied:

$ rclone copy Google\ Photos\ 2/Takeout/Google\ Photos/ Google\ Photos\ 2:/album/ --exclude "*.json" -P
Transferred: 81.912 MiB / 81.912 MiB, 100%, 232.454 KiB/s, ETA 0s
Checks: 3646 / 3646, 100%
Transferred: 116 / 116, 100%
Elapsed time: 3m35.9s

$ rclone copy Google\ Photos\ 2/Takeout/Google\ Photos/ Google\ Photos\ 2:/album/ --exclude "*.json" -P
Transferred: 81.912 MiB / 81.912 MiB, 100%, 18.763 KiB/s, ETA 0s
Checks: 3646 / 3646, 100%
Transferred: 116 / 116, 100%
Elapsed time: 4m11.8s

I was expecting the 2nd command to transfer nothing.

It seems rclone uploads to GP do not appear in Google's My Activity. Perhaps because it's through the API?

12

Just tried the previous command again now, hoping it was due to some transient issue, but got the same result:

$ rclone copy Google\ Photos\ 2/Takeout/Google\ Photos/ Google\ Photos\ 2:/album/ --exclude "*.json" -P
Transferred: 81.912 MiB / 81.912 MiB, 100%, 27.576 KiB/s, ETA 0s
Checks: 3646 / 3646, 100%
Transferred: 116 / 116, 100%
Elapsed time: 3m42.2s

Is this an issue that should be reported?

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.