Rclone dedupe not working with Google Photos?

[trevorb]

What is the problem you are having with rclone?

I use rclone to copy from Google Photos nightly to my NAS using the following command:

rclone copy gp-julia:media/by-year /photos_backup --fast-list --verbose --transfers 10

In the rclone output, there are a hundreds of Duplicate object found in source - ignoring messages. Interestingly enough, if I search Google Photos for a sample of the filenames listed, I only ever get one photo or video back. There doesn't appear to be duplicates. Therefore, I decided to try running the following:

rclone dedupe rename "remote:/media/by-year"

And get the following output up front:

2022/08/26 16:12:19 NOTICE: Google Photos path "media/by-year": Can't have duplicate names here. Perhaps you wanted --by-hash ? Continuing anyway.

Note: I have also tried "gp-julia:media" and `"gp-julia:Google Photos" with the same results

This eventually returns with the 100's of duplicates and states it ignores removing them, but then says All duplicates removed? Here's a sample of the output I get from dedupe.

2022/08/26 16:01:19 NOTICE: 2020/2017-11-04_22.21.12 {AOkVttRGrIt1_F7CvsEvmCyCVdIekzRB2dvE9lWrmpMQmfJaWGXVvwvLAJZqoaVA2cjMv5w1Myluto3_Jw0yEqeZBUvQuuK3pA}.jpg: Found 2 files with duplicate names
2022/08/26 16:01:19 NOTICE: 2020/2017-11-04_22.21.12 {AOkVttRGrIt1_F7CvsEvmCyCVdIekzRB2dvE9lWrmpMQmfJaWGXVvwvLAJZqoaVA2cjMv5w1Myluto3_Jw0yEqeZBUvQuuK3pA}.jpg: Ignoring as it appears 2 times in the listing and deleting would lead to data loss
2022/08/26 16:01:19 NOTICE: 2020/2017-11-04_22.21.12 {AOkVttRGrIt1_F7CvsEvmCyCVdIekzRB2dvE9lWrmpMQmfJaWGXVvwvLAJZqoaVA2cjMv5w1Myluto3_Jw0yEqeZBUvQuuK3pA}.jpg: Ignoring as it appears 2 times in the listing and deleting would lead to data loss
2022/08/26 16:01:19 NOTICE: 2020/2017-11-04_22.21.12 {AOkVttRGrIt1_F7CvsEvmCyCVdIekzRB2dvE9lWrmpMQmfJaWGXVvwvLAJZqoaVA2cjMv5w1Myluto3_Jw0yEqeZBUvQuuK3pA}.jpg: All duplicates removed
2022/08/26 16:01:19 NOTICE: 2022/188755010014 {AOkVttQtHvs_sbgmBKr8RyDOKaRonFXlX154HxZiin_OmPOp5zP11_lSGv6vUbrsXGxs2eimlpWS0owKH-lWRzyn5mLRyatPQg}.jpg: Found 2 files with duplicate names
2022/08/26 16:01:19 NOTICE: 2022/188755010014 {AOkVttQtHvs_sbgmBKr8RyDOKaRonFXlX154HxZiin_OmPOp5zP11_lSGv6vUbrsXGxs2eimlpWS0owKH-lWRzyn5mLRyatPQg}.jpg: Ignoring as it appears 2 times in the listing and deleting would lead to data loss
2022/08/26 16:01:19 NOTICE: 2022/188755010014 {AOkVttQtHvs_sbgmBKr8RyDOKaRonFXlX154HxZiin_OmPOp5zP11_lSGv6vUbrsXGxs2eimlpWS0owKH-lWRzyn5mLRyatPQg}.jpg: Ignoring as it appears 2 times in the listing and deleting would lead to data loss
2022/08/26 16:01:19 NOTICE: 2022/188755010014 {AOkVttQtHvs_sbgmBKr8RyDOKaRonFXlX154HxZiin_OmPOp5zP11_lSGv6vUbrsXGxs2eimlpWS0owKH-lWRzyn5mLRyatPQg}.jpg: All duplicates removed
2022/08/26 16:01:19 NOTICE: 2021/2021-07-18-13-00-0005 {AOkVttTXxAGOFjwAUKNXCrsPfUZe5CZimb7l3GmyF_Gzw19G-XzMyFuG_vo2rAUDsf_Cq25J6wnQw9q4BBzBxudM1QB_yzde8w}.jpg: Found 2 files with duplicate names
2022/08/26 16:01:19 NOTICE: 2021/2021-07-18-13-00-0005 {AOkVttTXxAGOFjwAUKNXCrsPfUZe5CZimb7l3GmyF_Gzw19G-XzMyFuG_vo2rAUDsf_Cq25J6wnQw9q4BBzBxudM1QB_yzde8w}.jpg: Ignoring as it appears 2 times in the listing and deleting would lead to data loss
2022/08/26 16:01:19 NOTICE: 2021/2021-07-18-13-00-0005 {AOkVttTXxAGOFjwAUKNXCrsPfUZe5CZimb7l3GmyF_Gzw19G-XzMyFuG_vo2rAUDsf_Cq25J6wnQw9q4BBzBxudM1QB_yzde8w}.jpg: Ignoring as it appears 2 times in the listing and deleting would lead to data loss
2022/08/26 16:01:19 NOTICE: 2021/2021-07-18-13-00-0005 {AOkVttTXxAGOFjwAUKNXCrsPfUZe5CZimb7l3GmyF_Gzw19G-XzMyFuG_vo2rAUDsf_Cq25J6wnQw9q4BBzBxudM1QB_yzde8w}.jpg: All duplicates removed

I then re-run the rclone copy command mentioned above and all duplicates still appear in the log output.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1
- os/version: unknown
- os/kernel: 4.4.180+ (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The rclone config contents with secrets removed.

Editing existing "gp-julia" remote with options:
- type: google photos
- read_only: false
- token: {***REDACTED***}
- client_id: ***REDACTED***
- client_secret: ***REDACTED***

A log from the command with the -vv flag

rclone_dedupe_output.log -- Run using rclone's token since I've exceeded my daily quota

[Animosity022]

You'd want to use by-hash:

2022/08/26 17:20:23 NOTICE: Google Photos path "media/by-year": Can't have duplicate names here. Perhaps you wanted --by-hash ? Continuing anyway.

You can check out:

Google Photos (rclone.org)

For some details on duplicates with Google Photos as well if you had not seen it.

[trevorb]

Thanks for the reply. I tried --by-hash before posting. That does not work either.

2022/08/28 13:56:38 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "dedupe" "rename" "gp-julia:media/by-year" "-vv" "--by-hash"]
2022/08/28 13:56:38 DEBUG : Creating backend with remote "gp-julia:media/by-year"
2022/08/28 13:56:38 DEBUG : Using config file from "/var/services/homes/helium/.config/rclone/rclone.conf"
2022/08/28 13:56:38 DEBUG : 2 go routines active
2022/08/28 13:56:38 Failed to dedupe: Google Photos path "media/by-year" has no hashes

(I tried --by-hash also using gp-julia:media and gp-julia:album both with the same results as above)

I've read over the link you provided and it mentions the following:

Rclone can remove files it uploaded from albums it created only.

I use rclone specifically to make a backup of my Google Photos - never upload. Considering everything on my Google Photos is not uploaded by rclone (photos are automatically uploaded when taking a photo) - it sounds like dedupe will not work in my case?

[Animosity022]

Seems like that might be a thing. I don't use Google Photos so I can't do any testing myself.

Dedupde was originally for Google Drive and remotes that allow duplicates and the Google Photos APIs is somewhat limited to my understanding. I don't see anything specific on dedupe documented though so might be something to add if that's the case to help out.

Not sure if someone else can chime in that uses it and validate if that's the case or perhaps we are missing something too.