Duplicate object found in source - ignoring - Google Photos

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I am seeing duplicates in a google photos mount location. However, I cannot run dedupe because it is google photos and this is listed as not possible?

Run the command 'rclone version' and share the full output of the command.

dulanic@mediaserver:~$ rclone sync gp-pers:media/all nc:gpbackup -P
2023-06-26 17:03:32 NOTICE: 2023-04-03 {ACaye7iGrO1t7u989lGCPki_ZN_QOu2SyWKIU-iopewihbHcMiNgFLH4IM-VoDuHeyv878MwC7nV1_4HY2B7kQ6Xb4QkfM0wUg}.jpg: Duplicate object found in source - ignoring
2023-06-26 17:03:32 NOTICE: 2023-04-03 {ACaye7iKYWEITLJv8gGf6MbTlxcY9ARtGEtrA9y-symfrL_UHC6WnQ-AYXBdDxt10DPzXWRjpZEqxxalUGdlgLfEgqCIK__A3g}.jpg: Duplicate object found in source - ignoring
2023-06-26 17:03:32 NOTICE: 2023-04-03 {ACaye7iM45W4Cx8XFXwQr9NK_j7JlszyIlTD3a3hlyVfkirJKa2Q-H3_2r1NT_CtWHjeeWamRMRlrnJyRVzVNIbx6Q_WRxJrBA}.jpg: Duplicate object found in source - ignoring
2023-06-26 17:03:32 NOTICE: 2023-04-03 {ACaye7iU7hh2aMTIu8BHUI1rdNNMbL0MK_gnQwK54oL0jC8zZ4LqGahBjwxXXvs1muPIKjvxJtvtr77T8LyaTACiiRXbVIvxLQ}.jpg: Duplicate object found in source - ignoring
2023-06-26 17:03:32 NOTICE: PXL_20220103_151559741 {ACaye7iI9nZDeB_k1w7nt5r4kq9dqVmp19FGlrCjyw3v6xxi_6WzL8yF_LYqSxCPscrwQ9fCQDWKp38FurjNkCwffDCGh11SXA}.jpg: Duplicate object found in source - ignoring
2023-06-26 17:03:32 NOTICE: PXL_20220626_204342138 {ACaye7iJWwDV3fWU9XX4kGdOfgr77SAzlVoc1cVn9nJrb855lAQSm3DBuzSvdAoTW0bI6dWqK05cioJ_Zyt_h6WeAuwe5b0GTQ}.jpg: Duplicate object found in source - ignoring

rclone v1.60.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.17.0-1020-oem (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

see above, rclone sync

The rclone config contents with secrets removed.

[gp-pers]
type = google photos
read_only = false
token = *reacted*
client_id = *reacted*apps.googleusercontent.com
client_secret = *reacted*

A log from the command with the -vv flag

There is thousands of files, not sure how to only get relevant data. If i copy just the single file, there is no error. If I use --include the log spams excluded files.
2

Unfortunately it looks like old problem without proper solution yet:

Might be related to limitations of Google Photos public API.

3

Yeah it may be, though it is very odd that I can sync just the idividual names and it works fine.

4

And when you run your original sync these files are not created on destination storage at all? I thought that file1 is synced then for strange reasons there is another file1 in source so it is skipped.

5

Ah I didnt catch that part. I went ahead and tested that and it does look like it still copies it.

6

if this is the case than it is not disaster - you end up with all files in your destination.

And based on linked post there are no really duplicates at all - if you run:

rclone lsf "gp-pers:media/all/2023-04-03 {ACaye7iGrO1t7u989lGCPki_ZN_QOu2SyWKIU-iopewihbHcMiNgFLH4IM-VoDuHeyv878MwC7nV1_4HY2B7kQ6Xb4QkfM0wUg}.jpg"

you see one file only?

And you can then check in Photos:

In the rclone output, there are a hundreds of Duplicate object found in source - ignoring messages. Interestingly enough, if I search Google Photos for a sample of the filenames listed, I only ever get one photo or video back. There doesn't appear to be duplicates.

Real duplicates are shown on gdrive:

rclone lsf drive:test.txt
test.txt
test.txt
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.