Duplicate object found in destination - ignoring

Help and Support
1

Hi,

just added the --fast-list to my jobs to speed up (it's working) but with a large list of files I have some warnings like this:
2021/02/07 11:56:47 NOTICE: folder/1522356085_63000000000.jpg: Duplicate object found in destination - ignoring

what this message means?
the folder structure are identical source/destination but I have only few messagges like this for a total of 89011 files checked

2

That almost always happen when the help and support template isn't filled out.

Based on what you said, it means you have duplicate objects in wherever you are copying to.

3

what's this support template?
BTW the message is clear but I don't understand it.
In the folder mentioned there are 24 files. in both source and destination. but the warning is only in one file.
I tried to manually delete the file from destination end re-start the sync.
no warning but the file is not uploaded anymore..
so there's some problems during the check..
her's some screens:
source:

2021-02-07_123543
2021-02-07_123543958×865 139 KB

mounted destination:

2021-02-07_123600
2021-02-07_123600958×865 134 KB

command line:

2021-02-07_123721
2021-02-07_1237211815×71 48.8 KB

output log:
media_20210207_121841.log (9.2 KB)

the file that was manually deleted was 1522356085_63000000000.jpg

4

When you first open a help and support topic, you get a big message like this:

image
image1478×716 87.5 KB

It has a nice template that collects all the relevant information when you post so things are easier to answer and validate.

If you want to hit those items and post the required information, it really helps to answer your question.

5

What is the problem you are having with rclone?

read first post

What is your rclone version (output from rclone version)

rclone v1.53.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64bit professional

Which cloud storage system are you using? (eg Google Drive)

Google suite (drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

N:\rclone\rclone.exe sync "H:\00 - Programmi" Media:"00 - Programmi" --backup-dir=Media:"00 - Backup/00 - Programmi/%mydate%_%mytime%" --fast-list --delete-during --checksum --verbose --transfers %RCLONE_TRANSFERS% --checkers %RCLONE_CHECKERS% --bwlimit %RCLONE_BWLIMIT% --contimeout 60s --timeout 300s --retries 5 --low-level-retries 10 --rc --stats %RCLONE_STATS% --stats-file-name-length 150 --exclude "desktop.ini" --exclude "*.log" --exclude "*.db-wal" --exclude "programdata/cache/**" --copy-links --log-file "r:\rclone scripts\media_%mydate%_%mytime%.log"

The rclone config contents with secrets removed.

[gdrive]
type = drive
scope = drive
root_folder_id = *
token = *
client_id = *
client_secret = *

[Media]
type = crypt
remote = gdrive:01
filename_encryption = standard
directory_name_encryption = true
password = *

A log from the command with the -vv flag

log attached before
6

Still missing a debug log, but I'd guess based on what you've shared, you want to dedupe the destination as you have duplicates.

2021/02/07 12:29:46 NOTICE: Backup PC/Backup/C/Users/elbaz/AppData/Roaming/Emby-Server/programdata/metadata/library/74/7466c54735be433f8c424dfdceac3c15/folder.jpg: Duplicate object found in destination - ignoring

Google Drive allows for duplicates, which rclone doesn't like so you have to remove them from the source/destination.

7

Maybe I found the problem.
the 2 files showing as duplicated are in this situation..

  1. using rclone mount the file are showed and can be deleted on destination. deleted.
  2. launched the sync. no file uploaded, strange... the incriminated file not showing in rclone mount.
  3. using rclone browser, after point 1. and 2. the files are still on remote. deleted from here -> done
  4. launched the sync. the files are now synced
  5. re-launched the sync. no files upload and not duplicated warnings..

the problem so was between the rclone mounted drive (that are showing the "duplicated" file) and the effective remote that show another version of file.

8

If you have duplicated files then rclone sync will just pick the first one returned by the remote server and the other will be ignored, so things can seem a bit random.

rclone dedupe is the way to fix.

9

dedupe is a standalone command to launch after the sync?
or is an option to add?
thx BTW

10

It's listed here as a separate command.

11

thx, I'll give it a try!

12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.