Duplicate object found in source - ignoring (unique name)

Hi

ref earlier comment on this issue

Duplicate object found in source - ignoring

I have these 2 files in same directory on my Synolgy nas, files made from win7

As you can see the save time is different:

I assume that the letter 'å' is making this possible.

Very annoying, but for this case not important since content is the same, bu in general the 2 files could be different.

What is a best practices to rename these files and also keep them alive for backup (copy)?

Is there a windows command I can run, maybe add a prefix like (1) and (2) based on save time, and then avoid "Duplicate object found in source - ignoring"?

LoA

There are very few storage systems which can cope with duplicate names. Google Drive is the most prominent example.

Ideally if you were syncing from Google Drive to Google Drive, rclone wouldn't care about this, but it does unfortunately.

So for a sync to complete you need to deduplicate the names.

The standard rclone way of doing this is with rclone dedupe which can rename duplicate files.

This only works on Google Drive though.

hi,

that screen snippet is from the synology nas, which runs linux.
i thought that linux file system could not have two files with the same name in the same dir?

Linux is the OS.

The file systems are EXT2/3/4, BTRFS, XFS, etc.

Generally the mainline ones do not support duplicate files/directories.

I thought there were some outliers that do.

Since the OP has very little detail on what they are running, no template used, it's tough to guess.

I think it violates the POSIX spec as there would be no way to address the duplicate files.

Hi

thanks for answers!

The file mentioned has Norwegian leter 'å', I assume this is the source of the error.

Destination is jottacloud

is there a rclone answer to solve this?

M

Are those files exactly the same or does the å have different encodings in each?

Can you cut and paste the output of rclone lsf source: so we can see the duplicate files please?

Hi @Modesty

å is a good candidate, but you might also want to check for spaces (and similar).

Here is an example illustrating how to create issues like these and how to locate the hidden difference:

mkdir testfolder
cd testfolder
touch "test"
touch "test "                   # note the extra space in the end
ls                              # shows 2 files
ls test                         # shows 1 file
ls test?                        # shows 1 file - the other
ls test*                        # shows 2 files
rclone ls .                     # shows 2 files
rclone ls . --include="test"    # shows 1 file
rclone ls . --include="test?"   # shows 1 file - the other
rclone ls . --include="test*"   # shows 2 files

A similar approach can be used to pinpoint extra/different character(s) in the middle of a filename.

Thanks for answers. I'm not so skilled in unix so for me doing advanced commands in Putty is not so easy, will see if I have some more time this weekend.

I planned to upload 2 duplicates (rclone mark them as duplicates) in a ZIP file and send you to investigate.

/Torsteins Kitzbühler alm Gulasch.docx: Duplicate object found in source - ignoring

When I started to ZIP them I got this message
image

If you like to cook, ask for the resepy, its one of my best home mades.

It seems like the troubling files have been copied to your Synology folder twice with different settings for both code page and time (note the 1 hour time difference). This is most likely the root cause of your issue and may have happened anytime between 2018 and now (based on the date in the examples you have shared)

A manual cleanup (by renaming) is probably the safest if you aren’t fully on top of code pages and time settings.

How many duplicates do you have?

How do you transfer the files from Win7 to Synology? (SMB, rclone mount, …)

Has there been any changes in your setup since 2018?

Hi @Ole

I have lots of issues with the backup, I'm starting to get the grip of it now, and will try to solv it step by step.

For one of my share on my nas i got:

220 Duplicate object found in source - ignoring

this share is not the biggest...

2 main issues I see

  1. duplicates (haven counted, but many...)
  2. path is too long, like 170 characters, also many

Method for fixing duplicates:

  1. hoping for automatic
  2. probably need to do it manually

Method for fixing path: same as for duplicates

Do I have time to do it, no.... :expressionless:

How do you transfer the files from Win7 to Synology? (SMB, rclone mount, …)

All ways is the world... from phone, from win7-11, from smb, from robocopy, you name it i have used it.

Has there been any changes in your setup since 2018?

he he, yes, old nas failed in 2020, I manage to fix it, copied all files from old to new by robocopy. Old nas from Netgear file system to Synology file system

Main settings NAS i have newer tuned, i install it and leave it. I'm not a NAS expert, and i don't have a life long strategy for tuning NAS storage, im more into wine and food :wink:

OK, I have researched a bit and it may be possible to use Bulk Rename Utility: https://www.bulkrenameutility.co.uk/

The Bulk Rename Utility er a very powerfull and complex tool, so I strongly recommend you work on a copy of your data and then replace the original files once everything looks good, this has saved me from catastrophe a couple of times :sweat_smile:
That is: COPY data data_new; WORK on data_new; VERIFY verify verify; RENAME data data_old; RENAME data_new data.

Here is a quick introduction to an approach that may be modified/extended to handle your situation:

First enable Renaming Option/Prevent Duplicates.

Then enter one or more character conversions:

Then select the files to be rename and click preview and verify the new names and finally click rename - et voila:

The proposed solution is heavily inspired by this:
https://superuser.com/questions/327816/convert-file-names-into-utf

Another solution if you are good at linux and knows the codepage you are converting from:
https://askubuntu.com/questions/568565/how-to-rename-file-names-to-different-encoding

1 Like

I use this software all the time. It's great :slight_smile:

1 Like

the OP is using total commander, that has a built-in renamer.
tho i have moved onto double commander, runs on windows and linux.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.