Rclone thinks "I" is the same as "İ" (NOTICE: config/data/metadata/People/İ: Duplicate directory found in source - ignoring)

Suspected Bug
1

First of all, I love Rclone. I use it all the time. But I noticed this very minor problem.

What is the problem you are having with rclone?

Rclone thinks "I" is the same as "İ". Without the -vv flag I get the following error:

NOTICE: config/data/metadata/People/İ: Duplicate directory found in source - ignoring

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: darwin 14.5 (64 bit)
- os/kernel: 23.5.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

n/a

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy '/Users/username/Docker/jellyfin/config/data/metadata/People' '/Users/username/Docker/jellyfin-backup/People' --transfers=8 -P -vv --log-file '/Users/username/Docker/scripts/rclone.log'

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[home-mac-mini]
type = sftp
host = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = none

[seedbox]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command that you were trying to run with the -vv flag

2

Not sure what it is yet but looks like it is macOS specific.

In Linux (Debian):

$ uname
Linux

$ ls
I  İ

$ rclone copy . ../test -vv
2024/06/10 19:49:52 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "." "../test" "-vv"]
2024/06/10 19:49:52 DEBUG : Creating backend with remote "."
2024/06/10 19:49:52 DEBUG : Using config file from "/home/test/.config/rclone/rclone.conf"
2024/06/10 19:49:52 DEBUG : fs cache: renaming cache item "." to be canonical "/home/test/Downloads/test1"
2024/06/10 19:49:52 DEBUG : Creating backend with remote "../test"
2024/06/10 19:49:52 DEBUG : fs cache: renaming cache item "../test" to be canonical "/home/test/Downloads/test"
2024/06/10 19:49:52 DEBUG : I: Directory modification time the same (differ by 0s, within tolerance 1ns)
2024/06/10 19:49:52 DEBUG : Added delayed dir = "I", newDst=I
2024/06/10 19:49:52 DEBUG : İ: Directory modification time the same (differ by 0s, within tolerance 1ns)
2024/06/10 19:49:52 DEBUG : Added delayed dir = "İ", newDst=İ
...

but in macOS:

$ uname
Darwin

$ ls
I  İ

$ rclone copy . ../test2/ -vv
2024/06/10 20:53:25 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "." "../test2/" "-vv"]
2024/06/10 20:53:25 DEBUG : Creating backend with remote "."
2024/06/10 20:53:25 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/06/10 20:53:25 DEBUG : local: detected overridden config - adding "{sb0-v}" suffix to name
2024/06/10 20:53:25 DEBUG : fs cache: renaming cache item "." to be canonical "local{sb0-v}:/Users/kptsky/Temp/test/test"
2024/06/10 20:53:25 DEBUG : Creating backend with remote "../test2/"
2024/06/10 20:53:25 DEBUG : local: detected overridden config - adding "{sb0-v}" suffix to name
2024/06/10 20:53:25 DEBUG : fs cache: renaming cache item "../test2/" to be canonical "local{sb0-v}:/Users/kptsky/Temp/test/test2"
2024/06/10 20:53:25 NOTICE: İ: Duplicate directory found in source - ignoring
2024/06/10 20:53:25 DEBUG : Added delayed dir = "I", newDst=I
...

rclone v1.65.2 - the same

3 
ver
Microsoft Windows [Version 10.0.22631.3593]

dir d:\files\test /b
I
İ

rclone copy d:\files\test d:\files\test2 -vv
DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "d:\\files\\test" "d:\\files\\test2" "-vv"]
DEBUG : Creating backend with remote "d:\\files\\test"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: renaming cache item "d:\\files\\test" to be canonical "//?/d:/files/test"
DEBUG : Creating backend with remote "d:\\files\\test2"
DEBUG : fs cache: renaming cache item "d:\\files\\test2" to be canonical "//?/d:/files/test2"
NOTICE: İ: Duplicate directory found in source - ignoring
1 Like
4

so maybe not just macOS but rather any case-insensitive file system...

For both I (U+0049) and İ (U+0130) lower case character variant is the same - i (U+0069). This probably trips rclone logic to detect duplicates. Like only case folded variants are compared for case-insensitive file systems.

I have tried to check in the source code what the problem might be but I can't see where it fails.

@ncw could you please have a look?

5

I think problem is here:

for case-insensitive filesystems characters are folded to lowercase before comparison which is too simplified way to deal with it.

I am not sure though how to fix it without breaking other things....

6

on windows, can use --local-case-sensitive

mkdir d:\files\CaseSensitiveFolder 

fsutil file setCaseSensitiveInfo   d:\files\CaseSensitiveFolder 
Case sensitive attribute on directory d:\files\CaseSensitiveFolder is enabled.

rclone copy d:\files\source d:\files\CaseSensitiveFolder -vv --local-case-sensitive 
DEBUG : rclone: Version "v1.66.0" starting with parameters ["c:\\data\\rclone\\rclone" "copy" "d:\\files\\source" "d:\\files\\CaseSensitiveFolder" "-vv" "--local-case-sensitive"]
DEBUG : Creating backend with remote "d:\\files\\source"
DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
DEBUG : local: detected overridden config - adding "{dTos-}" suffix to name
DEBUG : fs cache: renaming cache item "d:\\files\\source" to be canonical "local{dTos-}://?/d:/files/source"
DEBUG : Creating backend with remote "d:\\files\\CaseSensitiveFolder"
DEBUG : local: detected overridden config - adding "{dTos-}" suffix to name
DEBUG : fs cache: renaming cache item "d:\\files\\CaseSensitiveFolder" to be canonical "local{dTos-}://?/d:/files/CaseSensitiveFolder"
DEBUG : I: Making directory with metadata
INFO  : I: Made directory with metadata (mtime=2024-06-11T09:23:45.4171952-04:00)
DEBUG : Added delayed dir = "I", newDst=I
DEBUG : İ: Making directory with metadata
INFO  : İ: Made directory with metadata (mtime=2024-06-11T09:23:52.1877929-04:00)
DEBUG : Added delayed dir = "İ", newDst=İ
DEBUG : İ/I.txt: Need to transfer - File not found at Destination
DEBUG : I/I.txt: Need to transfer - File not found at Destination
DEBUG : I/İ.txt: Need to transfer - File not found at Destination
DEBUG : İ/İ.txt: Need to transfer - File not found at Destination
DEBUG : Local file system at //?/d:/files/CaseSensitiveFolder: Waiting for checks to finish
DEBUG : Local file system at //?/d:/files/CaseSensitiveFolder: Waiting for transfers to finish
DEBUG : İ/İ.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK
DEBUG : İ/I.txt: md5 = 7215ee9c7d9dc229d2921a40e899ec5f OK
DEBUG : I/İ.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK
DEBUG : I/I.txt: md5 = 7215ee9c7d9dc229d2921a40e899ec5f OK
DEBUG : İ/I.txt.xirikin2.partial: renamed to: İ/I.txt
INFO  : İ/I.txt: Copied (new)
DEBUG : İ/İ.txt.vojarad1.partial: renamed to: İ/İ.txt
INFO  : İ/İ.txt: Copied (new)
DEBUG : I/İ.txt.wiciroh6.partial: renamed to: I/İ.txt
INFO  : I/İ.txt: Copied (new)
DEBUG : I/I.txt.wuguhoz2.partial: renamed to: I/I.txt
INFO  : I/I.txt: Copied (new)
INFO  : İ: Set directory modification time (using SetModTime)
INFO  : I: Set directory modification time (using SetModTime)
INFO  : 
Transferred:   	          2 B / 2 B, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.0s


rclone tree d:\files\CaseSensitiveFolder 
/
├── I
│   ├── I.txt
│   └── İ.txt
└── İ
    ├── I.txt
    └── İ.txt
7

did another test, this time the dest folder was NOT case sensitive, and rclone was able to copy the files.

mkdir d:\files\not.case.sensitive.folder 

fsutil file queryCaseSensitiveInfo d:\files\not.case.sensitive.folder 
Case sensitive attribute on directory d:\files\not.case.sensitive.folder is disabled.

rclone copy d:\files\source d:\files\not.case.sensitive.folder -vv --local-case-sensitive 
DEBUG : rclone: Version "v1.66.0" starting with parameters ["c:\\data\\rclone\\rclone" "copy" "d:\\files\\source" "d:\\files\\not.case.sensitive.folder" "-vv" "--local-case-sensitive"]
DEBUG : Creating backend with remote "d:\\files\\source"
DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
DEBUG : local: detected overridden config - adding "{dTos-}" suffix to name
DEBUG : fs cache: renaming cache item "d:\\files\\source" to be canonical "local{dTos-}://?/d:/files/source"
DEBUG : Creating backend with remote "d:\\files\\not.case.sensitive.folder"
DEBUG : local: detected overridden config - adding "{dTos-}" suffix to name
DEBUG : fs cache: renaming cache item "d:\\files\\not.case.sensitive.folder" to be canonical "local{dTos-}://?/d:/files/not.case.sensitive.folder"
DEBUG : I: Making directory with metadata
INFO  : I: Made directory with metadata (mtime=2024-06-11T09:23:45.4171952-04:00)
DEBUG : Added delayed dir = "I", newDst=I
DEBUG : İ: Making directory with metadata
INFO  : İ: Made directory with metadata (mtime=2024-06-11T09:23:52.1877929-04:00)
DEBUG : Added delayed dir = "İ", newDst=İ
DEBUG : İ/I.txt: Need to transfer - File not found at Destination
DEBUG : İ/İ.txt: Need to transfer - File not found at Destination
DEBUG : I/I.txt: Need to transfer - File not found at Destination
DEBUG : I/İ.txt: Need to transfer - File not found at Destination
DEBUG : Local file system at //?/d:/files/not.case.sensitive.folder: Waiting for checks to finish
DEBUG : Local file system at //?/d:/files/not.case.sensitive.folder: Waiting for transfers to finish
DEBUG : I/İ.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK
DEBUG : İ/İ.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK
DEBUG : I/I.txt: md5 = 7215ee9c7d9dc229d2921a40e899ec5f OK
DEBUG : İ/I.txt: md5 = 7215ee9c7d9dc229d2921a40e899ec5f OK
DEBUG : I/İ.txt.kagehic7.partial: renamed to: I/İ.txt
INFO  : I/İ.txt: Copied (new)
DEBUG : I/I.txt.hikomig8.partial: renamed to: I/I.txt
INFO  : I/I.txt: Copied (new)
DEBUG : İ/İ.txt.dipewif0.partial: renamed to: İ/İ.txt
INFO  : İ/İ.txt: Copied (new)
DEBUG : İ/I.txt.hofegox8.partial: renamed to: İ/I.txt
INFO  : İ/I.txt: Copied (new)
INFO  : İ: Set directory modification time (using SetModTime)
INFO  : I: Set directory modification time (using SetModTime)
INFO  : 
Transferred:   	          2 B / 2 B, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.0s

rclone tree d:\files\not.case.sensitive.folder 
/
├── I
│   ├── I.txt
│   └── İ.txt
└── İ
    ├── I.txt
    └── İ.txt

2 directories, 4 files
8

I think you are right @kapitainsky . I had no idea that there were characters which differed in upper case but not in lowercase!

Hmm, I have no idea how to fix this either.

This should work on any OS as a workaround

9

--local-case-sensitive worked on my Mac! Thank you!

1 Like
10

Yes - @asdffdsa suggested workaround works perfectly in this case.

More general solution might be elusive. Interestingly APFS (macOS file system) case-insensitive flavour allows for İ and i dirs to co-exist:

$ ls -l
drwxr-xr-x@ 2 kptsky  staff    64B Jun 11 22:46 İ

$ mkdir i

$ ls -l
drwxr-xr-x@ 2 kptsky  staff    64B Jun 11 22:46 i
drwxr-xr-x@ 2 kptsky  staff    64B Jun 11 22:46 İ

despite İ correctly folding to i

$ x="İ";y=${x,,};echo $y
i

but not for I and i (as expected):

$ ls -l
drwxr-xr-x@ 2 kptsky  staff    64B Jun 11 22:49 I

$ mkdir i
mkdir: i: File exists

$ x="I";y=${x,,};echo $y
i

which would suggest that macOS itself does not take into account this folding oddity.

Wikipedia has all article about issues caused by this character in computing:

one more example that dealing with full Unicode set is hard:)

11

That wiki was an interesting read.

I guess a fix might be to compare both the upper case and lower case versions of the directory entry and see if they are both equal. There is probably problems with that idea too knowing unicode.

I have a feeling this is a giant can of worms, especially since Wikipedia mentioned doing different things when set to the Turkish locale.

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.