OSX Umlauts in Folder Names

MacC · April 11, 2024, 7:45am

Good day,

Problem:
while exporting, folder/directories with Umlauts like '_Büro'.
Exclusion directives are not respected.

Version
rclone v1.66.0

os/version: darwin 14.4.1 (64 bit)
os/kernel: 23.4.0 (x86_64)
os/type: darwin
os/arch: amd64
go/version: go1.22.1
go/linking: dynamic
go/tags: cmount

Task
Exporting from local SSD to backblaze2

Command

/Applications/rclone/rclone sync -v \
--config /........../rclone.conf \
--transfers 32 \
--checkers 32 \
--b2-hard-delete \
--skip-links \
--auto-confirm \
--fast-list \
--delete-excluded \
--delete-during \
--filter-from con-user-mac-filter.txt \
/Users/mac/ BB:........

and the filter-from:

- Desktop/_Büro/**
- \.Trash/
- $*
- /Bilder/**
- /Pictures/**
- /Documents/EasyCatalog/**
- /Calibre-Bibliothek/**
- /Desktop/_Jobs/**
- /FontExplorer?X/**
- /Musik/**
- Music/**

rclone config redacted

[BB]
type = b2
account = XXX
key = XXX
hard_delete = true

I tried the following

- Desktop/_Büro/** <- will be ignored

- Desktop/_B?ro/** <- will be ignored

- Desktop/_*/**    <-  is ok

A log from the command that you were trying to run with the `-vv` flag

2024/04/11 09:28:40 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/Applications/rclone/rclone" "sync" "-vv" "--config" "/Users/mac/.config/rclone/rclone.conf" "--transfers" "32" "--checkers" "32" "--b2-hard-delete" "--skip-links" "--auto-confirm" "--fast-list" "--delete-excluded" "--delete-during" "--filter-from" "con-user-mac-filter.txt" "/Users/mac/" "BB:con-user-mac"]
2024/04/11 09:28:40 DEBUG : Creating backend with remote "/Users/mac/"
2024/04/11 09:28:40 DEBUG : Using config file from "/Users/mac/.config/rclone/rclone.conf"
2024/04/11 09:28:40 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2024/04/11 09:28:40 DEBUG : fs cache: renaming cache item "/Users/mac/" to be canonical "local{HK82T}:/Users/mac"
2024/04/11 09:28:40 DEBUG : Creating backend with remote "BB:con-user-mac"
2024/04/11 09:28:40 DEBUG : BB: detected overridden config - adding "{jlU5h}" suffix to name
2024/04/11 09:28:40 DEBUG : fs cache: renaming cache item "BB:con-user-mac" to be canonical "BB{jlU5h}:con-user-mac"
2024/04/11 09:28:40 DEBUG : Waiting for deletions to finish
2024/04/11 09:28:40 DEBUG : Music: Excluded
2024/04/11 09:28:40 DEBUG : FontExplorer X: Excluded
2024/04/11 09:28:40 DEBUG : Pictures: Excluded
2024/04/11 09:28:40 DEBUG : Calibre-Bibliothek: Excluded
2024/04/11 09:28:40 DEBUG : .Trash: Excluded
2024/04/11 09:28:56 DEBUG : Favorite: Making directory
2024/04/11 09:28:56 DEBUG : Library: Making directory
...
2024/04/11 09:30:30 INFO  : Desktop/_Büro/testfiles.pdf: Copied (new)
..

A log from the command with - Desktop/_*/**

2024/04/11 09:38:36 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/Applications/rclone/rclone" "sync" "-vv" "--config" "/Users/mac/.config/rclone/rclone.conf" "--transfers" "32" "--checkers" "32" "--b2-hard-delete" "--skip-links" "--auto-confirm" "--fast-list" "--delete-excluded" "--delete-during" "--filter-from" "con-user-mac-filter.txt" "/Users/mac/" "BB:con-user-mac"]
2024/04/11 09:38:36 DEBUG : Creating backend with remote "/Users/mac/"
2024/04/11 09:38:36 DEBUG : Using config file from "/Users/mac/.config/rclone/rclone.conf"
2024/04/11 09:38:36 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2024/04/11 09:38:36 DEBUG : fs cache: renaming cache item "/Users/mac/" to be canonical "local{HK82T}:/Users/mac"
2024/04/11 09:38:36 DEBUG : Creating backend with remote "BB:con-user-mac"
2024/04/11 09:38:36 DEBUG : BB: detected overridden config - adding "{jlU5h}" suffix to name
2024/04/11 09:38:37 DEBUG : fs cache: renaming cache item "BB:con-user-mac" to be canonical "BB{jlU5h}:con-user-mac"
2024/04/11 09:38:37 DEBUG : Waiting for deletions to finish
2024/04/11 09:38:37 DEBUG : Music: Excluded
2024/04/11 09:38:37 DEBUG : FontExplorer X: Excluded
2024/04/11 09:38:37 DEBUG : Pictures: Excluded
2024/04/11 09:38:37 DEBUG : Calibre-Bibliothek: Excluded
...
2024/04/11 09:38:37 DEBUG : .Trash: Excluded
2024/04/11 09:38:52 DEBUG : Favorite: Making directory
...
2024/04/11 09:38:52 DEBUG : Desktop/_Büro: Excluded
2024/04/11 09:38:52 DEBUG : Desktop/_Jobs: Excluded
..

kapitainsky · April 11, 2024, 8:40am

It is not straightforward...

The same macOS version, the same rclone. All works:

$ rclone tree .
/
├── Desktop
│   ├── Test
│   ├── _Büro
│   └── _Test
├── Test
├── Test1
├── _Büro
├── _Test
└── test.sh

$ cat ~/Temp/filter
- _Büro/**

# sync without filter
$ rclone sync . drive:test2  --dry-run
2024/04/11 10:35:58 NOTICE: Desktop: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: Test: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: Test1: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: _Büro: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: _Test: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: test.sh: Skipped copy as --dry-run is set (size 597)
2024/04/11 10:35:58 NOTICE: Desktop/Test: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: Desktop/_Büro: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE: Desktop/_Test: Skipped make directory as --dry-run is set
2024/04/11 10:35:58 NOTICE:
Transferred:   	        597 B / 597 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.2s


# sync with filter
$ rclone sync . drive:test2 --filter-from ~/Temp/filter --dry-run
2024/04/11 10:40:33 NOTICE: Desktop: Skipped make directory as --dry-run is set
2024/04/11 10:40:33 NOTICE: Test: Skipped make directory as --dry-run is set
2024/04/11 10:40:33 NOTICE: Test1: Skipped make directory as --dry-run is set
2024/04/11 10:40:33 NOTICE: _Test: Skipped make directory as --dry-run is set
2024/04/11 10:40:33 NOTICE: Desktop/Test: Skipped make directory as --dry-run is set
2024/04/11 10:40:33 NOTICE: Desktop/_Test: Skipped make directory as --dry-run is set
2024/04/11 10:40:33 NOTICE: test.sh: Skipped copy as --dry-run is set (size 597)
2024/04/11 10:40:33 NOTICE:
Transferred:   	        597 B / 597 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.2s

kapitainsky · April 11, 2024, 8:44am

Does --include work for you?

e.g.

rclone lsf -R /Users/mac/ --include "Desktop/_Büro/**"

Also can you run your sync with --dump filters and post filters definition output:

rclone sync . drive:test2 --filter-from ~/Temp/filter --dry-run --dump filters
2024/04/11 10:46:23 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
- (^|/)_Büro/.*$
--- Directory filter rules ---
- (^|/)_Büro/.*$
--- end filters ---

MacC · April 11, 2024, 9:32am

Hmm, strange,

will check that.

atm I use the following, which works.

_B*/**

and the following works:

/Applications/rclone/rclone sync -v --config \
/Users/mac/.config/rclone/rclone.conf \
--transfers 32 \
--b2-hard-delete \
--skip-links \
--auto-confirm \
--fast-list \
--delete-during \
--exclude=.* \
/Users/mac/Desktop/_Büro BB:.......

[edited]

MacC · April 11, 2024, 10:15am

2024/04/11 12:05:06 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/Applications/rclone/rclone" "sync" "-vv" "--dry-run" "--config" "/Users/mac/.config/rclone/rclone.conf" "--transfers" "32" "--checkers" "32" "--b2-hard-delete" "--skip-links" "--auto-confirm" "--fast-list" "--delete-excluded" "--delete-during" "--exclude" "Desktop/_Büro/**" "/Users/mac/Desktop" "/Users/mac/tmp/test"]

/Applications/rclone/rclone sync -v \
--config /Users/mac/.config/rclone/rclone.conf \
--transfers 32 \
--checkers 32 \
--b2-hard-delete \
--skip-links \
--auto-confirm \
--fast-list \
--delete-excluded \
--delete-during \
--exclude  "Desktop/_Büro/**" \
/Users/mac/Desktop /Users/mac/tmp/test

gives me

2024/04/11 12:05:06 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/Applications/rclone/rclone" "sync" "-vv" "--dry-run" "--config" "/Users/mac/.config/rclone/rclone.conf" "--transfers" "32" "--checkers" "32" "--b2-hard-delete" "--skip-links" "--auto-confirm" "--fast-list" "--delete-excluded" "--delete-during" "--exclude" "Desktop/_Büro/**" "/Users/mac/Desktop" "/Users/mac/tmp/test"]
2024/04/11 12:05:06 DEBUG : Creating backend with remote "/Users/mac/Desktop"
2024/04/11 12:05:06 DEBUG : Using config file from "/Users/mac/.config/rclone/rclone.conf"
2024/04/11 12:05:06 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2024/04/11 12:05:06 DEBUG : fs cache: renaming cache item "/Users/mac/Desktop" to be canonical "local{HK82T}:/Users/mac/Desktop"
2024/04/11 12:05:06 DEBUG : Creating backend with remote "/Users/mac/tmp/test"
2024/04/11 12:05:06 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2024/04/11 12:05:06 DEBUG : fs cache: renaming cache item "/Users/mac/tmp/test" to be canonical "local{HK82T}:/Users/mac/tmp/test"
2024/04/11 12:05:06 DEBUG : Waiting for deletions to finish
2024/04/11 12:05:06 DEBUG : .DS_Store: Need to transfer - File not found at Destination
2024/04/11 12:05:06 DEBUG : .localized: Need to transfer - File not found at Destination
2024/04/11 12:05:06 NOTICE: _Büro: Skipped make directory as --dry-run is set
...
2024/04/11 12:05:06 DEBUG : _Büro/test.pdf: Need to transfer - File not found at Destination
...

Filters:

--- start filters ---
--- File filter rules ---
- (^|/)Desktop/_Büro/.*$
--- Directory filter rules ---
- (^|/)Desktop/_Büro/.*$
--- end filters ---

I use for editig BBedit - are there any Encoding Settings that influence??
I normally use UTF-8 UNIX LF.

kapitainsky · April 11, 2024, 10:55am

The same here. And all works. I do not understand yet what the issue might be as my attempt to reproduce it looks for me exactly like your setup.

MacC · April 11, 2024, 1:57pm

Good to know. Means I'm not that stupid

I will investigate that a little bit further.

MacC · April 11, 2024, 2:19pm

weird.

- Desktop/_B??ro/**

excludes:

2024/04/11 16:12:17 DEBUG : Desktop/_Büro: Excluded
2024/04/11 16:12:17 DEBUG : Desktop/_Baaro: Excluded

Last idea: I use an afps partition, non-encrypted.
An UTF voodoo?

I'm lost

kapitainsky · April 11, 2024, 2:53pm

Me too

It must be something related to Unicode but I can not figure out what - macOS? file system,? rclone?

kapitainsky · April 11, 2024, 3:00pm

As it is your /User folder and macOS 14.4.1 than it is encrypted - always:) Just encryption keys are not password protected. But it does not matter really IMO for this.

kapitainsky · April 11, 2024, 3:07pm

Maybe other macOS users reading it with similar problems. It would be worth to find out what is going on here.

One more test:

Could you run in terminal (below my results) in directory where you have _Büro:

$ ls -d --color=none _B?ro/ | hexdump -C
00000000  5f 42 c3 bc 72 6f 2f 0a                           |_B..ro/.|
00000008

I want to see if your u umlaut is using NFD or NFC normalisation.

MacC · April 11, 2024, 3:34pm

mac@MacBook-Pro Desktop % ls -d --color=none _B?ro/ | hexdump -C
00000000  5f 42 c3 bc 72 6f 2f 0a                           |_B..ro/.|
00000008

same same

kapitainsky · April 11, 2024, 3:37pm

what a pickle:) something must be different

kapitainsky · April 11, 2024, 3:39pm

and when in terminal you run:

$ rclone lsf . --include "_Büro/**"
_Büro/

? but do not copy paste my ü - type it on your keyboard or copy/paste from your filters file.

MacC · April 11, 2024, 4:33pm

rclone lsf . --include="/_Büro/**"

empty

rclone lsf . --include="/_B*/**"

results:
_Baaro/
_Büro/

I should name that 'office'

kapitainsky · April 11, 2024, 4:44pm

Yeah - it would make all easy:)

rclone should handle it better. macOS can be very hard nut to crack when it comes to Unicode normalisation. It is a long story which started immediately with APFS introduction. It is getting better but due to macOS design decisions most likely won't be ever 100% working.

In case of rclone it took years to sort out rclone mount but now it looks that maybe closer look how filtering works is needed too.

@nielash what is your opinion on this issue? Is it possible that rclone filtering does not apply Unicode normalisation correctly?

nielash · April 11, 2024, 6:34pm

I don't think there is actually any unicode normalization for filter matching, at the moment. There is an --ignore-case option but no --ignore-unicode-form option...maybe we should add one?

I am guessing you are right that the file is NFD and the filter is NFC, or vice versa.

Also worth noting that even --ignore-case is ignored for --files-from... that is probably a bug that should be fixed, or alternatively a limitation that should be disclosed in the docs.

kapitainsky · April 11, 2024, 6:40pm

Thank you! I will create github issue so it can be tracked and not forgotten.

kapitainsky · April 11, 2024, 6:53pm

Logged:

github.com/rclone/rclone

Filtering and Unicode normalisation

opened 06:53PM - 11 Apr 24 UTC

kapitainsky

Inspired by this rclone forum issue discussion: https://forum.rclone.org/t/os…x-umlauts-in-folder-names/45522/16 It seems that at the moment filtering does not take into account possible different characters normalisation - file names can use different one than filter itself. As a result especially for characters outside of basic English set filters can work not as intended. I am not 100% sure what is the best solution here. Possible options could be: 1. Normalise both objects' names and filters to common form 2. As suggested by @nielash add `--ignore-unicode-form` flag 3. ??? I do not think it is critical and urgent but definitely worth to record and maybe fix in due time.

MacC · April 11, 2024, 7:13pm

Thank you all for your efforts.
And I am happy not being the pebcak

OSX Umlauts in Folder Names

A log from the command that you were trying to run with the -vv flag

A log from the command with - Desktop/_*/**

A log from the command that you were trying to run with the `-vv` flag