Possible special character encoding issue on macOS

Suspected Bug
21

Ah, I see.

OK, now I think I understand the rclone sync code a bit better. rclone sync matches overwrites with normalized unicode. It does however not transform the copied name. Thus I get 4mvk0n0mss2apim05t4a72hbc4

Then, rclone mount does not show the NFD version in the mount and I can copy the file into the folder again with Finder. The mount converts NFD->NFC and I get 5bqaptil47irc1lcspojffi2gk.

Yes, that makes a lot of sense @kapitainsky. Thank you.

In that case I do agree with you that if mount handled normalization instead of a mapping in FUSE, the two versions of the file should not be allowed to coexist.

22

I think it would be much better if rclone always saves normalised names (NFC) in remotes. It saves number of characters (NFD is potentially longer) and creates consistency.

1 Like
23

on macOS files end up with NFD as this is what macOS is using internally - but you can create NFD name on Linux too - it just have to be deliberate - when on macOS it always happens.

But this would only fix things moving forward - still might be worth. To make all working including past repos we have to think how to do this job in FUSE-T

24

Since rclone copy/sync normalizes before comparing source and destination, thus preventing both versions to exist at the same time, it seems like we don't want both versions to coexist.

Might it not make sense for mount to do the same as well, regardless of which one is presented in Finder / FUSE?

Or perhaps that's exactly what you're suggesting? Thanks

25

I think we have to assume that NFC and NFD names are all over remotes people use.

We know that it does not bother Linux and Windows - at least as I am aware.

It breaks macOS mount (and yes it is partially due to Apple decision how to handle normalisation - but we wont change it neither).

Now mount is provided by FUSE-T. It does not read names from remote. It uses what rclone serves.

remote -> rclone -> fuse-t -> mount

So could it work if rclone serves to FUSE-T only NFC? Effectively doing NFD->NFC when encountered.Thoughts?

26

Am I understanding the suggested mapping correctly?

Old:

REMOTE           ->             MOUNT

NFC -> RCLONE -> NFC -> FUSE -> NFD
NFD -> RCLONE -> NFD -> FUSE -> [nothing]


MOUNT            ->             REMOTE

NFC -> FUSE -> NFC -> RCLONE -> NFC
NFD -> FUSE -> NFC -> RCLONE -> NFC

New:

REMOTE           ->             MOUNT

NFC -> RCLONE -> NFC -> FUSE -> NFD
NFD -> RCLONE -> NFC -> FUSE -> NFD


MOUNT            ->             REMOTE

NFC -> FUSE -> NFC -> RCLONE -> NFC
NFD -> FUSE -> NFC -> RCLONE -> NFC

That makes a lot of sense to me. There is now symmetry on both directions of conversions.

Only issue I can see right now is if NFC and NFD both exist in the remote. You'll have a collision in rclone now instead of in FUSE.

27

Sorry, another issue I can see is if NFD existed in the remote and you're copying the normalized NFC over it. You'd need to ensure comparison for existing files was also normalized, exactly like rclone copy/sync already does.

28

How I see it... but it is still sketchy

Old:

REMOTE           ->             MOUNT

NFC -> RCLONE -> ??? -> FUSE -> NFC
NFD -> RCLONE -> ??? -> FUSE -> NFD


MOUNT            ->             REMOTE

NFC -> FUSE -> ??? -> RCLONE -> NFC
NFD -> FUSE -> ??? -> RCLONE -> NFD

New:

REMOTE           ->             MOUNT

NFC -> RCLONE -> NFC -> FUSE -> NFC
NFD -> RCLONE -> NFC -> FUSE -> NFC


MOUNT            ->             REMOTE

NFC -> FUSE -> ??? -> RCLONE -> NFC
NFD -> FUSE -> ??? -> RCLONE -> NFC

clearly you can see I am still missing some bits:) We will get there.

The issue has been swept under the carpet for very long. Time to get it right.

1 Like
29

Yes it is can of worms. Agree

1 Like
30

This is what others do:

https://docs.syncthing.net/advanced/folder-autonormalize.html

And example of Apple advice how to "fix" similar issues - in this case NFS/SMB

https://openradar.appspot.com/FB8957502

when he talks POSIX I think shell (at the end macOS is proper UNIX) and NSURL is GUI API so Finder - hence we have this weird behaviour that shell works sometimes but not Finder - and vice versa.

31

Do you want to write this up as an issue with suggestions on how to fix?

It wouldn't be too difficult to make VFS autonormalize I think

32

Will do - just need more experiments and tests before - so it will come at some stage soon.

33

Have updated fuse-t github ticket with some suggestions:

Hopefully it can be done as it would give us more options.

I am sort of sure that there is no perfect solution for this problem - but I hope that it can be made much less painful.

Extra mount options in fuse-t will give more ways to tackle it from rclone perspective.

34

@ncw - question - is it intentional that copy/move/sync do check normalised names but not copyto/movto?

E.g:

Using copyto I can create two files:

rclone lsf cc:TEST
CONFLICTéééFILE.txt
CONFLICTéééFILE.txt

one is NFC and one is NFD - and they happily coexist (works in crypt and in onedrive - does not in local macOS remote)

but if I try to copy this folder:

rclone copy cc:TEST cc:TEST2 --dry-run
2023/06/19 16:01:26 NOTICE: CONFLICTéééFILE.txt: Duplicate object found in source - ignoring
2023/06/19 16:01:26 NOTICE: CONFLICTéééFILE.txt: Skipped copy as --dry-run is set (size 0)

rclone does normalised names check and detects duplicates.

35

Hmm, its probably an oversight. However I don't think we'll fix it.

copyto is designed to copy the file with the minimal number of transactions and it is probably necessary to list the directory to fix normalization.

36

I have created new feature issue:

on fuse-t front since yesterday we have new option -o nfc - it nicely makes macOS mount compatible with NFC content in VFS.

37

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.