macOS rclone mount - new FUSE-T released - old issues fixed

kapitainsky · July 1, 2023, 1:41pm

New FUSE-T has been released:

File access and modification times cannot be set separately as it seems to be an issue with the NFS client which always modifies both. Can be reproduced with 'touch -m' and 'touch -a' commands

Workaround with latest FUSE-T is to use -o noatime and/or -o nomtime

There is also new -o nfc (added in v1.0.21) option which improves UTF8 encoding issues when working in multiplatform environment by enforcing all content saved to mount to normalise to NFC (used on Linux and Windows).

New options can be used in rclone mount command:

rclone mount remote: mountpoint -o nfc -o noatime

There are still issues with UTF8 encoding when content is saved on macOS to remote via rclone copy/move/sync (as it is NFD normalised) but this should be alleviated by VFS normalisation which is aimed for rclone v1.64:

github.com/rclone/rclone

new feature: VFS auto normalization

opened 08:56AM - 21 Jun 23 UTC

kapitainsky

enhancement VFS / mount

Immediate goal of this enhancement is to fix macOS `rclone mount` which at the m…oment does not work properly regardless of mounting method used - macOSFUSE or FUSE-T. Optionally normalized VFS can bring other benefits as well e.g. some less common or future filesystems rclone users can work with and normalization constraints imposed. The background issue and possible solutions have been discussed on the forum: https://forum.rclone.org/t/possible-special-character-encoding-issue-on-macos/39048/30 At the moment many cloud storage destinations (and rclone virtual crypt remote) allow to store both NFC and NFD encoded files/directories names. Also rclone allows to upload/download both forms. This in general works trouble free on Linux and Windows but breaks rclone mount functionality in macOS - depending on mount options either only NFC or only NFD content is visible/accessible - this is [unfortunate](https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/) consequence of Apple design decision to be different than rest of the world and use exclusively NFD in their APIs. We won't change it. Experimentation and testing, including [attempts](https://github.com/macos-fuse-t/fuse-t/issues/16) to improve situation with FUSE-T (it has now - since v1.0.20 - `-o nfc` option which should help with NFC path) lead me to believe that last missing part of the puzzle is rclone VFS lack of normalization. I would like to propose to add optional normalization to VFS. New option: --vfs-normalize OFF/NFC/NFD - with OFF being default When set to NFC or NFD it would apply characters normalization to served content. Edge case to watch is situation where two names differ only by normalization and coexist in the same folder e.g.: ``` $ rclone lsf crypt:TEST CONFLICTéééFILE.txt CONFLICTéééFILE.txt ``` one file is NFC and other is NFD. Only solution I see for such case is use only one name and log ERROR.

dia3olik · July 29, 2023, 11:58am

do you suggest to wait for the 1.64 rclone release which should hopefully include VFS auto normalization?

kapitainsky · July 29, 2023, 12:09pm

Until this is implemented (hopefully in 1.64 but who knows) there is really no good solution.

My workaround at the moment is to use fuse-t:

rclone mount remote: mountpoint -o nfc -o noatime -o modules=iconv,from_code=UTF-8,to_code=UTF-8

this solves problem of new files uploaded to remote via mount (NFD or NFC they are normalised to NFC).

And if I have local files I want to sync to remote I make sure that they are all NFC normalised before.

$ brew install convmv
$ convmv -r -f utf8 -t utf8 --nfc --notest .

To make rclone macOS mount to work both in Finder and shell, only way now is to make sure that your mount does not have any files using NFD normalization and use -o nfc -o modules=iconv,from_code=UTF-8,to_code=UTF-8 with fuse-t. macFUSE is totally broken.

jwink3101 · July 30, 2023, 1:56pm

Thanks you for the info! I have resisted FUSE-T on my home mac but as soon as I hit reply, I am uninstalling the KEXT and installing FUSE-T

kapitainsky · July 30, 2023, 8:25pm

If you have any observations re fuse-t - good/bad does not matter - please share.

jwink3101 · July 31, 2023, 9:56am

I actually just switched back to the KEXT. I know the days are numbered but, at least for now, it was a (mediocre) solution to my problem.

I haven't had time to fully jump in and debug or gather a working minimal example but I keep running into problems that I think are more related to an idiosyncrasy in VLC than in FUSE/FUSE-T and/or rclone.

When you open a video in VLC, it opens in a mode that allows for some modification (sorry, I don't recall the flag). These are on crypt-wrapped local "remotes" stored on an external drive.

macFUSE:
- --vfs-cache-mode off: Reads the file perfectly fine. When you go to unmount, you get lots of errors in the logs but the files are untouched and they don't have an effect other than to slow down the unmount. (acceptable)
- --vfs-cache-mode writes: Writes the files back which is very slow and updates the ModTime (unacceptable and slow)
FUSE-T:
- --vfs-cache-mode off: VLC just can't read the file. It gets itself in a loop. (NOT usuable)
- --vfs-cache-mode writes: Reads the file and doesn't write the whole thing back from the cache but does update ModTime (not ideal but fast)

So, for now, I am sticking with macFUSE and no cache. I am fairly certain that the updated ModTime is correct as far as rclone behavior goes because VLC is opening to write so it isn't related to the fixed issue in FUSE-T nor is there a bug in rclone.

And I am know my days are numbered. If the KEXT becomes too unstable or Apple blocks them for good, I will have to make due (probably decide to stop caring about ModTimes but that will make me sad. And I don't like --size-only for later backups). But given that it is working, I am going to stay in my ways.

And of course, relevant XKCD

nielash · July 31, 2023, 10:21pm

@jwink3101 I'm curious what happens if you try the same VLC experiment with a normal local file (not mounted via rclone). Does the ModTime still get updated?

I'm interested because I also care about ModTimes and have seen some similar unexplained issues with them when mounting on macOS (and specifically upon unmounting, like you described). More recently I've been wondering if there might be an issue in the VFS layer itself, as I've also seen some odd behavior with rclone serve sftp, which shouldn't be using FUSE at all (but I'm not yet sure whether or not the issues are related).