macOS rclone mount - new FUSE-T released - old issues fixed

kapitainsky · July 1, 2023, 1:41pm

New FUSE-T has been released:

File access and modification times cannot be set separately as it seems to be an issue with the NFS client which always modifies both. Can be reproduced with 'touch -m' and 'touch -a' commands

Workaround with latest FUSE-T is to use -o noatime and/or -o nomtime

There is also new -o nfc (added in v1.0.21) option which improves UTF8 encoding issues when working in multiplatform environment by enforcing all content saved to mount to normalise to NFC (used on Linux and Windows).

New options can be used in rclone mount command:

rclone mount remote: mountpoint -o nfc -o noatime

There are still issues with UTF8 encoding when content is saved on macOS to remote via rclone copy/move/sync (as it is NFD normalised) but this should be alleviated by VFS normalisation which is aimed for rclone v1.64:

github.com/rclone/rclone

new feature: VFS auto normalization

opened 08:56AM - 21 Jun 23 UTC

kapitainsky

enhancement VFS / mount

Immediate goal of this enhancement is to fix macOS `rclone mount` which at the m…oment does not work properly regardless of mounting method used - macOSFUSE or FUSE-T. Optionally normalized VFS can bring other benefits as well e.g. some less common or future filesystems rclone users can work with and normalization constraints imposed. The background issue and possible solutions have been discussed on the forum: https://forum.rclone.org/t/possible-special-character-encoding-issue-on-macos/39048/30 At the moment many cloud storage destinations (and rclone virtual crypt remote) allow to store both NFC and NFD encoded files/directories names. Also rclone allows to upload/download both forms. This in general works trouble free on Linux and Windows but breaks rclone mount functionality in macOS - depending on mount options either only NFC or only NFD content is visible/accessible - this is [unfortunate](https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/) consequence of Apple design decision to be different than rest of the world and use exclusively NFD in their APIs. We won't change it. Experimentation and testing, including [attempts](https://github.com/macos-fuse-t/fuse-t/issues/16) to improve situation with FUSE-T (it has now - since v1.0.20 - `-o nfc` option which should help with NFC path) lead me to believe that last missing part of the puzzle is rclone VFS lack of normalization. I would like to propose to add optional normalization to VFS. New option: --vfs-normalize OFF/NFC/NFD - with OFF being default When set to NFC or NFD it would apply characters normalization to served content. Edge case to watch is situation where two names differ only by normalization and coexist in the same folder e.g.: ``` $ rclone lsf crypt:TEST CONFLICTéééFILE.txt CONFLICTéééFILE.txt ``` one file is NFC and other is NFD. Only solution I see for such case is use only one name and log ERROR.

dia3olik · July 29, 2023, 11:58am

do you suggest to wait for the 1.64 rclone release which should hopefully include VFS auto normalization?

kapitainsky · July 29, 2023, 12:09pm

Until this is implemented (hopefully in 1.64 but who knows) there is really no good solution.

My workaround at the moment is to use fuse-t:

rclone mount remote: mountpoint -o nfc -o noatime -o modules=iconv,from_code=UTF-8,to_code=UTF-8

this solves problem of new files uploaded to remote via mount (NFD or NFC they are normalised to NFC).

And if I have local files I want to sync to remote I make sure that they are all NFC normalised before.

$ brew install convmv
$ convmv -r -f utf8 -t utf8 --nfc --notest .

To make rclone macOS mount to work both in Finder and shell, only way now is to make sure that your mount does not have any files using NFD normalization and use -o nfc -o modules=iconv,from_code=UTF-8,to_code=UTF-8 with fuse-t. macFUSE is totally broken.

jwink3101 · July 30, 2023, 1:56pm

Thanks you for the info! I have resisted FUSE-T on my home mac but as soon as I hit reply, I am uninstalling the KEXT and installing FUSE-T

kapitainsky · July 30, 2023, 8:25pm

If you have any observations re fuse-t - good/bad does not matter - please share.

jwink3101 · July 31, 2023, 9:56am

I actually just switched back to the KEXT. I know the days are numbered but, at least for now, it was a (mediocre) solution to my problem.

I haven't had time to fully jump in and debug or gather a working minimal example but I keep running into problems that I think are more related to an idiosyncrasy in VLC than in FUSE/FUSE-T and/or rclone.

When you open a video in VLC, it opens in a mode that allows for some modification (sorry, I don't recall the flag). These are on crypt-wrapped local "remotes" stored on an external drive.

macFUSE:
- --vfs-cache-mode off: Reads the file perfectly fine. When you go to unmount, you get lots of errors in the logs but the files are untouched and they don't have an effect other than to slow down the unmount. (acceptable)
- --vfs-cache-mode writes: Writes the files back which is very slow and updates the ModTime (unacceptable and slow)
FUSE-T:
- --vfs-cache-mode off: VLC just can't read the file. It gets itself in a loop. (NOT usuable)
- --vfs-cache-mode writes: Reads the file and doesn't write the whole thing back from the cache but does update ModTime (not ideal but fast)

So, for now, I am sticking with macFUSE and no cache. I am fairly certain that the updated ModTime is correct as far as rclone behavior goes because VLC is opening to write so it isn't related to the fixed issue in FUSE-T nor is there a bug in rclone.

And I am know my days are numbered. If the KEXT becomes too unstable or Apple blocks them for good, I will have to make due (probably decide to stop caring about ModTimes but that will make me sad. And I don't like --size-only for later backups). But given that it is working, I am going to stay in my ways.

And of course, relevant XKCD

nielash · July 31, 2023, 10:21pm

@jwink3101 I'm curious what happens if you try the same VLC experiment with a normal local file (not mounted via rclone). Does the ModTime still get updated?

I'm interested because I also care about ModTimes and have seen some similar unexplained issues with them when mounting on macOS (and specifically upon unmounting, like you described). More recently I've been wondering if there might be an issue in the VFS layer itself, as I've also seen some odd behavior with rclone serve sftp, which shouldn't be using FUSE at all (but I'm not yet sure whether or not the issues are related).

kapitainsky · January 21, 2024, 11:15am

For anybody interested. The latest FUSE-T also supports smb style mount

It seems that macos SMB client is more stable than NFS and doesn't have the following bugs: #45, #17
Keep in mind this feature is highly experimental and might not be ready for production yet.

GwynethLlewelyn · January 24, 2024, 5:26pm

Hmm. More stable? Probably; SMB is quite stable overall, and has been so for several years.

But the issue is "speed". SMB is "ok" I guess; but NFS is blindingly fast (for many reasons, a few of which include the little regard they have towards complex authentication features, although v4 is supposed to play well with Kerberos and such).

kapitainsky · January 24, 2024, 5:30pm

As a frontend to serve remote cloud storage I think speed is not really critical - your bottleneck is somewhere else anyway. Similarly all security and authentication or their lack is not so important when only used for local machine.

We are talking here about local server not client. If you try serving nfs and/or smb from macOS you will discover soon that none is rock solid stable. Clearly their dev in macOS environment is not priority.

GwynethLlewelyn · January 27, 2024, 10:52am

I stand corrected! Aye, you're absolutely correct in that; in fact, I have never managed to share anything on my Mac to any of the other computers in the network — no matter what I tried (granted, I didn't try very hard).

My apologies for the confusion. Indeed, I was thinking about the client-side, not the server-side of sharing...

jwink3101 · January 30, 2024, 4:57am

Did it work for you? When I passed those args through rclone, it didn’t work

kapitainsky · January 30, 2024, 7:27am

Yes it works:

rclone mount crypt: ~/Temp/mount --vfs-cache-mode=full -vv -o backend=smb

and indeed creates smb share

$ df -h
Filesystem                                 Size    Used   Avail Capacity iused ifree 
...
//Guest:@fuse-t-0._smb._tcp.local/crypt   1.1Ti   736Gi   353Gi    68%    193M   92M   68%   /Users/kptsky/Temp/mount

jwink3101 · January 30, 2024, 2:37pm

Nice! Thanks.

I was doing something with --fuse-flag but in playing with -vv mode, I saw to make it passed properly, I had to do something like --fuse-flag '-o' --fuse-flag 'backend=smb'. I should have read the other. It is a bit confuising:

--fuse-flag stringArray     Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
-o, --option stringArray    Option for libfuse/WinFsp (repeat if required)

I will test it when I am at work today (I stull use the kext FUSE at home).

jwink3101 · January 30, 2024, 2:39pm

I am sorry I totally missed this until the thread was revived.

That is a good question and I don't know. I will have to experiment because I don't think so but I could be wrong.

nielash · January 31, 2024, 7:34pm

@kapitainsky interesting! I am curious what you think and how it compares to the nfs flavor of FUSE-T. Does it seem relatively stable? Any bugs so far? Haven't tried it myself yet, but I am interested as smb seems to be the most reliable protocol on mac these days (ironic, as it was developed for windows...)