Possible issue with accent (é) in folder path using cmount on macos 11.2

Floris · February 5, 2021, 12:59pm

What is the problem you are having with rclone?

When mounting an s3 mount from minio, some contents of folders are shown, but not accessible, except that they are.
This looks really strange written down, so allow me to explain.

For instance, I have a plain .txt file inside a folder that I mounted with rclone from minio S3. The folder, its contents and the .txt file itself can be seen from the Finder and the file size is shown, but it cannot be accessed or opened in any way from the Finder. When looking at the file (or any other file or directory inside that directory) from the terminal, this is what I see:

❯ open /Users/fvanlint/Drive/Google-Drive/Kippers/Privé/sha3sum.txt
The file /Users/fvanlint/Drive/Google-Drive/Kippers/Privé/sha3sum.txt does not exist.

However! If I use cat, the contents are shown. Go figure!

❯ cat /Users/fvanlint/Drive/Google-Drive/Kippers/Privé/sha3sum.txt
E1618DD7B9D08DF95AA96678AEB16ED6F1B1AD8BD8AD20838D90A2EA31CC30AD  channel_case.ex
2BB78022FD6121D0D52C4B6796732EC148CFB00D336087B5946A03C1CC207039  conn_case.ex
4EFF72D3B87D9EE5945067A108E6E0E2F22A17557F47FB3A40F66590A6E36917  data_case.ex

It seems to me that either minio-s3 or rclone does not like that é in the directory path. Directories with 's, or , in their name appear to work fine.

What is your rclone version (output from `rclone version`)

❯ rclone version
rclone v1.54.0
- os/arch: darwin/amd64
- go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS 11.2 20D64

Which cloud storage system are you using? (eg Google Drive)

Minio S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

I mount a minio S3 share using a launchd file in ~/Library/LaunchAgents/

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>KeepAlive</key>
	<true/>
	<key>Label</key>
	<string>rclone-drive-pve6</string>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/local/bin/rclone</string>
		<string>mount</string>
		<string>drive:</string>
		<string>/Users/fvanlint/Drive</string>
		<string>--config=/Users/fvanlint/.config/rclone/rclone.conf</string>
		<string>--vfs-cache-mode=writes</string>
		<string>--allow-other</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>StandardErrorPath</key>
	<string>/Users/fvanlint/Documents/logs/rclone-drive.log</string>
	<key>StandardOutPath</key>
	<string>/Users/fvanlint/Documents/logs/stdout-drive</string>
</dict>
</plist>

The rclone config contents with secrets removed.

[drive]
type = s3
provider = Minio
access_key_id = xxx
secret_access_key = xxx
endpoint = http://minio-pve6:9000

A log from the command with the `-vv` flag

I mounted the remote to a temporary directory from the command line using the following command:
❯ rclone mount drive: ~/drive-test --vfs-cache-mode=writes --allow-other -vv --log-file=mylogfile.txt
I only browsed to the problematic folder and tried to open sha3sum.txt, which took around 3 seconds.
The log file was huge, around 6 Mb, so I am pasting what seems to be some relevant parts here. A lot of the log was repeated mentions of DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0 2021/02/05 13:50:40 DEB and similar.

2021/02/05 13:50:40 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "mount" "drive:" "/Users/fvanlint/drive-test" "--vfs-cache-mode=writes" "--allow-other" "-vv" "--log-file=mylogfile.txt"]
2021/02/05 13:50:40 DEBUG : Creating backend with remote "drive:"
2021/02/05 13:50:40 DEBUG : Using config file from "/Users/fvanlint/.config/rclone/rclone.conf"
2021/02/05 13:50:40 INFO  : S3 root: poll-interval is not supported by this remote
2021/02/05 13:50:40 DEBUG : vfs cache: root is "/Users/fvanlint/Library/Caches/rclone/vfs/drive"
2021/02/05 13:50:40 DEBUG : vfs cache: metadata root is "/Users/fvanlint/Library/Caches/rclone/vfs/drive"
2021/02/05 13:50:40 DEBUG : Creating backend with remote "/Users/fvanlint/Library/Caches/rclone/vfs/drive"
2021/02/05 13:50:40 DEBUG : Mounting on "/Users/fvanlint/drive-test" ("drive")
2021/02/05 13:50:40 DEBUG : S3 root: Mounting with options: ["-o" "attr_timeout=1" "-o" "fsname=drive:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "allow_other" "-o" "volname=drive" "-o" "noappledouble"]
2021/02/05 13:50:40 DEBUG : S3 root: Init: 
2021/02/05 13:50:40 DEBUG : S3 root: >Init: 
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:40 DEBUG : /: Access: mask=00
2021/02/05 13:50:40 DEBUG : /: >Access: errc=0
2021/02/05 13:50:40 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/02/05 13:50:40 DEBUG : /: >Getattr: errc=0
2021/02/05 13:50:40 DEBUG : /: Access: mask=00
2021/02/05 13:50:40 DEBUG : /: >Access: errc=0
2021/02/05 13:50:40 DEBUG : /: Access: mask=00
2021/02/05 13:50:40 DEBUG : /: >Access: errc=0
2021/02/05 13:50:40 DEBUG : /: Statfs: 
2021/02/05 13:50:40 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: >Getattr: errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: Access: mask=00
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: >Access: errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: Access: mask=02
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: >Access: errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: Access: mask=04
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: >Access: errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: Access: mask=01
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: >Access: errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: >Getattr: errc=-2
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: Access: mask=00
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé/sha3sum.txt: >Access: errc=0
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: >Getattr: errc=-2
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: >Getattr: errc=-2
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/02/05 13:50:56 DEBUG : /Google-Drive/Kippers/Privé: >Getattr: errc=-2

Thanks in advance for any suggestions!

ncw · February 5, 2021, 4:42pm

There have been problems with this sort of thing with macOS and its use of unicode normalisation NFD rather than the more traditional NFC

The problem can be seen here

The first line Privé/sha3sum.txt: gives

>>> u"é"
u'\xe9'

Wheras the second line gives /Privé:

>>> u"é"
u'e\u0301'

So something - the finder? - is asking for denormalised accents, but when you do it through the cli the are passed as-is.

This is ringing a bell..

From this issue

You could try this option

-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC

This should probably be the default on macOS I think.

Do you get the same problem with rclone mount and v1.53?

Floris · February 5, 2021, 4:59pm

Hello @ncw thank you for also replying here, it is much appreciated.

Do you get the same problem with rclone mount and v1.53?

Yes, I have had this problem in the past. Most likely with some of the last versions of 1.53, but I am afraid I cannot say exactly what version. I used to use rclone through homebrew and the auto-update tap, but since cmount had to be manually added with homebrew I switched to the precompiled download link to which version was current at that point. However, I do not think that I noticed this before with the fuse based mount command.

You could try this option

Ok, I will try that tomorrow and report back.

From this issue

MacOS - rclone mount with accent in filename can't be open by Plex and mis Help and Support

Yes, that does sound quite similar. I did search around for similar problems but apparently I missed that topic, I hope you do not mind.

The problem can be seen here

That is really odd, that the terminal and the Finder would access the file in different ways!

ncw · February 5, 2021, 5:19pm

Let me know what happens. I might add that option as the default for macOS if a different iconv option isn't supplied...

Floris · February 9, 2021, 4:05pm

Hello @ncw, I tested the option -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC today in conjunction with your rclone build from here. This seems to work great, just as intended really. I can read and write inside the offending folder as hoped.

ncw · February 10, 2021, 5:01pm

That is great

Could you see if -o modules=iconv,from_code=UTF-8 works OK - in theory it should but I don't know!

Can you also try this build which adds

-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC

If it isn't already being used.

v1.55.0-beta.5166.242026906.fix-cmount-mac-options on branch fix-cmount-mac-options (uploaded in 15-30 mins)

If -o modules=iconv,from_code=UTF-8 works OK I'll update the build to use that simpler configuration.

Floris · February 11, 2021, 11:55am

Hello @ncw, thank you for the suggestion and the new build

-o modules=iconv,from_code=UTF-8 caused the same issue as without any options before, files inside /privé/ are not able to be opened or viewed using the Finder.

I tried your new build

❯ rclone version
rclone v1.55.0-beta.5166.242026906.fix-cmount-mac-options
- os/arch: darwin/amd64
- go version: go1.16rc1

With the following command:
rclone mount drive: ~/drive-test --vfs-cache-mode=writes --allow-other -vv --log-file=accent-logfile.txt
This time around, the files inside /privé/ are shown in the Finder as intended and they are able to be written from and to, moved around and so on.

So apparently, there is a difference on MacOS between -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC and -o modules=iconv,from_code=UTF-8 that stops the latter from working in the intended fashion.

ncw · February 11, 2021, 3:15pm

Great, thank you very much for testing and your super clear explanation of what happened.

I've merged the above fix to master now which means it will be in the latest beta in 15-30 mins and released in v1.55

Floris · February 11, 2021, 3:44pm

Hello @ncw thank you for being so amazingly responsive and helpful.
It is very much appreciated!

system · February 14, 2021, 3:44pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.