Rclone mount and UTF-8 problem on macOS

[WonCheol (WonCheol Kim)]

Rclone mount and UTF-8 problem on macOS

What is the problem you are having with rclone?

1. Some of the directories are not accessable or hidden on Finder, and look erroneous on Midnight Commander like this:

2. It looks like UTF characters are the problem, but some works OK. The difference is probably the OS on which I created the file/folder names: the problamatic ones seems to be created years ago on Windows.

3. When I tried to mount with option,

-o modules=iconv,from_code=UTF-8,to_code=UTF-8

All the file and directory at least appears, but problematic ones are not accessable. Almost simillar when I tried 'mount2' except the mount is done read-only.

4. sudo rclone ls works just fine.

5. Everything works just fine on Linux.

What is your rclone version (output from rclone version)

rclone v1.55.1

• os/type: darwin
• os/arch: amd64
• go/version: go1.16.3
• go/linking: dynamic
• go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS Big Sur Version 11.3

Which cloud storage system are you using? (eg Google Drive)

Google Drive, SFTP, Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

sudo rclone mount GDrive:/ /System/Volumes/Data/mnt/GDrive --daemon --allow-other

sudo rclone mount GDrive:/ /System/Volumes/Data/mnt/GDrive --daemon --allow-other -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC

sudo rclone mount GDrive:/ /System/Volumes/Data/mnt/GDrive --daemon --allow-other -o modules=iconv,from_code=UTF-8,to_code=UTF-8

sudo rclone mount2 GDrive:/ /System/Volumes/Data/mnt/GDrive --daemon --allow-other

sudo rclone ls GDrive:/(mypath)

The rclone config contents with secrets removed.

Paste config here

[TIMF]
type = drive
client_id = (my id)
client_secret = (my secret)
scope = drive
token = (mytoken)

A log from the command with the -vv flag

Paste  log here

(snipped)
2021/04/30 16:09:11 DEBUG : /: Statfs:
2021/04/30 16:09:11 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/04/30 16:09:13 DEBUG : /rec_video: >Readdir: items=251, errc=0
2021/04/30 16:09:13 DEBUG : /rec_video: Access: mask=01
2021/04/30 16:09:13 DEBUG : /rec_video: >Access: errc=0
(snipped)
2021/04/30 16:09:13 DEBUG : /rec_video/Hindemith Mathis der Maler (Serge Baudo, Svizzera Italiana) Lugano 1997: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/04/30 16:09:13 DEBUG : /rec_video/Hindemith Mathis der Maler (Serge Baudo, Svizzera Italiana) Lugano 1997: >Getattr: errc=0
2021/04/30 16:09:13 DEBUG : /rec_video/Hindemith Mathis der Maler (Serge Baudo, Svizzera Italiana) Lugano 1997: Access: mask=00
2021/04/30 16:09:13 DEBUG : /rec_video/Hindemith Mathis der Maler (Serge Baudo, Svizzera Italiana) Lugano 1997: >Access: errc=0
(snipped)
2021/04/30 16:09:13 DEBUG : /rec_video/Eötvös: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/04/30 16:09:13 DEBUG : /rec_video/Eötvös: >Getattr: errc=0
2021/04/30 16:09:13 DEBUG : /rec_video/Eötvös: Access: mask=00
2021/04/30 16:09:13 DEBUG : /rec_video/Eötvös: >Access: errc=0
(snipped)
2021/04/30 16:09:13 DEBUG : /rec_video: Access: mask=05
2021/04/30 16:09:13 DEBUG : /rec_video: >Access: errc=0
2021/04/30 16:09:13 DEBUG : /rec_video: Releasedir: fh=0x1
2021/04/30 16:09:13 DEBUG : /rec_video: >Releasedir: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=00
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/04/30 16:09:16 DEBUG : /: >Getattr: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=00
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=02
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=04
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=01
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=04
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=00
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Access: mask=00
2021/04/30 16:09:16 DEBUG : /: >Access: errc=0
2021/04/30 16:09:16 DEBUG : /: Statfs:
2021/04/30 16:09:16 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
(snipped)

[ncw (Nick Craig-Wood)]

On macOS rclone uses -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC by default which is normally what is required

@WonCheol

It looks like UTF characters are the problem, but some works OK. The difference is probably the OS on which I created the file/folder names: the problamatic ones seems to be created years ago on Windows.

Do you know what file system encoding they use? Maybe you need a different from_code= ? Note that the to_code=UTF-8-MAC is almost certainly what you want.

[WonCheol (WonCheol Kim)]

Thank you for your reply.

I've tried:
-o modules=iconv,from_code=EUC-KR,to_code=UTF-8-MAC

and also

-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC -o modules=iconv,from_code=EUC-KR,to_code=UTF-8-MAC

Now I can see some more directories but not all of them. And some European characters now look like broken Korean characters.

[ncw (Nick Craig-Wood)]

Where are these files coming from? Are they on a mac disk or are they being served via samba or something like that from a linux server?

Maybe you need to fix the source first to make it UTF-8 - the conv tool is helpful here.

[WonCheol (WonCheol Kim)]

Most of the files are now remote-only (Google Drive etc) and some are on NAS served via samba or SFTP to macOS, Linux and Windows.

And I've just noticed that macOS is rather a real source of this problem: if I convert a file name to UTF-8 NFC, I can not see it when I mount the network volume with another protocol than rclone!

My main computer is Mac and I daily create UTF-8 NFD files. So my conclusion for now is that I should limit my use of rclone on Mac, but I can still use rclone fully on Linux. Thank you Nick.

[ncw (Nick Craig-Wood)]

In theory rclone should be doing the right thing, provided that the original file names were in UTF-8.

That is the part I'm not sure about - do you think all the original file names are in UTF-8 or might there be other encodings in use? Linux is very sloppy with file name encodings (they are just strings of bytes) whereas Windows and macOS require file names to be UTF-8.

[WonCheol (WonCheol Kim)]

What I've tried earlier was to copy a file name from a network volume mounted with 'from_code=UTF-8,to_code=UTF-8' and rename the file from the volume mounted with default encoding option.

This time, I've tried to convert the file names in linux using convmv, and the problems are getting solved!

One remaining issue is that some (actually many) file names are longer than 256 bytes and this still seems to make troubles on the volume mounted with rclone for macOS.

[ncw (Nick Craig-Wood)]

@WonCheol

This time, I've tried to convert the file names in linux using convmv, and the problems are getting solved!

convmv is great. I've spent a lot of time cleaning up old filenames on Linux boxes converting them from strange encodings (mostly latin1) to UTF-8.

@WonCheol

One remaining issue is that some (actually many) file names are longer than 256 bytes and this still seems to make troubles on the volume mounted with rclone for macOS.

Hmm, rclone and fuse don't have a character limit. I wonder if it is macOS... a bit of reading seems to indicate the limit for macfuse is 255 bytes (not characters).