Another mac encoding issue :-)

Help and Support
1

What is the problem you are having with rclone?

MacOS creates files NFD style (decomposed Unicode) when done from Finder. I thought it was HFS+ related but true even on an rclone mount!

For example, when I create the "métro" directory from Finder, I can see the following entry (ls | od -c):

m   e    ́  **   t   r   o

When I type (echo métro | od -c) the same word, from any terminal, Ubuntu (bash) or MacOS (zsh) I can see the following sequence (known as NFC):

m   é  **   t   r   o

This is well explained here: https://stackoverflow.com/questions/40062427/globbing-accented-files-in-bash

Btw, plex on Ubuntu, Windows or MacOS seems to perfectly understand both ways to store this kind of characters..

My problem is: I can't work with directories from Terminal as any of my Perl tools can't find any of the directories stored as NFD !

So, when I type this:

echo métro | iconv -t utf-8-mac | od -c

I can see the result as NFD:

m   e    ́  **   t   r   o

Do you think I'd change the mount command which is the result of a previous topic?
(I use -o modules=iconv,from_code=UTF-8)

What is your rclone version (output from rclone version)

rclone v1.53.1

  • os/arch: darwin/amd64
  • go version: go1.15.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Catalina

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/local/bin/rclone cmount \
    --dir-cache-time 1000h \
    --log-level INFO \
    --log-file "$LOG_DIR/$CONFIG.log" \
    --poll-interval 15s \
    --umask 002 \
    --user-agent mine \
    --rc \
    --rc-addr :$PORT \
    --rc-no-auth \
    --cache-dir="$CACHE_DIR" \
    --vfs-cache-mode full \
    --vfs-cache-max-size 50G \
    --vfs-cache-max-age 336h \
    --transfers 8 \
    --volname "$VOL_NAME" \
    -o modules=iconv,from_code=UTF-8 \
    "${CONFIG}:/" "$MOUNT_POINT"

The rclone config contents with secrets removed.

[gd]
type = drive
scope = drive

[ex]
type = crypt
remote = gd:ex

A log from the command with the -vv flag

Paste  log here
2

I found the following article

and made the following test:

  1. Created the "métro" directory from Finder
  2. Try to get in from Terminal with cd métro => KO
  3. Used convmv -r -f utf8 -t utf8 --nfc --notest .
  4. Try to get in from Terminal => OK
  5. Is Finder lost: no ..

This is why I thought of changing the "-o modules" from my mound command...

% cd métro
cd: no such file or directory: métro
% convmv -r -f utf8 -t utf8 --nfc --notest .
mv "./métro"	"./métro"
Ready! I converted 1 files in 1 seconds.
% ls -l
drwxrwxr-x  1 me  staff  0 Oct 20 22:08 métro
% cd métro
1 Like
3

I don't think this is just an rclone problem is it? Since HFS+ uses NFD don't you get the same problems there?

That is worth a try definitely. you could try

-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC

Which is I think equivalent to what you have already or this which is equivalent to your convmv command

-o modules=iconv,from_code=UTF-8,to_code=UTF-8

Rclone doesn't change the UTF from NFC to NFD so it is possible the underlying files are NFD and those need to change - did you investigate this? try rclone lsf remote:dir | od -c

Running convmv is changing the underlying files from NFD to NFC I think.

4

Hello,
thanks for your answer.
I agree, I don't think it's an issue. I was requesting advices

Here is the result of the test:

Using a "métro" directory created with finder within a folder "test"

% rclone lsf ex:test | od -c
0000000    m   e    ́  **   t   r   o   /  \n                            
0000011
5

Interesting, so the underlying file is stored NFD. Looks like your plan with convmv is a good one :slight_smile:

1 Like
6

Thanks for your answers and advice

1 Like
7

Hello,
to share my experience, any mac user welcome :slight_smile:

I'm managing my Plex lib from a Mac, renaming, sorting and storing on Gdrive using either rclone copy/move/sync or Finder. (my library is encrypted)

Any plex server running on either Ubuntu or Windows is reading all the files perfectly.

That damn MacOS is not able to read its own files from Terminal or Plex !

So I need to use convmv to store all files the "NFC" way

And everything is Ok

Just sharing

Take care of you

1 Like
8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.