Strange behavior with accentuated characters

Hi,

I installed the latest beta of rclone this morning and had a library refresh from Plex.

I encountered a weird behavior that did not happen before : all the files with an accentuated character were marked as unavailable after the refresh in Plex.

Yet when checking the file in the mount, everything is in order and the culprit files are present and can be played in vlc.

I use Osx
My setup in rclone is
Gdrive - crypt - rclone union - mount on local folder.

It seems only Plex has issues with these files and this behavior only began to happen when I switched to the latest beta version of rclone.

But the fact the files are properly mounted leaves me perplex as to why. And wonder if it could be related to the latest beta.

Did anybody have such behavior observed before ?

PROBLEM IDENTIFIED :

"After much reading I learn't there are multiple ways to construct the same unicode character, and so for efficient comparision there is a concept called normalisation ( http://www.unicode.org/reports/tr15/ ). After normalising with the same algorithm the representiation of any character will be consistent.

Sadly there are multiple normalisation algorithms and no concensus on which to use. Apple use NFD, where as most other operating systems use NFC. NFS does not specify a normalisation method, so OSX cannot confidently convert out of the box.

All I needed to do to fix this was to tell NFS that my NFS shares used NFC. Then all the files with the odd characters appeared and read fine.

I did this by adding this line to /etc/nfs.conf on the Mac.

nfs.client.mount.options = intr,locallocks,nfc"

Sadly, editing /etc/nfs.conf with the NFC tag does not solve the issue with rclone.
I tried to check what was up with OSXFuse on the subject and found :


" OSXFUSE includes support for converting file names from one encoding to another. This can be done by using module iconv. iconv can be enabled by applying the following mount-time option.
-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"

Not sure how to apply that with an rclone mount though.

mounting with the option results with :
Mac-mini:~ yericus$ rclone mount --allow-other --vfs-cache-max-size=10G --drive-chunk-size=64M --buffer-size=96M --dir-cache-time 1h --vfs-cache-mode writes -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC runion: /Volumes/Raid/mountpoint

2019/08/29 17:15:12 ERROR : -o/--option not supported with this FUSE backend

So maybe trying to integrate this OSXFuse options in the mac build of rclone would help ?

Can you share the rclone version and run the mount in debug and share the error log?

Hi there,

I am using the v.148.0-227-g077b4532- beta / darwin/amd64 / go1.12.9
Using OSX, mountpoint is on an APFS Raid drive (non case sensitive).

my mount command is :
rclone mount --allow-other --vfs-cache-max-size=10G --drive-chunk-size=64M --dir-cache-time 1h --vfs-cache-mode writes runion: /Volumes/Raid/mountpoint

I have a couple errands to run before running it all with debug logging enabled.

BUT I have just tried to see if the problem is indeed with rclone beta somewhere and it appears to be :
1 - I took an accentuated named file ' Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p' which appears unavailable with no audio or video stream in plex (but is still added when removed from plex if I scan the library).
2 - copied this file locally and created a new plex library with it. File got scanned by plex and added and can be played no issue.
3 - took the same file and copied it in my encrypted remote with rclone in a new folder. Added that folder to plex and same thing happens again ! File is indeed added to plex and appears in library but is marked unavailable as it is added with no recognisable audio/video stream.

This behavior happened to ALL accentuated filenames in my library with the latest beta (and sadly the release seems to be crashing for me). The rest of the files are fine.
Funnily, all files appear properly and can be played in vlc in my mounted rclone union folder !

Next step will be to try to isolate this in logs in the coming hours.

2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: ReadDirAll:
2019/08/26 16:31:55 DEBUG : /: Lookup: name=".metadata_never_index"
2019/08/26 16:31:55 DEBUG : /: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:55 DEBUG : /: Lookup: name=".metadata_never_index_unless_rootfs"
2019/08/26 16:31:55 DEBUG : /: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:55 DEBUG : /: Attr:
2019/08/26 16:31:55 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >ReadDirAll: item=3, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv: Attr:
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.en.srt"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.en.srt, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.en.srt: Attr:
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.en.srt: >Attr: a=valid=1s ino=0 size=39532 mode=-rw-r--r--, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.fr.srt"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.fr.srt, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.fr.srt: Attr:
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.fr.srt: >Attr: a=valid=1s ino=0 size=33354 mode=-rw-r--r--, err=
2019/08/26 16:31:55 DEBUG : accent/: Attr:
2019/08/26 16:31:55 DEBUG : accent/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/26 16:31:55 DEBUG : : Statfs:
2019/08/26 16:31:55 DEBUG : : >Statfs: stat={Blocks:2930077568 Bfree:2072094129 Bavail:2071794684 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2019/08/26 16:31:55 DEBUG : accent/: ReadDirAll:
2019/08/26 16:31:55 DEBUG : accent/: >ReadDirAll: item=2, err=
2019/08/26 16:31:55 DEBUG : accent/: Attr:
2019/08/26 16:31:55 DEBUG : accent/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/26 16:31:55 DEBUG : : Statfs:
2019/08/26 16:31:55 DEBUG : : >Statfs: stat={Blocks:2930077568 Bfree:2072094129 Bavail:2071794684 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2019/08/26 16:31:55 DEBUG : accent/: ReadDirAll:
2019/08/26 16:31:55 DEBUG : accent/: >ReadDirAll: item=2, err=
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:55 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:56 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:56 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:56 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:56 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:31:56 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:31:56 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory

i think this is the first thing that happens when I created a library in plex with just one folder with just an accentuated file on my encrypted gdrive mounted as a rclone union.

The file is on gdrive. the file is found and at the same time not found ?
Tell me if you need me to send something else as I am lost right now on all this.

2019/08/26 16:45:50 DEBUG : /: Attr:
2019/08/26 16:45:50 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/26 16:45:50 DEBUG : : Statfs:
2019/08/26 16:45:50 DEBUG : : >Statfs: stat={Blocks:2930077568 Bfree:2072094129 Bavail:2071794684 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2019/08/26 16:45:50 DEBUG : : Statfs:
2019/08/26 16:45:50 DEBUG : : >Statfs: stat={Blocks:2930077568 Bfree:2072094129 Bavail:2071794684 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Attr:
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
2019/08/26 16:45:52 DEBUG : accent/Weeds/Season 7/: >Lookup: node=, err=no such file or directory

this last log is when I launch an analyse fro Plex on the episode in question.

and for reference this is the alnalyse by plex on a non accentuated filename episode (working correctly) :

2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Attr:
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Attr: a=valid=1s ino=0 size=2315690761 mode=-rw-r--r--, err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Open: flags=OpenReadOnly
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Open: flags=O_RDONLY
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Open: fd=tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r), err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Open: fh=&{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}, err=
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: Flush:
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: >Flush: err=
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: Release:
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: >Release: err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Attr:
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Attr: a=valid=1s ino=0 size=2315690761 mode=-rw-r--r--, err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Open: flags=OpenReadOnly
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Open: flags=O_RDONLY
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Open: fd=tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r), err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Open: fh=&{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}, err=
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: Flush:
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: >Flush: err=
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: Release:
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: >Release: err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/: Attr:
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Attr:
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Attr: a=valid=1s ino=0 size=2315690761 mode=-rw-r--r--, err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Open: flags=OpenReadOnly
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: Open: flags=O_RDONLY
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Open: fd=tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r), err=
2019/08/26 16:47:16 DEBUG : tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv: >Open: fh=&{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}, err=
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: Flush:
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: >Flush: err=
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: Release:
2019/08/26 16:47:16 DEBUG : &{tv/Black Monday/Season 1/Black Monday - S01E03 - 339 WEBDL-1080p.mkv (r)}: >Release: err=

I reverted to the stable and the same problem with accents exist so it doesn't seem limited to the latest beta,
Oh god.

How can i force a total refresh of the vfs cache ?

Here is a successful lookup

And here is a failed lookup of apparently the same file

>>> a = u"Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
>>> b = u"Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
>>> a == b
False
>>> a
u"Weeds - S07E11 - Une Me\u0300re Que J'aimerais Baiser HDTV-720p.mkv"
>>> b
u"Weeds - S07E11 - Une M\xe8re Que J'aimerais Baiser HDTV-720p.mkv"
>>> 

So these actually aren't the same names - the second one has had unicode normalization applied to it.

Can you tell me which rclone version you were using before? I'd like to get some idea as to when this started happening?

We are working on a proper fix for this here: https://github.com/rclone/rclone/pull/3148 amongst other things.

I began uploading all my data to gdrive at the end of may (upload folder was created on 22 may), and it lasted for about a month.

I will plug my backup hard drive to check the version I used then.

FOUND IT

rclone v1.47.0

  • os/arch: darwin/amd64
  • go version: go1.12.4

That was what I used for uploading all my files.
I now run latest beta.

I just made the switch to using my online libraries so I probably did not notice the accentuated characters filenames not being able to being played.

Its not a big deal for now, but will his be corrected in a short timeframe like a month or 2 so I can access these files later ? Are they safe to appear later and I shouldnt worry about those right now ?

Thanks for your help !

Oh, by the way, I just re uploaded my test file with the latest beta and then tried to play it with it, and the problem still happened.

So even using last versions to upload seems to make it incompatible to play with itself ?

For what it is worth, I try to avoid any funky characters in titles just to make life simpler as different OS'es tend to do odd stuff to them at times.

Sonarr/Radarr both have options to remove special characters and use "Clean Titles" as they call them.

I use these to keep life easy.

Was thinking of doing that yes, not sure if I can rename everything directly in my mount, will do tests !

The renames all should be server side moves so that shouldn't be an issue I would think.

Just realized this setting was already on for my collection and it does not remove simple accentuated caracters (Nausicaä, Mère etc.).

You'd want to make the change and preview rename and it should ask you to do it. I'm not sure with the folder names though.

image

The actual episode names definitely would work though.

I mean the setting is already on in my radarr and sonarr and it leaves the normal accentuated characters.
RADARR or sonarr consider normal accentuation proper.

Ahhhh ! Well I hope this will get solved rclone side cause I am otherwise at a lost.

I think I will switch to radarr v3 and sonarr v2 or 3 soon.

You have both replace illegal characters and clean title enabled and you still have the ' character in your title?