Strange behavior with accentuated characters

yericus · August 26, 2019, 2:06pm

Yes I do !

/Volumes/Raid/mountpoint/movies/L'Enfer (1994)/L'Enfer (1994) - [BluRay-720p].mkv
/Volumes/Raid/mountpoint/movies/L'Enfer (1994)/L'Enfer (1994) - Bluray-720p.mkv

This is a preview orf a movie that should not be named with [BluRay-720p] in the first place !
Renamed file still has the apostrophe.

Animosity022 · August 26, 2019, 2:11pm

Can you share this screen from Radarr?

Is that mark a

or something else?

yericus · August 26, 2019, 2:22pm

Ohh I don't have the Clean Title anchor.
Well, I guess we are gonna go for a huge filename update for both radarr and sonarr and plex yet another time ahahah.

Animosity022 · August 26, 2019, 2:25pm

The Clean Title is what removes a bunch of other stuff from the names.

Linux works fine with spaces. I remove them cause I rather not have them as it is a personal preference.

I tend to think having less odd characters makes life easier.

yericus · August 26, 2019, 2:32pm

Well it would sure do in my case considering Plex inability to handle the way things are now, thanks a bunch. Renaming the files through the mount seems to take forever, but I will do some tests before launching a big change.

yericus · August 26, 2019, 2:45pm

Hmm got really strange behavior when trying to rename things with the rclone union mount, looks like renaming is not an option.

Also the renaming does not seem to want to get rid of the accents.
And a whole file got deleted on the remote without being copied oO.

Argh
Naming pattern: {Movie CleanTitle} ({Release Year}) - {Quality Full}
Volumes/Raid/mountpoint/movies/Nausicaä of the Valley of the Wind (1984)/Nausicaä of the Valley of the Wind (1984) - [BluRay-720p].mkv
/Volumes/Raid/mountpoint/movies/Nausicaä of the Valley of the Wind (1984)/Nausicaä of the Valley of the Wind (1984) - Bluray-720p.mkv

CleanTitle still leaves accentuated characters which is the one problem I have now with rclone on osx as ncw posted :

a = u"Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
b = u"Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv"
a == b
False

a
u"Weeds - S07E11 - Une Me\u0300re Que J'aimerais Baiser HDTV-720p.mkv"

b
u"Weeds - S07E11 - Une M\xe8re Que J'aimerais Baiser HDTV-720p.mkv"

Animosity022 · August 26, 2019, 2:53pm

What's your union mount config look like? I use mergerfs so both my mounts are read/write so I can rename without uploading again.

yericus · August 26, 2019, 2:57pm

The irony is because of the bug in rclone, I can't actually rename the file at all it seems :
2019/08/26 21:38:54 ERROR : tv/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv: Failed to copy: failed to open source object: open /Volumes/Raid/medias/tv/Weeds/Season 7/Weeds - S07E11 - Une Mère Que J'aimerais Baiser HDTV-720p.mkv: no such file or directory

I use a basic mount :
rclone mount --allow-other --vfs-cache-max-size=10G --drive-chunk-size=64M --buffer-size=96M --dir-cache-time 1h --vfs-cache-mode writes runion: /Volumes/Raid/mountpoint

I have a folder on grdrive called medias which has 2 folders movies and tv ; this remote (gdrive -> gcrypt) is setup as a rclone union remote (no mergefs on OSX from what I read) and mounted with the command above.

Animosity022 · August 26, 2019, 2:59pm

Could you roll back to a previous version to test it out and see if you can rename on that?

yericus · August 26, 2019, 3:23pm

I will try tomorrow some more but spent kinda the last 4 days trying to get it all working only to be confronted by this accent thing that hides some files somehow.

My wife is tired of me sitting in front of the screen on that just as much as i am for different reasons ah.

When the files are on the local folder of the mount, they can be played. so it's more probably something in the crypt aspect of the setup that is broken between encodings.

I reverted to 1.47 but the files still cant be played and every refresh takes forever. I am gonna upload a test file

yericus · August 26, 2019, 3:30pm

I deleted a movie with accent from Plex and rescanned the library ; same thing happens with the rclone 1.47 i initially used for uploading.

It would be nice to have everything working but at least the files are not quite lost and can still be downloaded via the file system that has no problem listing them so ... wait and see what happens with that for now I guess.

Thanks for trying to help in all this.

ps :
meh I re downloaded in sabnzbd the file from weed that was causing problems that had got deleted on rename try. it worked in local ... but once i uploaded it on a test folder on the remote and added it in the library, it actually also worked oO
So some file encoding somewhere got messed up.
Worst thing is I will re download every problematic file if I manage to find a filter in plex for showing me the unavailable ones.

A solution is here, quick ! I escape from the screen !

ppps : renaming the files directly in the filesystem works ; so there is always that solution whenever I find an accentuated file.

yericus · August 26, 2019, 4:48pm

Ok one last remark that is very puzzling to me but may help figure the bug if there is one,
I had uploaded a new version of an episode with an accent in a test library on plex.

I was puzzled to find out this newly upladed episode was actually recognized and could be played in plex after it had been encrypted and sent to my google drive.

I did not touch anything about this item. 20 minutes or so later the item became unplayable !
As if there had been a cache operation somehow after a while or a refresh of the cache or what not that had corrupted the filename for rclone.
It really looks like a different encoding is used at some point during the whole crypt / vfs caching that does not sit well with plex.

Animosity022 · August 26, 2019, 5:44pm

For the Sonarr/Radarr rename, I'm quite puzzled as instead of a rename, it seems to be issuing a link command:

2019/08/26 13:36:34 DEBUG : TV/Undercover.Boss.US/: Link: req=Link [ID=0x5040f6 Node=0x135a Uid=1000 Gid=1000 Pid=2383] node 5043 to "Undercover.Boss.US.S09E06.mp4", old=TV/Undercover.Boss.US/Undercover.Boss.(US).S09E06.mp4
2019/08/26 13:36:34 DEBUG : TV/Undercover.Boss.US/: >Link: new=<nil>, err=function not implemented

I was testing that out as well and that is why it downloads/reuploads it. Both Sonarr and Radarr seem to do the same thing. I'll let you know if I see a work around for that.

ncw · August 26, 2019, 7:25pm

I'd like to get to the bottom of this if we can. I don't have a mac or plex so it might be a little tricky...

I think that this PR might fix the problem as it seems to be a unicode normalization problem: https://github.com/rclone/rclone/pull/3148

Can you give this beta a go which is a build of that PR?

https://beta.rclone.org/branch/v1.49.0-031-ge5c6c14b-filename-encode-beta/ (uploaded in 15-30 mins)

yericus · August 27, 2019, 1:25am

Will do today and post results here.
New rclone still does not recognize old uploaded files.

Posted the results below for new test.

Quickly :

uploaded problematic accentuated file with rclone copy in test2 folder on my gdrive
created new library with it
file was recognized at first (same as previous rclone version) .
After 1 hour + (i have a dircachetime 1h in my mount if that helps), file is once more not available to plex anymore ; i captured the log moment when it happened and pasted it below.

Is there a different encoding used when refreshing the directory cache compared to the first scan somehow ?

yericus · August 27, 2019, 3:16am

Just launched a copy of a problematic episode :
rclone -q --bwlimit 8.5M copy /Volumes/Raid/accent/ rcrypt:test2

yericus · August 27, 2019, 3:21am

The new upload gets properly recognized and can be read by plex.
Yesterday, this stopped working after maybe half an hour or something (edit : maybe was one hour ; my mount specifies a dir cache time of 1 hour, will meet ths threshold soon).

2019/08/27 10:20:26 DEBUG : test2/: >ReadDirAll: item=1, err=
2019/08/27 10:20:26 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 10:20:26 DEBUG : test2/: >Lookup: node=test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv, err=
2019/08/27 10:20:26 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 10:20:26 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=

2019/08/27 10:20:29 DEBUG : test2/: ReadDirAll:
2019/08/27 10:20:29 DEBUG : test2/: >ReadDirAll: item=1, err=
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=OpenReadOnly
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=O_RDONLY
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fd=test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r), err=
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fh=&{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}, err=
2019/08/27 10:20:29 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Read: len=65536, offset=0
2019/08/27 10:20:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: ChunkedReader.openRange at 0 length 134217728

yericus · August 27, 2019, 4:57am

Ok so I was in the garden but here what happened since.
It's been more than 1 hour, i tried to replay the file and it worked, and now 2 minutes later the same behavior happened, the latest uploaded file is now marked as unavailable in the test2 library and can't be played !

This log extract below captures the moment where the file was being played ok to when it suddenly is not found anymore I think :

2019/08/27 11:46:20 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: ChunkedReader.Read at 348131328 length 1048576 chunkOffset 242229248 chunkSize 134217728
2019/08/27 11:46:20 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: ChunkedReader.Read at 349179904 length 1048576 chunkOffset 242229248 chunkSize 134217728
2019/08/27 11:46:20 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: ChunkedReader.Read at 350228480 length 1048576 chunkOffset 242229248 chunkSize 134217728
2019/08/27 11:46:20 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Flush:
2019/08/27 11:46:20 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Flush: err=
2019/08/27 11:46:20 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Release:
2019/08/27 11:46:20 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: ReadFileHandle.Release closing
2019/08/27 11:46:20 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Release: err=
2019/08/27 11:46:21 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 11:46:21 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=
2019/08/27 11:46:21 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=OpenReadOnly
2019/08/27 11:46:21 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=O_RDONLY
2019/08/27 11:46:21 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fd=test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r), err=
2019/08/27 11:46:21 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fh=&{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}, err=
2019/08/27 11:46:21 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Flush:
2019/08/27 11:46:21 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Flush: err=
2019/08/27 11:46:21 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Release:
2019/08/27 11:46:21 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Release: err=
2019/08/27 11:46:21 DEBUG : test2/: Attr:
2019/08/27 11:46:21 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:20 DEBUG : test2/: Attr:
2019/08/27 11:47:20 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:24 DEBUG : test2/: Attr:
2019/08/27 11:47:24 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:27 DEBUG : test2/: Attr:
2019/08/27 11:47:27 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:27 DEBUG : test2/: ReadDirAll:
2019/08/27 11:47:27 DEBUG : test2/: >ReadDirAll: item=2, err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.en.srt: Attr:
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.en.srt: >Attr: a=valid=1s ino=0 size=39400 mode=-rw-r--r--, err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.en.srt"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:27 DEBUG : test2/: Attr:
2019/08/27 11:47:27 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=OpenReadOnly
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=O_RDONLY
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fd=test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r), err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fh=&{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}, err=
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Flush:
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Flush: err=
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Release:
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Release: err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=OpenReadOnly
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Open: flags=O_RDONLY
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fd=test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r), err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Open: fh=&{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}, err=
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Flush:
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Flush: err=
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: Release:
2019/08/27 11:47:27 DEBUG : &{test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv (r)}: >Release: err=
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 11:47:27 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=
2019/08/27 11:47:27 DEBUG : test2/: Attr:
2019/08/27 11:47:27 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:27 DEBUG : test2/: Attr:
2019/08/27 11:47:27 DEBUG : test2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:27 DEBUG : test2/: Lookup: name="Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv"
2019/08/27 11:47:27 DEBUG : test2/: >Lookup: node=, err=no such file or directory
2019/08/27 11:47:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: Attr:
2019/08/27 11:47:29 DEBUG : test2/Weeds - S07E11 - Une Mère Que J'aimerais Baiser Bluray-720p.mkv: >Attr: a=valid=1s ino=0 size=1563542924 mode=-rw-r--r--, err=

yericus · August 27, 2019, 5:09am

To Animosity, If I can find a way to remove all accents, that would solve the issue for my collection but leave this behavior of accentuated filename in rclone unsolved.

Radarr/Sonarr don't see accentuated characters as problematic and don't replace them.

ncw · August 27, 2019, 9:16am

Is it possible this is a plex problem since it works with Radarr/Sonarr? Have you updated it recently? Could you try downgrading it?