My VFS SweetSpot - Updated 30-Aug-2018


#201

See this article for more details on some testing that was done:

https://www.chpc.utah.edu/documentation/software/rclone.php


#202

so, it has no role in playing files but only if uploading files from the mount?


#203

Hi @Animosity022,

I’ve been getting my server sorted out for a while now and found massive performance improvements switching to using VFS. Thanks for that.

However, one area I’m still struggling with is uploading files. Files will upload and initially appear in Radarr, then can end up disappearing days later. I’m not really sure why/how it’s happening. Any ideas?

I’m using a VFS mount with rclone crypt and no cache, using the same arguments you’re using.

Cheers


#204

I’m sorry but I don’t write directly to the mount with Sonarr/Radarr. I user mergerfs and upload at night.

What type of issue are you seeing? Is there something in the logs on those files?


#205

I got things working and mounted but I noticed with vfs it is a little slower, and in some cases will download the entire file to local disk before playing it. What am I doing wrong that doesn’t seem like an efficient method.


#206

What’s your mount command? What version are you using? What are some logs when the issue actually happens?


#207

ExecStart=/opt/rclone/rclone --config=/opt/rclone/rclone.conf mount --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 32M --fast-list --log-level INFO --log-file /home/plex/logs/rclone.log --umask 002 --vfs-cache-mode full kickcrypt: /mnt/plxcrypt
Here is the mount command.
Which logs do you want? I had to take the mount down and move back to plexdrive. I can redo it just want to know what logs you want before I mount it etc.
I am encypted so some logs are encrypted as I had stated previously.


#208

Yeah, you aren’t using my config so that’s your issue.

–vfs-cache-mode full

that downloads the file completely.


#209

Ok thank you let me turn that off can give it a go again. Thanks for the help again.


#210

change it to “writes” mode.
Please check the wiki to understand:
https://rclone.org/commands/rclone_mount/#vfs-cache-mode-writes


#211

Thank you now it is working and I can properly test it versus plexdrive. Appreciate you help and patience man.
I copied your command from github and removed the bind and running with that, for testing.


#212

I have been trying to figure out how to get the API results to show up in google. I created my own keys and enabled them in the config file but no data ever shows up in the API dashboard. Is there something else magical that I need to do to see this data? Yes I’m a data whore.


#213

Nope. If you are sure you are using the key, it shows up the console.


#214

I’m having very slow mediainfo times, ~ 25-60s each. If I run the command again, then it takes a fraction of a second. That said, my Plex movie start times are what I’d consider very reasonable – 5-7 seconds to load a movie, then 3-5 seconds to start after hitting play. Plex server status shows “Direct Play”.

i7-4785t, 16GB RAM, 1TB Evo 850 SSD, 100Mbit FIOS

Proxmox host is running the rclone services, bind-mounting the MergerFS pool into an unprivileged LXC container running Plex. (mediainfo takes a long time no matter what mount it’s run on). I’m using an rclone version built with cmount.

Relevant section of my rclone service:

ExecStart=/usr/bin/rclone mount gcrypt_username:/media /mnt/bind/gd/username/media \
   --allow-other \
   --buffer-size 1G \
   --dir-cache-time 72h \
   --drive-chunk-size 32M \
   --fast-list \
   --log-level INFO \
   --log-file /home/rclone/logs/rclone.log \
   --umask 002 \
   --vfs-read-chunk-size 128M \
   --vfs-read-chunk-size-limit off
ExecStop=/bin/fusermount -uz /mnt/bind/gd/username/media

Relevant section of my mount service:

[Mount]
What = /mnt/bind/data/local/username:/mnt/bind/gd/username/media
Where = /mnt/bind/gmedia/username
Type = fuse.mergerfs
Options = use_ino,hard_remove,auto_cache,sync_read,allow_other,category.action=all,category.create=ff

Unlikely to be relevant, but for completeness, the lxc mount command:

lxc.mount.entry: /mnt/bind/gmedia/username plex none bind,create=dir 0 0

Should I just disregard the mediainfo times if Plex is working within a few seconds?


#215

I’d probably want to know why if it was me. You can turn the logs up to debug and run the first mediainfo and post that.

Some may take a bit longer but if all are taking 25-60 seconds, that seems odd.


#216

The last few weeks I observe the same issue


#217

Without any specifics or logs or details, I can’t offer any advice. I’m not seeing that so it would be something with your setup.


#218

What are your thoughts on this presentation? I started using direct io and saw some decent improvement. For my setup, I’m on a VPS connected to the cloud and point directly to the fuse mount


#219

Apologies for the slow response. Busy week.

Had a bunch of files ‘delete’ yesterday. I don’t know when the files were actually deleted as Radarr only displays when it couldn’t find the file, so when it did a library scan. My debug logs for my VFS mount are set to INFO level and didn’t display anything over the weekend. Seems there’s at least 10 movies that have just disappeared at some point.

Here’s my mount options, if you see anything glaringly wrong?

rclone mount crypt: /home/files/crypt \
    --allow-other \
    --buffer-size=512M \
    --dir-cache-time=72h \
    --drive-chunk-size=128M \
    --fast-list \
    --log-level INFO \
    --log-file /home/logs/rclone.log \
    --vfs-cache-mode=writes \
    --vfs-read-chunk-size=128M \
    --vfs-read-chunk-size-limit=off \

I’m now thinking I’ll make the switch to mergerfs if it’s more reliable. One thing I’m not really clear on is how that interacts with the VFS cache. My understanding is that Radarr will copy the files into mergerfs, then at night you have it so that the VFS cache layer will upload it to your remote mount.

Is it possible for the file appear to stay in the same location to other programs like Plex and Radarr?

Cheers for your time and wisdom on this.


#220

I don’t think they disappear per se, but unionfs uses some .hidden files and such and I really didn’t like that/found odd things happened here/there with it which is why I switched to mergerfs instead.

I use that to copy locally first and my script uploads at night whatever is local. The path name never changes for Sonarr/Radarr/Plex so it all thinks it is the same path/location.

I do not see anything odd/wrong in your mount and never had an issue with files just disappearing. Usually, I’d see an error in the logs about Radarr having issues moving a file or something so I’d see a file ending with ~partial hanging around.