Help with migration to VFS cache from cache backend

Hello all,

i would kindly appreciate help with 2 issues i have. I m trying to migrate / test to VFS cache from old cache backed. I have 2 issues, first is that i cant mount same gdrive that was working on old cache backend without cache section, uts working if cache section is in config. Second one is that even on old cache backend and new VFS cache i have weird issue that stream is started all working fine..but when i stop the stream in Emby downloading from remote is not stopping and cache is still filling for some time or when whole file is done, depends really, still investigating.

Here are details about config and mount commands:

[gd3]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = {"access_token":"XXXX"}
root_folder_id = 0ABeyc5U163RdUk9PVA

[gcache3]
type = cache
remote = gd3:/gd3

Commented out (old cache settings)
.chunk_size = 16M
.info_age = 1d
.chunk_total_size = 300G

[gcrypt3]
type = crypt
remote = gcache3:/gd3
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

Mount command service:
[Unit]
Description=gdrive3mount
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
KillMode=none
Environment=RCLONE_CONFIG=/home/xbmc/.config/rclone/rclone.conf

ExecStart=/usr/bin/rclone mount gcrypt3: /mnt/md5/gdrive3
--config /home/xbmc/.config/rclone/rclone.conf
--log-level DEBUG
--log-file /home/xbmc/gdrive3.log
--allow-other
--allow-non-empty
--dir-cache-time 1000h
--umask 002
--cache-dir=/mnt/md5/_vfscache/gdrive3vfscache
--vfs-cache-mode full
--vfs-cache-max-size 500G
--vfs-cache-max-age 336h
--buffer-size 384M
--async-read=false \

I want to get rid of old cache completly, but dont know how, cos i count mount if i dont have config like this.
Old cache is active i see folders and so on. Other problem is that when streaming stops...downloading from remote is still going...and that is weird for bot old and new cache backeds.

Cheers !

ExecStop=/bin/fusermount -uz /mnt/md5/gdrive3
Restart=always
RestartSec=5

The only thing you need to do is:

  • Pick the Google Drive remote, in your case looks to be "gd3"
  • Setup a crypt remote to point to your crypt folder in "gd3"
  • Run a mount command with the new version of rclone and vfs-cache-mode full with minimal settings. Only change what you need to.

You should remove that as it can hide/break things.

You can remove that and just use the default buffer.

Thanks @Animosity022

Yeah i thought crypt remote should be reconfigured, since i used recommended setup, gdrive -> crypt - Cache. Honestly i don't know how to do it but will try :slight_smile:
Any info on why is remote still downloading even after stream is stopped ?

Rclone only download if something is requesting it. If you see downloads, something is requesting it.

Yes it seems Emby is the issue, at least he is a suspect. All worked fine until 2 latest beta versions of emby. It was working fine on old cache backend for a year, now once of a sudden stream stops but downloading continues.

Thank you

I have fixed the mounting part, thank you very much for pointing me in right direction, so stupid of me not to see correlation of remotes :frowning: All working fine now without old cache part.

Now the emby problem is in troubleshooting, seems something is buggy in last few versions, streams are downloading after stopped, going mad with this.

thanks again for your kind help.

I really wanted Emby to be my jam, but unfortunately, it just never seems stable and once it seems stable, something breaks again.

For me, Plex just works as I have lifetimes to both.

Well i m on beta channel so more issues are possible but really they are rare, stable is really ok but it is updated very slowly and rare in last year or two. There is no choice for me really between emby and plex, emby is different league if u ask me, plex is ok but emby has so much more, simpler, more automated, good support on forum directly from devs. I m really satisfied with it, plex is fine, but for me it nowhere near emby. Only one thing emby lacks opposite plex, it sucks in IPTV part in management and processing, that's like it is 1986, other than that, emby rocks.

It's dependent on your setup and players. I use ATVs and Emby is so far behind, it's not even funny.

My take is always use what works best for your personal setup. I like Emby's guide and overall layout and such, but direct play and HDR playback has been in for me for like 1.5 years or so if not more.

Definitely. Direct play and HDR are fine on emby, at least for me on Andorid app wich i mainly use.

Now testing VFS cache, and stumbled on small issue, my 1Gbit dedicated server is mostly saturated
when playing, looking for way to limit how "fast" rclone downloads to ram/cache. Started 2 streams and 1 gbit download is full. Any suggestions for this ?

What's the mount command you are using?

Herfe it is, i have removed --buffer command as per your suggestion, but rclone is downloading like mad now :slight_smile:

[Service]
Type=notify
KillMode=none
Environment=RCLONE_CONFIG=/home/xbmc/.config/rclone/rclone.conf

ExecStart=/usr/bin/rclone mount gcrypt3: /mnt/md5/gdrive3
--config /home/xbmc/.config/rclone/rclone.conf
--log-level INFO
--log-file /home/xbmc/gdrive3.log
--allow-other
--dir-cache-time 1000h
--umask 002
--cache-dir=/mnt/md5/_vfscache/gdrive3vfscache
--vfs-cache-mode full
--vfs-cache-max-size 500G
--vfs-cache-max-age 336h
--async-read=false \

I don't see anything obvious as rclone does do quite well at utilizing bandwidth.

There have been a few features raised for per file bandwidth control, which would help out.

You could either cap the overall mount if that is your concern. I personally just let rclone do its thing and prioritize my other traffic as I want to use as much of my bandwidth as I can.

Thing is i have 3 separate mounts divided by content. On normal 720p/1080p content it is ok, its downloading way more than old cache but under control, 1 gbit link is not full.

But on 4K movies..things are out of control, downloading is fully utilized with any options i tried, without buffersize, or with small buffersize, or with read ahead. 4k when i play 1gbit line is fully utilized, even i after i close the stream for 30 seconds or more downloading to cache is still ongoing and 1gbit line fully utilized. Seems i don't know how to limit the 4k files to download only what emby client needs.

Not sure as I don't use Emby and Plex does not exhibit the same behavior.

If I had to guess, I would assume you transcoding and it's filling the transcoding buffer setting in Emby.

Nope, no transcoding at all, all playback is direct.

Too me looks like this is the vfs cahce doing, nothing to do with emby since on smaller files its fine for now.
Seems to me old cache backend pulled from remote exactly how much client needed, while VFS is pulling much much more to fill the cache faster, which in some cases is ok but for mine is not since bandwith is gone :frowning:

Run the mount in debug log.
Recreate the issue.

You should see a file open in the log and a file close and when it's closed, it'll stop downloading.

I did, after stopping stream for 5 secs rclone is still downloading. But that's not the main issue / point.
Main issue is that rclone is downloading way more and way faster than it really needs maxing my connection.

Only workaround i found for now is --bwlimit-file 50M, where i limit the speed on file, that helps but will see
is it the best solution because bit-rates on some files are not the same. But at least i can control it.

I would rather see that rclone "pulls" at the speed that the client in this case emby really needs. Old cache
backend did that really well, and on VFS cache this is problematic for ppl who have lets say emby server
and 10 users on 1 gbit link.

There are a number of enhancements to help bandwidth controls.

Rclone has no idea what is 'needed' as the request from Emby like a hard disk is pulling as much as it can and the challenge is rclone is very efficient at what it does and works to tell.

The old backend did worse and wasn't as efficient so what you are seeing is a lot of great progress and a program that works very well.

If you have 10 users, it wouldn't be an issue as Emby/Plex will split up and grab what's needed as the downloads do spawn multiple connection so the OS handles the majority of that.

Oh man, I missed this.

Use this!

Add --bwlimit-file 6 flag to limit speeds of individual file transfers (Nick Craig-Wood)