Moving from Cache to VFS - Plex Streaming Recommended Settings

Help and Support
1

Hi All,
LONNNNNNNNNNG time rclone user (early 2018) and I have a modest (100+TB) plex library sitting in google.

I have been using rclone for a long time and specifically the implementation of Cache. I have taken some time to do some Christmas clean up on my environment and have noticed that CACHE will be deprecated shortly and to use VFS instead. I have taken a stab at putting VFS in, and I have reviewed a lot of threads of how to use VFS and am still not able to get anywhere near the performance (specifically listing directories and contents) of CACHE with VFS.

Im hoping that this thread can be not only for myself but a placeholder for others to use as a default set of recommendations.

With some INCREDIBLY BASIC troubleshooting, when mounting without cache, opening my TV directory takes ~7minutes (based on -vv outputting each folder name as it index's it and then refreshes explorer).

With cache this is near instant.

What is the problem you are having with rclone?

looking for recommended settings to ensure fast listing of directories and access on par with cache

What is your rclone version (output from rclone version)

C:\Users\david\Downloads\rclone>rclone.exe --version
rclone v1.57.0

  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.19044.1415 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

C:\Users\david\Downloads\rclone>

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

With VFS

rclone.exe mount decrypt: c:\gDrive

I have also tried

rclone.exe mount decrypt: c:\gDrive --vfs-cache-mode full

Without VFS (therefore using cache)

rclone.exe mount this_one: c:\gDrive

The rclone config contents with secrets removed.

C:\Users\david\AppData\Roaming\rclone>more rclone.conf
[gdrive]
type = drive
client_id = ****.apps.googleusercontent.com
client_secret = ****T22Ba
scope = drive
token = {"access_token":"***e2j","token_type":"Bearer","refresh_token":"1****ouaqCLE","expiry":"2022-01-04T16:23:41.9326749+11:00"}
team_drive =

[decrypt]
type = crypt
remote = gdrive:
password = ***
password2 = ***

[this_one]
type = cache
remote = decrypt:

C:\Users\david\AppData\Roaming\rclone>

A log from the command with the -vv flag

Paste  log here
2

hello and welcome to the forum,

that has already is the case, for many months now.

fwiw, i run a emby server locally to stream from a seedbox.
i do not use the vfs file cache, in effect --vfs-cache-mode=off

note: rclone has two vfs caches.
i wrote a short post about it here

--- one option is to use --network-mode, as this will prevent windows explorer from generating thumbnails.
--- @VBB has a way to prime, the vfs dir cache, hopefully IT can stop soon by and explain.

3

image

4

Anywho, seems this topic comes up at least once every two weeks :wink:

Here's some copypasta, courtesy of yours truly:

Here are the three batch files I use. The first one is a read-only mount, which stays up until I get a chance to update it with a newer (beta) version of Rclone. So, if no one is watching at the time, I do this about once a day:

@echo off
title Rclone Mount READ ONLY
D:\Programs\Rclone\rclone mount --attr-timeout 5000h --dir-cache-time 5000h --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --no-checksum --poll-interval 0 --rc --read-only --user-agent ******* -v Google_Drive_Crypt: G:
pause

Note the --rc flag, as that is necessary for the prime command to find the mount. And here it is:

@echo off
title Rclone Prime
D:\Programs\Rclone\rclone rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --user-agent
pause

I run the above about once a day, usually after all uploads are done and I'm about to run a Plex scan. Lastly, here's my read-write mount, which I use for moving things around after uploading:

@echo off
title Rclone Mount READ/WRITE
D:\Programs\Rclone\rclone mount --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --user-agent ******* -v Google_Drive_Crypt: F:
pause

2 Likes
5

Using the above, there is no slowdown in Windows Explorer or while scanning.

6

hi,

  • in the past, were you using a read/write mount or rclone browser?
  • why two mounts?
7

I use both:

RcloneBrowser for all my uploading.

Read/Write mount for moving things around once they're uploaded.

8

ok but why not just use one mount for both use-cases? streaming and file copy/move?
in other words, what is the reason for a dedicated mount using --read-only?

9

I prefer having a read-only mount for Plex, because I feel it allows for the fastest access times (with the appropriate flags), and I don't have to worry about accidentally deleting stuff.

10

Thank you very much for your reply and I have no doubt that it comes up regularly. With a lot of data being relied on its important to get the right setup working.

I will absolutely use your first 2 as I have other avenues of dropping content into the drive. Thank you very much and I will report back what I find :slight_smile:

On initial review, it doesn't appear that the mount that you are using to access the library (Rclone Mount READ ONLY) uses VFS at all. My presumption is that you are using this as your library location. Do you not run into issues with slowness / API limits / general unresponsive issues? Is this overcome by the Rclone Prime action?

11

You're correct. I do not use the VFS cache, and I've never run into any issues, although my last initial Plex scan was back in 2017. The priming (directory cache) helps tremendously with Explorer listing and therefore media scans. I've also been running additional media analysis every night for the past six months or so without running into API limits.

You might have noticed that my main mount is read-only, so feel free to change that to whatever suits you. For a writeable mount, --no-checksum does not make sense, so I'd recommend removing that, along with --poll-interval 0.

12

hey, me too, did you see my initial post?

13

You need to make your posts prettier, you know, with pictures and stuff in bold :wink:

14

Thanks, i did note that and the mount I do locally - read only is fine.

I have started to recreate my plex database on this new machine (for reasons). I have currently started the scan based on your settings and thus far its seems to be working well.

Can I please ask how you are scheduling and running your commands? My plex work has been always on linux and used crontab or process monitoring software, yours appears to be a windows install? This new box is also a windows box - are you just doing scheduled tasks?

15

Absolutely i did mate, and i read your post too on the 2 diff caches of VFS (I think I had seen the same post before too).

Fingers crossed that the works that VBB has done and the common position from yourself of not needing VFS cache for just the reading of the mount will serve as a best practice for all people to use.

16

I do everything manually, except for the scheduled tasks from within Plex itself. The mounts and prime commands are run as batch files, also manually, and they stay open until I close them. All downloading, uploading, moving around, etc. are also done by hand. Can you tell I'm a glutton for punishment? :pleading_face:

17

ok great thanks :slight_smile: yes you are :wink: haha

just quickly I have had to amend your script a little to get RC to work. It appears that the rc was never working - as it originally got this error:

C:\Users\david>c:\Users\david\Downloads\rclone\rclone.exe rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --rc-user rclone --rc-pass rclone -vv
2022/01/05 10:05:15 DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\Users\david\Downloads\rclone\rclone.exe" "rc" "vfs/refresh" "recursive=true" "--drive-pacer-burst" "200" "--drive-pacer-min-sleep" "10ms" "--timeout" "30m" "--rc-user" "rclone" "--rc-pass" "rclone" "-vv"]
2022/01/05 10:35:15 DEBUG : 2 go routines active
2022/01/05 10:35:15 Failed to rc: connection failed: Post "http://localhost:5572/vfs/refresh": net/http: timeout awaiting response headers

I then amended my mount script to include

--rc-web-gui --rc-user rclone --rc-pass rclone --rc-addr=10.4.100.141:5572

to the prime script I also added the above (apart from rc-web-gui.

I am doing -vv and not getting any update from the vfs/refresh ( its now midday and I started this at 11:51am)

C:\Users\david>c:\Users\david\Downloads\rclone\rclone.exe rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --rc-user rclone --rc-pass rclone --rc-addr=10.4.100.141:5572 -vv
2022/01/05 11:51:30 DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\Users\david\Downloads\rclone\rclone.exe" "rc" "vfs/refresh" "recursive=true" "--drive-pacer-burst" "200" "--drive-pacer-min-sleep" "10ms" "--timeout" "30m" "--rc-user" "rclone" "--rc-pass" "rclone" "--rc-addr=10.4.100.141:5572" "-vv"]

for completeness this is my mount command which is running. i would expect to have had some output in the VFS Refresh console though - any ideas why its not?

c:\users\David\Downloads\rclone\rclone.exe mount --attr-timeout 5000h --dir-cache-time 5000h --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --no-checksum --poll-interval 0 --rc --rc-addr 10.4.100.141:5572 --rc-web-gui --rc-user rclone --rc-pass rclone --read-only -v decrypt: c:\gDrive

18

Makes sense that RC didn't work, because you set a user/password combo. The IP you can probably omit, since you're using the default port anyway, but if it works, just leave it like you have it now.

Unfortunately, there is no visible progress while priming. So, as long as you have an appropriate amount of minutes set as timeout, it will finish. Mine takes about 4-5 minutes.

19

--- 2022/01/05 10:05:15 - 2022/01/05 10:35:15 is exactly 30 minutes.
-- that exactly maches --timeout 30m
--- thus timeout awaiting response headers

20

yup added auth to both so now that seems to at least allow the GUI to work. i will report back in half hour (I guess that's when the timeout happens) based on --timeout 30m and see if error is still around.

----edit-----
of course it just finished when reading this message.

C:\Users\david>c:\Users\david\Downloads\rclone\rclone.exe rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --rc-user rclone --rc-pass rclone --rc-addr=10.4.100.141:5572 -vv
2022/01/05 11:51:30 DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\Users\david\Downloads\rclone\rclone.exe" "rc" "vfs/refresh" "recursive=true" "--drive-pacer-burst" "200" "--drive-pacer-min-sleep" "10ms" "--timeout" "30m" "--rc-user" "rclone" "--rc-pass" "rclone" "--rc-addr=10.4.100.141:5572" "-vv"]
{
"result": {
"": "OK"
}
}
2022/01/05 12:04:23 DEBUG : 4 go routines active

C:\Users\david>

Great thanks I will see how it goes now with a new scan :slight_smile: