Rclone mount buffer not filling when playing large files

What is the problem you are having with rclone?

Rclone buffer is not being filled when playing large files on Plex (Direct Play).

What is your rclone version (output from rclone version)

rclone v1.56.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-1042-raspi (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.16.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive - crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone mount crypt: /mnt/rc-crypt \
	--vfs-cache-mode writes \
	--vfs-read-chunk-size 64M \
	--buffer-size 700M --rc

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Don't have -vv logs since it wasn't possible to interrupt playback and restart the mount daemon.

The buffer appears to build but drops shortly thereafter.

Your answer would be in the log file.

If the file is closed, the buffer drops.

The file’s playing and is not being paused either. This the official Plex app.

Any way to keep the buffer and not drop it??

You'd have to share the log file as we're just guessing on the root cause.

Ok, here you go:

I've set a lower buffer size (300M) to see whether it would affect this, but nope.

If you look at the logs, you can see "Flush" in there and that's the file being closed and re-opened which drops the buffer.

On Linux, you can use something like:

egrep 'Flush|Open' crypt.txt

For a quick look at the file being opened and closed.

If your player is doing that, you may want to use full mode which does a good job at handling that.

I see that, yes. I tried full mode, creating a 2G memory tmpfs because I don't want to store cache files in secondary storage due to high IOwait and stuff.

rc-cache /mnt/rc_cache tmpfs size=2G,noauto,defaults,uid=1001,gid=1001,mode=0755 0 0

However, as you might expect once it was full, another file could not be opened. It loads in chunks but cannot empty in FIFO manner.

	#--vfs-cache-mode=full \
	#--vfs-cache-max-size=2000M \
	#--vfs-cache-max-age 3h \
	#--vfs-read-ahead 500M \
	#--cache-dir=/mnt/rc_cache/crypt \
	#--multi-thread-streams 4 \

The current implementation I'm using is almost perfect, except if it didn't drop the cache.

I'm using mergerFS and the dropcacheonclose option doesn't seem to have an effect on this.

Rclone drops the cache when a file is closed as that's what is happening as shown in the debug log.

Not sure I understand why a secondary disk would cause problems streaming as streams are much slower than disk IO in general. If that was the case, people would local storage wouldn't ever be able to stream.

Depends on what you are doing and how many concurrent streams / bitrates of those streams / the type of secondary disk / etc.

I've created a feature request for this:

Appreciate your help, @Animosity022.

Why Plex client closes file during playback?
Is it a bug?
What causes this strange behavior?
Did you talk to its developers?

Quite a number of the them unfortunately.

The TV apps are the worst.

Full mode solves the problem pretty much and I doubt they really don't fix bigger things let alone something very low level like this as with local storage, it isn't a problem.

OK :open_mouth:
So why not use --vfs-cache-mode=full with some small max cache size and short cache time? This will cache just the last movie(s) as sparse files for a short period of time just to compensate for stupid tv applications.
I think that

  1. VFS cache is versatile enough to compensate in a wide range of situations
  2. Too complex in development/testing to add new unwarranted features

That's what I suggested as well as I don't see any need for the feature.

hi,

some use-cases, like using raspbery pi, have no hard drives, use sdcard.
--vfs-cache-mode is not an option.

for example, a pi zero has slow wifi and having a read ahead with the buffer staying full would be helpful.

Or just use a device that is better suited for the job.

The problem with cheap solutions is that they are cheap.

Similar to use a built in Plex App.

The time invested in trying to work around a bad device / under powered solution would be better spent using a better solution.

Raspberry PIs also have USB ports so you can plug in a flash drive for cache storage if you really wanted to, but if you have slow WiFi, delivering a consistent streaming experience would be tough anyway.

the pizero is fantastic, runs rclone mount, a media server, streams well, runs vpn server.
i use a small usb 5g wifi with it.
and since it supports wired network over usb otp, i use it as a travel router and hotspot.

and yes, i have a pi4 connected with a usb ssd drive.

some rcloners, including myself, do not use --vfs-cache-mode on any device and having a better implemented buffer would be helpful.

i guess time will tell, if there is enough interest in that github issue.

1 Like

For a very small niche case it might be useful.

It can't transcode.
I wouldn't ever have my media server on Wifi.

It really depends on the use case.

For a cheap roaming device like you describe, it be great.

For a dedicated media server, it's just a bad solution if the goal is a reliable device. Few more dollars gets a real solution that just works all the time.

It's a not really a buffer issue per se as it is a poorly implement client opening/closing the files rather than streaming them.

There's really no debating on what's better as it's a personal choice. I value stability and lack of touch over being the cheapest so it doesn't work for me.

So why not use --vfs-cache-mode=full with some small max cache size and short cache time?

I did try that, yes:

There's no issue with the performance of the Pi that would matter here I think.

Animosity, you used to use buffer-size IIRC, but since the VFS overhaul, you switched to that... may I ask why?

vfs-cache-mode set to full isn't really something I want to hammer my SSD with tbh, especially when I have a lot of memory to spare.

buffer-size for in-memory buffer is perfect, as there's nothing touching my storage.