Option 'vfs-read-ahead' has no effect

What is the problem you are having with rclone?

I'm having problems achieving the expected behavior with 'rclone mount'. It is as if '--vfs-read-ahead' does not have any effect. I'm trying to playback a media file, and I would like playback to start after 'buffer-size' plus 'vfs-read-ahead'. But it seems that playback doesn't start until the entire file is read, which is much larger than 'buffer-size' plus 'vfs-read-ahead'. Have I misunderstood the 'vfs-read-ahead' option?
My objective is to be able to start media playback relative fast (5-10 second delay is fine) and having the media file transferred at once. I was expecting to be able to achieve this with 'vfs-read-ahead'.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0

  • os/version: slackware 15.0 (64 bit)
  • os/kernel: 5.15.46-Unraid (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Local remote

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount local-remote: /mnt/user/media-cache/movies/ --allow-other -vv --vfs-cache-mode full --vfs-cache-max-size 64Gi --vfs-read-ahead 16Mi --vfs-read-chunk-size 256Mi --buffer-size 1Gi --log-file "rclone-mount.log"

Only if I set 'vfs-cache-mode' to off, playback starts immediately. However, then the file is read continuously, instead of reading the entire file at once.

The rclone config contents with secrets removed.

[local-remote]
type = alias
remote = /mnt/user/media/rclone-test

A log from the command with the -vv flag

Nothing really happens for 35 seconds, 22:16:01 to 22:16:36.

The image show HDD read activity from 'local-remote'. Playback doesn't start until around 22:16:35, approximately when the HDD activity drops.

This is actually the HDD activity I would like, I just want playback to start much sooner.

What is the player you are using?

Cache mode off/full do the same thing on Plex/Emby as an example. If the player is waiting for the full file, that seems strange.

Also, whats the reason for rclone with a local remote? Why not point to the location directly?

Currently, I testing it with VLC on Windows, however, when I figure out a suitable rclone mount command, I'll be using Kodi on a nVidia Shield.
Anyway, I don't think it is the player, because setting --vfs-cache-mode to off, playback starts immediately (all other options remain unchanged).
The local-remote is a HDD and the mount destination is a SDD. The HDD is quite noisy reading small chunks of data, whereas it is "silent" when reading an entire file at once. Moreover, the HDD could potentially spin down during playback.

Can you share a debug log of the off? The log you shared is also stopped mid reading at 20:16 so doesn't seem to have the full graphic above.

Why would an in use HDD spin down?

I'd remove buffer-size all together and just use 1G read ahead if that's what you want. You'd be hard pressed to really notice a 16M read ahead.

I've never used rclone to run a local disk cache though but full/no operate the same in terms of a file playing. The full just adds a layer of caching in the mix as your file is being served based on the player so doesn't make sense to me why full or off would operate differently unless there is some odd bug which very well could be.

I'm not seeing it in the log though as the log shows data being read back properly from what I can see and nothing looks strange.

grep "Pos:" rclone\ mount\ log.txt

shows everything from cache as well so that should almost all being from whatever the cache dir is.

I'll provide a new log with cache mode off as soon as possible. Full log is here: ZeroBin.net

The HDD is not used if the entire file is copied to the SDD. I just realized, I forgot the read-only argument. However, this is not the objective, so if not possible, no worries.

I deliberately choose a small read ahead to see if that would start playback "immediately", but even with 16M playback doesn't start until after 30 seconds.

With the following command, playback starts immediately.

rclone mount local-remote: /mnt/user/media-cache/movies/ --allow-other -vv --vfs-cache-mode off --vfs-cache-max-size 64Gi --vfs-read-ahead 16Mi --vfs-read-chunk-size 256Mi --buffer-size 1Gi --log-file "rclone-mount.log"

From the picture below, you can see that 1GiB is read from the HDD, then it drops to continues reading small chunks of data.

You can find the log here: ZeroBin.net

My takeaways:

  • The playback behavior is different when changing --vfs-cache-mode to off
  • I don't see any effect from --vfs-read-ahead
  • I don't see chunk size of 256 MiB either

I'm not sure what that picture is showing me in relation to your setup as there no legend/details on what I'm looking at other than a bump in a line with no numbers.

We have many folks that don't use cache as @VBB can share and playback starts before the file is fully downloaded as that's how rclone was used prior to the updates to cache mode full being changed a number of releases back.

As I shared before, remove the 1G buffer-size, use 1G read ahead if that's what you are going for.

I did try reproducing it with a local file and had no issues on/off via Plex. I don't use VLC as I'm not sure how that works in this situation.

Thanks for your reply.

Should have included the y-axis, which is HDD read rate. The peak shows that 1 GiB is read (=buffer size) and then very small amount of data is read continuously. Why is this not equal to --vfs-read-chunk-size?

What I am going for is to read the media file in a few large chunks, instead of a huge number of small chunks, as described in rclone mount
In this example I set --vfs-read-chunk-size to 256 MiB, the duration of the 8 GB media file is 100 minutes, thus I would expect to see HDD read activity every 3 minutes (= 100 minutes / (8GB/256MiB)). But this is not the case.

I see the behavior with VLC and Kodi.

I think the confusion is streaming vs a copy command. If you are streaming in Plex/Emby/etc, you aren't maintaining a copy rate as it's only playing/requesting what is needed for the streams. Anything these days are variable bit rates to it's going to spike/dip as it streams when it fills / empties a buffer.

Buffer size / read ahead work together to stay ahead of that by filling up that space as quickly as possible and as it plays, it works at keeping up on that.

Here's an example from Plex as someone is playing something back so you'll see spikes in the bandwidth as it gets data and serves it.

I only use Plex/Emby so I can't comment on how those work.