I'm having problems achieving the expected behavior with 'rclone mount'. It is as if '--vfs-read-ahead' does not have any effect. I'm trying to playback a media file, and I would like playback to start after 'buffer-size' plus 'vfs-read-ahead'. But it seems that playback doesn't start until the entire file is read, which is much larger than 'buffer-size' plus 'vfs-read-ahead'. Have I misunderstood the 'vfs-read-ahead' option?
My objective is to be able to start media playback relative fast (5-10 second delay is fine) and having the media file transferred at once. I was expecting to be able to achieve this with 'vfs-read-ahead'.
Run the command 'rclone version' and share the full output of the command.
os/version: slackware 15.0 (64 bit)
os/kernel: 5.15.46-Unraid (x86_64)
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone mount local-remote: /mnt/user/media-cache/movies/ --allow-other -vv --vfs-cache-mode full --vfs-cache-max-size 64Gi --vfs-read-ahead 16Mi --vfs-read-chunk-size 256Mi --buffer-size 1Gi --log-file "rclone-mount.log"
Only if I set 'vfs-cache-mode' to off, playback starts immediately. However, then the file is read continuously, instead of reading the entire file at once.
The rclone config contents with secrets removed.
type = alias
remote = /mnt/user/media/rclone-test
A log from the command with the -vv flag
Nothing really happens for 35 seconds, 22:16:01 to 22:16:36.
The image show HDD read activity from 'local-remote'. Playback doesn't start until around 22:16:35, approximately when the HDD activity drops.
Currently, I testing it with VLC on Windows, however, when I figure out a suitable rclone mount command, I'll be using Kodi on a nVidia Shield.
Anyway, I don't think it is the player, because setting --vfs-cache-mode to off, playback starts immediately (all other options remain unchanged).
The local-remote is a HDD and the mount destination is a SDD. The HDD is quite noisy reading small chunks of data, whereas it is "silent" when reading an entire file at once. Moreover, the HDD could potentially spin down during playback.
Can you share a debug log of the off? The log you shared is also stopped mid reading at 20:16 so doesn't seem to have the full graphic above.
Why would an in use HDD spin down?
I'd remove buffer-size all together and just use 1G read ahead if that's what you want. You'd be hard pressed to really notice a 16M read ahead.
I've never used rclone to run a local disk cache though but full/no operate the same in terms of a file playing. The full just adds a layer of caching in the mix as your file is being served based on the player so doesn't make sense to me why full or off would operate differently unless there is some odd bug which very well could be.
I'm not seeing it in the log though as the log shows data being read back properly from what I can see and nothing looks strange.
grep "Pos:" rclone\ mount\ log.txt
shows everything from cache as well so that should almost all being from whatever the cache dir is.
I'm not sure what that picture is showing me in relation to your setup as there no legend/details on what I'm looking at other than a bump in a line with no numbers.
We have many folks that don't use cache as @VBB can share and playback starts before the file is fully downloaded as that's how rclone was used prior to the updates to cache mode full being changed a number of releases back.
As I shared before, remove the 1G buffer-size, use 1G read ahead if that's what you are going for.
I did try reproducing it with a local file and had no issues on/off via Plex. I don't use VLC as I'm not sure how that works in this situation.
Should have included the y-axis, which is HDD read rate. The peak shows that 1 GiB is read (=buffer size) and then very small amount of data is read continuously. Why is this not equal to --vfs-read-chunk-size?
What I am going for is to read the media file in a few large chunks, instead of a huge number of small chunks, as described in rclone mount
In this example I set --vfs-read-chunk-size to 256 MiB, the duration of the 8 GB media file is 100 minutes, thus I would expect to see HDD read activity every 3 minutes (= 100 minutes / (8GB/256MiB)). But this is not the case.
I think the confusion is streaming vs a copy command. If you are streaming in Plex/Emby/etc, you aren't maintaining a copy rate as it's only playing/requesting what is needed for the streams. Anything these days are variable bit rates to it's going to spike/dip as it streams when it fills / empties a buffer.
Buffer size / read ahead work together to stay ahead of that by filling up that space as quickly as possible and as it plays, it works at keeping up on that.
Here's an example from Plex as someone is playing something back so you'll see spikes in the bandwidth as it gets data and serves it.
Having spent some time experimenting, I have finally gotten the wanted behavior. Using the flag --vfs-fast-fingerprint in combination with --vfs-cache-mode full playback starts immediately. And now the flag --vfs-read-ahead works as expected.