I'm having problems achieving the expected behavior with 'rclone mount'. It is as if '--vfs-read-ahead' does not have any effect. I'm trying to playback a media file, and I would like playback to start after 'buffer-size' plus 'vfs-read-ahead'. But it seems that playback doesn't start until the entire file is read, which is much larger than 'buffer-size' plus 'vfs-read-ahead'. Have I misunderstood the 'vfs-read-ahead' option?
My objective is to be able to start media playback relative fast (5-10 second delay is fine) and having the media file transferred at once. I was expecting to be able to achieve this with 'vfs-read-ahead'.
Run the command 'rclone version' and share the full output of the command.
rclone v1.59.0
os/version: slackware 15.0 (64 bit)
os/kernel: 5.15.46-Unraid (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18.3
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Local remote
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone mount local-remote: /mnt/user/media-cache/movies/ --allow-other -vv --vfs-cache-mode full --vfs-cache-max-size 64Gi --vfs-read-ahead 16Mi --vfs-read-chunk-size 256Mi --buffer-size 1Gi --log-file "rclone-mount.log"
Only if I set 'vfs-cache-mode' to off, playback starts immediately. However, then the file is read continuously, instead of reading the entire file at once.
The rclone config contents with secrets removed.
[local-remote]
type = alias
remote = /mnt/user/media/rclone-test
A log from the command with the -vv flag
Nothing really happens for 35 seconds, 22:16:01 to 22:16:36.
The image show HDD read activity from 'local-remote'. Playback doesn't start until around 22:16:35, approximately when the HDD activity drops.
Currently, I testing it with VLC on Windows, however, when I figure out a suitable rclone mount command, I'll be using Kodi on a nVidia Shield.
Anyway, I don't think it is the player, because setting --vfs-cache-mode to off, playback starts immediately (all other options remain unchanged).
The local-remote is a HDD and the mount destination is a SDD. The HDD is quite noisy reading small chunks of data, whereas it is "silent" when reading an entire file at once. Moreover, the HDD could potentially spin down during playback.
Can you share a debug log of the off? The log you shared is also stopped mid reading at 20:16 so doesn't seem to have the full graphic above.
Why would an in use HDD spin down?
I'd remove buffer-size all together and just use 1G read ahead if that's what you want. You'd be hard pressed to really notice a 16M read ahead.
I've never used rclone to run a local disk cache though but full/no operate the same in terms of a file playing. The full just adds a layer of caching in the mix as your file is being served based on the player so doesn't make sense to me why full or off would operate differently unless there is some odd bug which very well could be.
I'm not seeing it in the log though as the log shows data being read back properly from what I can see and nothing looks strange.
grep "Pos:" rclone\ mount\ log.txt
shows everything from cache as well so that should almost all being from whatever the cache dir is.
I'll provide a new log with cache mode off as soon as possible. Full log is here: ZeroBin.net
The HDD is not used if the entire file is copied to the SDD. I just realized, I forgot the read-only argument. However, this is not the objective, so if not possible, no worries.
I deliberately choose a small read ahead to see if that would start playback "immediately", but even with 16M playback doesn't start until after 30 seconds.
I'm not sure what that picture is showing me in relation to your setup as there no legend/details on what I'm looking at other than a bump in a line with no numbers.
We have many folks that don't use cache as @VBB can share and playback starts before the file is fully downloaded as that's how rclone was used prior to the updates to cache mode full being changed a number of releases back.
As I shared before, remove the 1G buffer-size, use 1G read ahead if that's what you are going for.
I did try reproducing it with a local file and had no issues on/off via Plex. I don't use VLC as I'm not sure how that works in this situation.
Should have included the y-axis, which is HDD read rate. The peak shows that 1 GiB is read (=buffer size) and then very small amount of data is read continuously. Why is this not equal to --vfs-read-chunk-size?
What I am going for is to read the media file in a few large chunks, instead of a huge number of small chunks, as described in rclone mount
In this example I set --vfs-read-chunk-size to 256 MiB, the duration of the 8 GB media file is 100 minutes, thus I would expect to see HDD read activity every 3 minutes (= 100 minutes / (8GB/256MiB)). But this is not the case.
I think the confusion is streaming vs a copy command. If you are streaming in Plex/Emby/etc, you aren't maintaining a copy rate as it's only playing/requesting what is needed for the streams. Anything these days are variable bit rates to it's going to spike/dip as it streams when it fills / empties a buffer.
Buffer size / read ahead work together to stay ahead of that by filling up that space as quickly as possible and as it plays, it works at keeping up on that.
Here's an example from Plex as someone is playing something back so you'll see spikes in the bandwidth as it gets data and serves it.
Having spent some time experimenting, I have finally gotten the wanted behavior. Using the flag --vfs-fast-fingerprint in combination with --vfs-cache-mode full playback starts immediately. And now the flag --vfs-read-ahead works as expected.
