Vfs-cache-mode writes/minimal behaving as --vfs-cache-mode full

What is the problem you are having with rclone?

Cache mode writes and minimal behaving as full and rclone keeps caching to disk. When I used rclone in windows, I used the following command

start rclone --no-console mount GDrive: C:\Users\Rafee\Documents\.Mount\GDrive --no-modtime --rc --rc-addr=:3750 --vfs-cache-mode writes --vfs-cache-max-age 30m --vfs-write-back 25s --cache-dir E:\cache --dir-cache-time 25m

Behaviour on Windows: When playing a video or audio file, it would cache to RAM and vfs cache folder on disk would remain empty with 0 disk write activity.
Behaviour on Linux (MX Linux): Even with --vfs-cache-mode set to minimal (same behaviour with writes), it writes to disk the whole as --vfs-cache-mode full would. Disk activity is really high and the cache folder contains all the played back files i.e the whole file is being cache.
Desired behaviour: rclone will cache to RAM when using vfs-cache-mode writes.
Note: On Windows, the video which is being played usually cached a few second ahead, whereas on Linux it keeps caching until the whole video is cached showing the behaviour of vfs-cache-mode full

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: debian 11.3 (64 bit)
  • os/kernel: 5.10.0-14-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount GDrive: /home/Rafee/Documents/GDrive/ --vfs-cache-mode minimal --cache-dir /home/Rafee/Documents/cache

The rclone config contents with secrets removed.

[GDrive]
type = drive
client_id = 
client_secret = 
scope = drive
root_folder_id = 
token = 
team_drive = 0AME6v_oGR2AdUk9PVA


A log from the command with the -vv flag

None of the cache mode cache to RAM as that's not a thing.

rclone mount

I'm not seeing anything wrong in the logs as that's what I would expect to happen.

The only thing that uses memory is buffer-size as that is how much of the file is kept in memory. Buffer is dumped if the file is closed.

That's not how the cache mode works. This only writes files locally to disk and uploads them. This is used for certain applications/programs that need a more compatible file system to write to. You'll see an error in the log saying you need this if the program is asking to write in a mode that isn't compatible with off.

Both OS'es do the same thing here. It's more dependent on how the file is being accessed. If the file is accessed sequentially, it continues to read ahead based on the buffer size until it's closed out. If you are using full, vfs-read-ahead comes into play as well.

I'm not sure if minimal uses sparse files as it's not called out as I've never used it or found a need for minimal.

Full cache mode definitely uses sparse files so you'll see a difference in what is being reported and what is actually used:

felix@gemini:/cache/TV$ du -sh *
499G	vfs
3.5M	vfsMeta
felix@gemini:/cache/TV$ du -sh * --apparent-size
2.4T	vfs
376K	vfsMeta

as an example as I'm using full so files are sparse which means only parts I've read are 'filled'. No idea how Windows shows that offhand as I'm primarily a Linux user.

1 Like

My bad, I mixed up the terms. What I meant was on videos, when playing back a video, it'd buffer in RAM and that's all. It wouldn't cache anything in disk with vfs-cache-mode writes.

What I want to achieve is when playing back a video it'll only buffer it in RAM and not cache anything in disk at all. Do I need a separate flag for that ( like --buffer-size 1G)? Thanks!

If you don't want to cache anything to disk, you want 'off'.

Buffer-size can be what you want but really depends on the app playing it. If it closes out, it drops the buffer so it can actually waste things.

What problem are you trying to solve?

I was trying to get same behaviour in Windows and the behaviour vfs-cache-mode writes should show which is

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

since I was only reading the video file, yet it was buffering/caching the file to disk on Linux.
I managed to get the desired behaviour for playing back video vfs-cache-mode off and --buffer-size 1G, but I get errors when trying to upload/edit files.

If you are using cache-mode writes, that should not be the case if you reading. Would need to a debug log file and we can validate why as if the app opens it up in write mode even if it's reading only, that would cause a problem. Easy to see in the debug log.

That depends on how the application works and how it opens files. As you said, you'll see ERRORs in the logs if writes cache mode is required. That's nothing rclone can influence as that's application specific on how it asks for a file.

Command:

rclone mount GDrive: /home/Rafee/Documents/Rclone/ --buffer-size 3G --vfs-cache-mode writes --cache-dir /home/Rafee/Documents/cache --log-file=/home/Rafee/Documents/rclonemount.txt --log-level DEBUG

All I did was play back 10 seconds of a video (Brooklyn Nine Nine S01E10) of file size 2.2GB. Size on the cache folder doing the same thing in Windows, 0 MB

image

Log:

Your player is opening files in read / write mode.

2022/05/10 05:41:12 DEBUG : Music/01. Danny Gonzalez - Yummy.flac: Open: flags=OpenReadWrite
2022/05/10 05:41:13 DEBUG : TV Shows/WandaVision/WandaVision.S01E08.HDR.2160p.WEB-DL.DDP.5.1.x265-TOMMY.mkv: Open: flags=OpenReadWrite
2022/05/10 05:41:15 DEBUG : TV Shows/WandaVision/WandaVision.S01E08.HDR.2160p.WEB-DL.DDP.5.1.x265-TOMMY.mkv: Open: flags=OpenReadWrite
2022/05/10 05:41:17 DEBUG : TV Shows/B99/Season 1/Brooklyn.Nine-Nine.S01E01.1080p.WEB-DL.DD+.5.1.x264-TrollHD.mkv: Open: flags=OpenReadWrite
2022/05/10 05:41:18 DEBUG : TV Shows/B99/Season 1/Brooklyn.Nine-Nine.S01E01.1080p.WEB-DL.DD+.5.1.x264-TrollHD.mkv: Open: flags=OpenReadWrite
2022/05/10 05:41:25 DEBUG : TV Shows/B99/Season 1/Brooklyn.Nine-Nine.S01E22.1080p.WEB-DL.DD+.5.1.x264-TrollHD.mkv: Open: flags=OpenReadWrite

That's why you see things on disk.

hi,

might try

  • run the mount with --read-only.
  • not using --vfs-cache-mode flag, in effect, --vfs-cache-mode=off
    as it is not needed to stream media from gdrive.

It depends on the player / software being used. What player the OP is doing is opening via flags=OpenReadWrite so off or read only would fail to work.

yes, it depends on the app and how the app adjusts its behavior.
the app might default to r/w but fall back to readonly.

anyhoo, i would try the options i suggested.

In my testing, mpv doesn't so by all means perhaps an update changed how it works so test away.