Vfs minimal caching

Hello. When vfs cache mode is set to minimal, will it still store only the parts that are read from a file? For example, on a 10GB file I read the first 1GB and then read last 1GB. Would this work without downloading the entire file? This will answer whether you can seek with minimal and only download the parts that are requested.

I know the documentation is well put together but for the minimal mode it doesn't seem to be.

The documentation says:

--vfs-cache-mode off -- * Files opened for write can't be seeked

--vfs-cache-mode minimal -- does not explicitly state that files can or can't be seeked, only talks about things in relation to writing.

--vfs-cache-mode full -- explicitly says "In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded."

I hope I've made sense. Thanks in advance.

Right, it's only for writing and not reading.

Apologies I'm not sure what you mean. Can you clarify please?

So if you want to store things locally and use sparse files, you'd use full.

If you use minimal and read a file, it doesn't store anything:

2021/06/29 19:43:38 DEBUG : /: Lookup: name="jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv"
2021/06/29 19:43:38 DEBUG : /: >Lookup: node=jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv, err=<nil>
2021/06/29 19:43:38 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Attr:
2021/06/29 19:43:38 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: >Attr: a=valid=1s ino=0 size=1504953150 mode=-rw-rw-r--, err=<nil>
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Open: flags=OpenReadOnly
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Open: flags=O_RDONLY
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: >Open: fd=jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r), err=<nil>
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: >Open: fh=&{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}, err=<nil>
2021/06/29 19:43:39 DEBUG : &{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}: Read: len=16384, offset=0
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ChunkedReader.openRange at 0 length 134217728
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ChunkedReader.Read at 0 length 4096 chunkOffset 0 chunkSize 134217728
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ChunkedReader.Read at 4096 length 8192 chunkOffset 0 chunkSize 134217728
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ChunkedReader.Read at 12288 length 16384 chunkOffset 0 chunkSize 134217728
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ChunkedReader.Read at 28672 length 32768 chunkOffset 0 chunkSize 134217728
2021/06/29 19:43:39 DEBUG : &{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}: >Read: read=16384, err=<nil>
2021/06/29 19:43:39 DEBUG : &{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}: Flush:
2021/06/29 19:43:39 DEBUG : &{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}: >Flush: err=<nil>
2021/06/29 19:43:39 DEBUG : &{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}: Release:
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ReadFileHandle.Release closing
2021/06/29 19:43:39 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: ChunkedReader.Read at 61440 length 65536 chunkOffset 0 chunkSize 134217728
2021/06/29 19:43:39 DEBUG : &{jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv (r)}: >Release: err=<nil>
2021/06/29 19:43:46 DEBUG : /: Lookup: name="~"
felix@gemini:~/test$ ls -alR ~/.cache/rclone/
/home/felix/.cache/rclone/:
total 16
drwxr-xr-x 4 felix felix 4096 Jun 29 19:41 .
drwx------ 5 felix felix 4096 Feb  7 14:31 ..
drwx------ 3 felix felix 4096 Jun 29 19:41 vfs
drwx------ 3 felix felix 4096 Jun 29 19:41 vfsMeta

/home/felix/.cache/rclone/vfs:
total 12
drwx------ 3 felix felix 4096 Jun 29 19:41 .
drwxr-xr-x 4 felix felix 4096 Jun 29 19:41 ..
drwx------ 2 felix felix 4096 Jun 29 19:41 GD{TKSWb}

/home/felix/.cache/rclone/vfs/GD{TKSWb}:
total 8
drwx------ 2 felix felix 4096 Jun 29 19:41 .
drwx------ 3 felix felix 4096 Jun 29 19:41 ..

/home/felix/.cache/rclone/vfsMeta:
total 12
drwx------ 3 felix felix 4096 Jun 29 19:41 .
drwxr-xr-x 4 felix felix 4096 Jun 29 19:41 ..
drwx------ 2 felix felix 4096 Jun 29 19:41 GD{TKSWb}

/home/felix/.cache/rclone/vfsMeta/GD{TKSWb}:
total 8
drwx------ 2 felix felix 4096 Jun 29 19:41 .
drwx------ 3 felix felix 4096 Jun 29 19:41 ..

but if I use full and head the same file:

felix@gemini:~/test$ ls -alR ~/.cache/rclone/
/home/felix/.cache/rclone/:
total 16
drwxr-xr-x 4 felix felix 4096 Jun 29 19:41 .
drwx------ 5 felix felix 4096 Feb  7 14:31 ..
drwx------ 3 felix felix 4096 Jun 29 19:41 vfs
drwx------ 3 felix felix 4096 Jun 29 19:41 vfsMeta

/home/felix/.cache/rclone/vfs:
total 12
drwx------ 3 felix felix 4096 Jun 29 19:41 .
drwxr-xr-x 4 felix felix 4096 Jun 29 19:41 ..
drwx------ 2 felix felix 4096 Jun 29 19:44 GD{TKSWb}

/home/felix/.cache/rclone/vfs/GD{TKSWb}:
total 132
drwx------ 2 felix felix       4096 Jun 29 19:44 .
drwx------ 3 felix felix       4096 Jun 29 19:41 ..
-rw------- 1 felix felix 1504953150 Dec 21  2016 jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv

/home/felix/.cache/rclone/vfsMeta:
total 12
drwx------ 3 felix felix 4096 Jun 29 19:41 .
drwxr-xr-x 4 felix felix 4096 Jun 29 19:41 ..
drwx------ 2 felix felix 4096 Jun 29 19:44 GD{TKSWb}

/home/felix/.cache/rclone/vfsMeta/GD{TKSWb}:
total 12
drwx------ 2 felix felix 4096 Jun 29 19:44 .
drwx------ 3 felix felix 4096 Jun 29 19:41 ..
-rw-rw-r-- 1 felix felix  286 Jun 29 19:44 jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv

You can see it's a sparse file as it only grabbed a little bit of it:

felix@gemini:~/.cache/rclone/vfs/GD{TKSWb}$ du -sh *
124K	jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
felix@gemini:~/.cache/rclone/vfs/GD{TKSWb}$ du -sh * --apparent-size
1.5G	jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
felix@gemini:~/.cache/rclone/vfs/GD{TKSWb}$

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.