Cache->Crypt Remote vs Crypt Remote with "--cache-mode" VFS

Help and Support
2

There are two caches in rclone at the moment, both under development!

I can see them merging in the future though.

First there is the cache backend which can be used with rclone mount, or with any rclone command.

Second there is the VFS cache. The VFS is only used for rclone mount or rclone serve webdav or rclone serve http.

The VFS interprets the --cache-mode flag - I think I should probably rename that to --vfs-cache-mode to save confusion with the backend cache.

Due to the way object storage systems work it isn’t possible to make a fully featured file system without doing some on disk caching. Eg if you want to read and write to the same file at once. Hence the VFS cache. You can read more about it with rclone mount --help - here is an exceprt.

So the VFS cache is just to make rclone mount a better file system. If you set it to full cache mode then it does do caching, but it will download the files in full before you can use them. (I may improve this in future.) The backend cache is far more sophisticated in that it caches the directory structure in a db and caches chunks of the files on disk.

If you are doing simple file systems operations to an rclone mount then you don’t need the VFS cache. If you want to do more demanding stuff like torrent to it, then you will need the VFS cache. --cache-mode writes should be sufficient for most purposes though which will buffer the file to disk while being written, then upload it when it is closed.

I hope that is a bit clearer and sorry for the confusion!

File Caching

NB File caching is EXPERIMENTAL - use with care!

These flags control the file caching options.

--cache-dir string               Directory rclone will use for caching.
--cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
--cache-mode string              Cache mode off|minimal|writes|full (default "off")
--cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)

If run with -vv rclone will print the location of the file cache. The
files are stored in the user cache file area which is OS dependent but
can be controlled with --cache-dir or setting the appropriate
environment variable.

The cache has 4 different modes selected by --cache-mode.
The higher the cache mode the more compatible rclone becomes at the
cost of using disk space.

Note that files are written back to the remote only when they are
closed so if rclone is quit or dies with open files then these won’t
get written back to the remote. However they will still be in the on
disk cache.

–cache-mode off

In this mode the cache will read directly from the remote and write
directly to the remote without caching anything on disk.

This will mean some operations are not possible

  • Files can’t be opened for both read AND write
  • Files opened for write can’t be seeked
  • Existing files opened for write must have O_TRUNC set
  • Files open for read with O_TRUNC will be opened write only
  • Files open for write only will behave as if O_TRUNC was supplied
  • Open modes O_APPEND, O_TRUNC are ignored
  • If an upload fails it can’t be retried

–cache-mode minimal

This is very similar to “off” except that files opened for read AND
write will be buffered to disks. This means that files opened for
write will be a lot more compatible, but uses the minimal disk space.

These operations are not possible

  • Files opened for write only can’t be seeked
  • Existing files opened for write must have O_TRUNC set
  • Files opened for write only will ignore O_APPEND, O_TRUNC
  • If an upload fails it can’t be retried

–cache-mode writes

In this mode files opened for read only are still read directly from
the remote, write only and read/write files are buffered to disk
first.

This mode should support all normal file system operations.

If an upload fails it will be retried up to --low-level-retries times.

–cache-mode full

In this mode all reads and writes are buffered to and from disk. When
a file is opened for read it will be downloaded in its entirety first.

In this mode, unlike the others, when a file is written to the disk,
it will be kept on the disk after it is written to the remote. It
will be purged on a schedule according to --cache-max-age.

This mode should support all normal file system operations.

If an upload or download fails it will be retried up to
–low-level-retries times.

1 Like