Vfs cache read without write

Feature
1

Current vfs cache offers, none, full and write options. Would it make sense to have a read option as well.

Use case: I’ve an application that reads very frequently from a remote and writes infrequently. The network connection could go down, here a read cache will speed up repeated reads but having write cache is a problem, since, if the remote goes down cache will never be sent.

1 Like
2

If you use --vfs-cache-mode full then the files will be uploaded when the network comes back.

Or if you want synchronous upload use --vfs-write-back 0

--vfs-write-back duration              Time to writeback files after last use when
3

files will be uploaded when the network comes back.

I was running this on a rasp pi and it suddenly crashed, it’s running an immich server that does 2 relevant tasks

  1. Upload a file to uploads/file.jpg
  2. Mark that uploads/file.jpg is successfully uploaded in a remote db

Heres uploads is a network mount with vfs cache full enabled.

I suspect the following happened:

  1. Upload a file to uploads/file.jpg
  2. File cached at uploads/file.jpg [not yet sent to network mount]
  3. Mark that uploads/file.jpg is successfully uploaded
  4. crash
  5. recover (local fs wiped)
  6. db inconsistent as file.jpg does not exist on network mount but db says that it is

It was never actually able to upload the said file because of write cache.

4

If the vfs cache is wiped there isn't much rclone can do about that. If the vfs cache was intact, rclone would have uploaded the file on restart.

5

Would it make sense to have a vfs cache for reads only in that case ?

6

I think having a vfs cache only for reads would be possible but I think it would give a bad user experience. It would have all the limitations of --vfs-cache-mode off but with all the downsides of having a local cache.

You could try the union writeback cache with --vfs-cache-mode off that might be more what you are looking for

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.