Uncertainties regarding vfs cache and copy/sync options

Off Topic
1

Hello all, been digging into rclone deeper these days and found a few things I am not certain about, any idea is appreciated!

  1. Is the vfs/refresh option only for directory and file names? What does the _async true option people use after that do?
    (Sorry my googling has failed me)

  2. When I set the max read ahead size, is it dynamically adjusted for an app to read ahead that much, depending on how much it requests, or does rclone actually go ahead and does it by download the whole amount allotted?

  3. When I have multiple drives mounted I always set a max cache size, does that mean each drives I mount each get that much or does it all total up to that much?

(They all use the same cache dir, like so:)

--cache-dir=/home/user/rclone/cache \
--vfs-read-chunk-size 1M \
--vfs-cache-mode writes \
--vfs-cache-max-size 40G \
--vfs-cache-max-age 2h \
--vfs-cache-poll-interval 5m \
--vfs-read-ahead 5G \
  1. After reading this post: New Feature: vfs-read-chunk-size
    I realized that I should only being using either cache or read-chunk. But since I do sometimes want to write to the drives I think the cache dir and mode is still necessary.

But for things like pure media streaming and storage I should prolly add the

  --vfs-read-chunk-size 128M \
  --vfs-read-chunk-size-limit 2G \

options and set a --read-only so that way the read chunk and work properly. Is this line of thought correct?

  1. Lastly, are the read-chunk-limit also a per drive basis or is it a global limit? Since my vps only got limited amount of space I would like to have a total allowance.

Thanks for looking!

2

Yes.

It puts the job issues to the remote control daemon in the background and it runs so it returns in a service file for example immediately.

It's not dynamic. It will read ahead if the file stays open. If the file closes, it'll stop reading.

The cache is unique to each mount. It cannot be shared.

Just leave at it the default. It is only the range request that is sent to the remote. If the file closes, it doesn't download anymore.

It's per mount. I think that is what you are asking. It doesn't relate to any disk space though. It's the max size the range request will go to if you are sequentially reading a file.

1 Like
3

First of all thank you very much for going through my post, many thanks on my part.

Have some follow ups if you won't mind to elaborate:

  1. (More of a comment)

It puts the job issues to the remote control daemon in the background and it runs so it returns in a service file for example immediately.

I don't have a good enough understanding to know what you are talking about, but it sounds good enough that I will leave the option there as is.

  1. I think this helped a lot:

It's per mount. I think that is what you are asking. It doesn't relate to any disk space though. It's the max size the range request will go to if you are sequentially reading a file.

And from there, lemme explain my understanding.

a. If I use the read-chunk option with the size-limit option, it will read ahead at chunk until the file is closed right?
b. So if I use the read-chunk function, I am actually downloading/reading the files live rather than cache, which literally takes up disks space, is this correct? (Sounds too good to be true)

  1. I think I partially understand this:

Just leave at it the default. It is only the range request that is sent to the remote. If the file closes, it doesn't download anymore.

a. "By default" - do you mean I don't need to use the --vfs flags at all? I know rclone doesn't default to writes at least, so for drives that I need to write to I think I should at least put that there.
b. If I don't set a cache location, rclone cache should still function normally right?
c. I just set the read-ahead for the sake of streaming vids more smoothly, how large do you think I need?

  1. Lastly, I just want to say that I will mount most of my drives at read only with vfs-chunk for best performance. Just now the over large cache blew my machine, it didn't cache but things sure did get stuck quite hard.
4

Think of when you type a command and wait for it to come back. If the command takes 60 seconds, you have to wait 60 seconds to get a prompt. That's a synchronous action. If you run it async, it runs but does so in the background so it takes the same 60 seconds, but instead, you get a prompt back immediately.

Yes

Not sure what you are asking here. The cache is a separate thing and if files are read from 'cache', that's on disk so you aren't making any requests. You can use rclone without using any of the cache backends.

No. That was a specific answer to your question on read and limit.

It would use the default location.

I use 2G for my own settings. I find it really isn't needed imo but I add there for fun.

1 Like