Is there a way to full cache popular file with rclone + script?

[JamesY (James)]

So here's what I want:

User access a file (fileA). rclone/a script checks /mnt/cache (local) if file exists.
If exist user access that file.
If doesn't exist, do a script/rclone copy remote:/path/fileA /mnt/cache/path/fileA.
Then user access from /mnt/cache. Or maybe that user access from rclone remote for the first access time while rclone copy the file to /mnt/cache.
The second time any user access that file, the user will access from /mnt/cache instead of rclone remote.

Once month or when the local partition /mnt/cache is full, whichever one come first, the least accessed files gets deleted to make some space.

Is there a feature like this with rclone or is rclone easily extensible to do above with a script?

[ncw (Nick Craig-Wood)]

@JamesY

Is there a feature like this with rclone or is rclone easily extensible to do above with a script?

You've describe pretty much what --vfs-cache-mode full does.

You can use these flags control how long objects get cached for

  --vfs-cache-max-age duration             Max age of objects in the cache. (default 1h0m0s)
  --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache. (default off)

The only disadvantage of --vfs-cache-mode full is that it downloads the whole file first before letting the user have access to it.

[JamesY (James)]

@ncw

The only disadvantage of --vfs-cache-mode full is that it downloads the whole file first before letting the user have access to it

I'm fine with that but my experience with rclone mount vfs-cache-mode full is that rclone trying to download entire content of Google drive without me doing anything other than executing commands like : find, ncdu, du against root path of a remote. Or issuing ffprobe, ffmpeg against the files in remote.
For this commands I had expected rclone not to download the entire Google drive content,but instead it should just retrieve file/folder metadata or small bits necessary to get the information requested.

[Animosity022]

@JamesY

I'm fine with that but my experience with rclone mount vfs-cache-mode full is that rclone trying to download entire content of Google drive

That's literally what full does. It downloads the entire file before you can access it. It works well if you aren't streaming large files and want to keep things in local storage.

It's not a great use case for streaming, imo.

[JamesY (James)]

@Animosity022

That's literally what full does. It downloads the entire file before you can access it.

But that's not what I mean. Downloading the entire file before user can access is fine. Downloading the entire Google drive isn't fine at all. Say you have 50TB. rclone tries to download all 50TB or that's what it looks like. From my end rclone seems to download all files at all times.

For example you issue commands such as find, ncdu into rclone mount with vfs cache mode full, by doing that you trigger rclone to download all files... Just to finish command 'find'

[ncw (Nick Craig-Wood)]

@JamesY

I'm fine with that but my experience with rclone mount vfs-cache-mode full is that rclone trying to download entire content of Google drive without me doing anything other than executing commands like : find, ncdu, du against root path of a remote. Or issuing ffprobe, ffmpeg against the files in remote.
For this commands I had expected rclone not to download the entire Google drive content,but instead it should just retrieve file/folder metadata or small bits necessary to get the information requested.

WIth --vfs-cache-mode full you should be able to list files and stat them without causing them to be downloaded. You can open them too, but if you read from them then they will be downloaded.

So you should be OK with find, ncdu, du but ffprobe I think reads bytes from the file so will download the entire file first.

I plan to improve this in the next couple of releases...

[Animosity022]

If you try to read a file, it will download it.

If you are reading one byte of a file on a drive, it will download them all.

A 'find' command does not touch the file, as it only reads the metadata about the file.

If you run the mount in debug, you can see it here:

2020/01/28 09:50:59 DEBUG : Movies/: >Lookup: node=Movies/Blue Ruin (2013)/, err=<nil>
2020/01/28 09:50:59 DEBUG : Movies/Blue Ruin (2013)/: Attr:
2020/01/28 09:50:59 DEBUG : Movies/Blue Ruin (2013)/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/28 09:50:59 DEBUG : Movies/Blue Ruin (2013)/: ReadDirAll:
2020/01/28 09:50:59 DEBUG : Movies/Blue Ruin (2013)/: >ReadDirAll: item=1, err=<nil>
2020/01/28 09:50:59 DEBUG : Movies/: Lookup: name="Bo Burnham Make Happy (2016)"
2020/01/28 09:50:59 DEBUG : Movies/: >Lookup: node=Movies/Bo Burnham Make Happy (2016)/, err=<nil>
2020/01/28 09:50:59 DEBUG : Movies/Bo Burnham Make Happy (2016)/: Attr:
2020/01/28 09:50:59 DEBUG : Movies/Bo Burnham Make Happy (2016)/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/28 09:50:59 DEBUG : Movies/Bo Burnham Make Happy (2016)/: ReadDirA

It just hits readdirs and such and doesn't download anything.

rclone ncdu does the same thing.

Here is ncdu running now.

It's all just listing info. If you are download, something else is requesting information and download it as it is not find or ncdu.

[JamesY (James)]

@Animosity022

all just listing info. If you are download, something else is requesting information and download it as it is not find or ncdu.

Hmm weird. Maybe other scripts cause this. I'll have to test.

But I tried find /mnt/Gdrive/ -size +5000M and rclone tried to download everything when I play around with the vfs cache mode full.