Cache Streaming

I am currently having trouble opening a very large file in "rclone mount".

I've tried everything, but it does the full download before opening the file.
I had to enable caching because I use small files to make edits.

Is there any option to open the large file and it will download during playback? (caching enabled)

Command I am using:

sudo rclone mount GoogleDrive: /home/cantalupo/GoogleDrive --allow-other --daemon --vfs-cache-mode full --cache-dir /home/cantalupo/.rclone-cache --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 2G --vfs-cache-max-size 5120M --buffer-size 64M

My system:
rclone v1.48.0

  • os/arch: linux/amd64
  • go version: go1.12.6

If you use vfs-cache-mode full, it downloads the whole file.

https://rclone.org/commands/rclone_mount/#file-caching

You'd want to take that out.

So what would be the best option for caching?
In my case, full cache performs very well when opening small files (eg pdf)

But to open large files is a problem.

If you use full cache, it has to download the whole. That's how it works so if you want to use full cache, you have to wait for the full download of the file.

There isn't anything to tune/fix on that as it just depends on how fast your Internet is.

You can use the cache backend if you want I suppose if that's what you are looking to do. What's your use case you are trying to solve?

1 Like

Exactly...
If I disable caching, streaming will work fine, but to use small files (eg pdf) is very slow and unstable.

You can use the cache backend if that's what you want:

https://rclone.org/cache/

This is what is causing the full file to be downloaded. Try using --cache-mode writes instead which should work for editing files.

1 Like

@Animosity022 @ncw

I tested it, but the problem is not solved.
"full" is great for small files
"write" is great for large files

Would you have any excellent options for both situations?

I recorded this video to demonstrate the performance

VFS write mode shouldn't be any slower than full mode for small files.

Keep in mind that small files are inherently inefficient to transfer compared to large ones. It takes some time to request a file no matter it's size so even tiny files will not feel instant, and transferring a large folder of small files will inevitably not give efficient use of bandwidth. It should not perform any WORSE than full mode though...

I do agree that the PDF you demonstrate in your video is abnormally slow. 2-3 would be perfectly normal, but this is significantly longer. I can't point to any obvious cause, but I do not experience this personally (using VFS write mode).

1 Like

There is a difference between full and write.

full does a complete download of a file for read and write.
write does a download for "most" write activities against a file.

If you put the log into DEBUG and share the log, you can see what's going on.

If you have multiple opens and closes on a file, the cache backend is going to do well for you as you get the chunks local making multiple opens and closes quickly much faster.

1 Like

My solution was to create two mount of the same separate disk.
"Full" for when I am handling small and medium files.
And "no cache" for large files.

In "writes mode", it can't open libreoffice files and small files are also very slow...

I hope that in the future rclone will have some solution about it.
Thank you anyway.
Excellent project.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.