How does VFS read buffering handle massive files?

Help and Support
1

What is the problem you are having with rclone?

No problem yet, I just have a question. I would like to run a public Monero node (the database is one giant .lmdb file around 100GB+) on my VPS, which has a 20GB disk. I really cannot fit this file on the disk, so I decided to link in my 1TB OneDrive account. I've tweaked the VFS settings as seen in the command below. Now I wonder - I've read about read buffering in rclone, but would like to confirm if I understand it correctly - would this scenario work? A local program constantly reading a massive file from a mounted remote on a machine that cannot possibly contain the whole file without the cache making the disk run out of space (--vfs-cache-min-free-space)? I'm also thinking about setting up chunker over crypt over onedrive, as the maximum file size is limited by OneDrive. Thanks in advance.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: unknown
  • os/kernel: 6.10.0-artix1-2 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.5
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

crypt over onedrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config "./rclone.conf" mount \
	--allow-other \
	--vfs-cache-mode full \
	--vfs-cache-max-age 2h \
	--vfs-cache-min-free-space 5G \
	cryptdrive:/ "./cryptdrive"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[cryptdrive]
type = crypt
remote = onedrive:/cryptdrive
password = XXX
password2 = XXX
description = encrypted
filename_encryption = obfuscate

[onedrive]
type = onedrive
drive_type = personal
description = decrypted
token = XXX
drive_id = XXX
2

--vfs-cache-min-free-space is soft limit. The VFS cache is only purged when possible. If your giant file is open by some database app all the time it will grow beyond your limited disk space eventually.

3

Hello, thanks for the quick reply. If this way will make my internal drive run out of space, is there any other way to make it work somehow using OneDrive and not getting a VPS with more disk space? Unfortunately, the node software does not support splitting and can only one huge file.

4

AFAIK there is no way to solve your problem with rclone. You have to look somewhere else.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.