VFS cache exceeds maximum size when caching writes

What is the problem you are having with rclone?

rclone exceeds the VFS cache limits on writes when caching writes that exceed the backing storage write rate. There are several threads about this already (e.g. this).

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1-DEV
- os/version: debian trixie/sid (64 bit)
- os/kernel: 6.7.12-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

e.g. WebDAV, sftp/scp, but should affect all

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone \
  --syslog \
  --log-level INFO \
  --vfs-cache-mode full \
  --vfs-cache-max-size 100G \
  --cache-dir /bak/rclone \
  --use-mmap \
  mount \
  --daemon \
  --daemon-wait 10m \
  "$@"

AFAICT there are just no cache quota checks on the call path of WriteFileHandle and Write/WriteAt.

Could this be added? If yes, could someone describe the required changes, so that maybe someone else can implement them?

To hopefully illustrate better what I'm referring to, here's some pseudo code (most likely garbage, I'm unfamiliar with the code base and haven't even tried to compile this):

---
 vfs/read_write.go     | 3 +++
 vfs/vfscache/cache.go | 6 ++++++
 vfs/vfscache/item.go  | 4 ++++
 3 files changed, 13 insertions(+)

diff --git a/vfs/read_write.go b/vfs/read_write.go
index 4130e68b0..adba673e7 100644
--- a/vfs/read_write.go
+++ b/vfs/read_write.go
@@ -363,6 +363,9 @@ func (fh *RWFileHandle) _writeAt(b []byte, off int64, release bool) (n int, err

 // WriteAt bytes to the file at off
 func (fh *RWFileHandle) WriteAt(b []byte, off int64) (n int, err error) {
+	for !fh.d.vfs.cache.QuotasOK() {
+		fh.item.Wait()
+	}
 	fh.mu.Lock()
 	n, err = fh._writeAt(b, off, true)
 	if fh.flags&os.O_APPEND != 0 {
diff --git a/vfs/vfscache/cache.go b/vfs/vfscache/cache.go
index a0af31bd1..8d7fc9b7a 100644
--- a/vfs/vfscache/cache.go
+++ b/vfs/vfscache/cache.go
@@ -740,6 +740,12 @@ func (c *Cache) quotasOK() bool {
 	return c.maxSizeQuotaOK() && c.minFreeSpaceQuotaOK()
 }

+func (c *Cache) QuotasOK() bool {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return c.quotasOK()
+}
+
 // Return true if any quotas set
 func (c *Cache) haveQuotas() bool {
 	return c.opt.CacheMaxSize > 0 || c.opt.CacheMinFreeSpace > 0
diff --git a/vfs/vfscache/item.go b/vfs/vfscache/item.go
index b76688db1..87908ab32 100644
--- a/vfs/vfscache/item.go
+++ b/vfs/vfscache/item.go
@@ -1416,6 +1416,10 @@ func (item *Item) Sync() (err error) {
 	return nil
 }

+func (item *Item) Wait() {
+	item.cond.Wait()
+}
+
 // rename the item
 func (item *Item) rename(name string, newName string, newObj fs.Object) (err error) {
 	item.preAccess()

welcome to the fourm,

so you want rclone to wait(), would there be a timeout, how long would the os wait until it times out?


fwiw, rclone v1.60.1-DEV is a two year old, custom compiled version of rclone.
might want to rclone selfupdate or https://rclone.org/install/#script-installation

1 Like

I don't think a timeout would be strictly needed or generally useful, but one could be added of course. I think the safest option is to just block all writes until uploads have progressed enough that the cache size shrinks below configured limits again.

Thanks, that's from the Debian package though, not a custom build. Looking at the current code I doubt anything regarding this issue would change :slight_smile:

welcome, as per the -DEV, custom compiled, not by rclone project.


yeah, that makes sense.

i think at some point the OS will error out, timeout waiting on rclone.

anyhoo, let's see what other forum member's think of your idea. if it can work, that would be great.