What is the problem you are having with rclone?
rclone exceeds the VFS cache limits on writes when caching writes that exceed the backing storage write rate. There are several threads about this already (e.g. this).
Run the command 'rclone version' and share the full output of the command.
rclone v1.60.1-DEV
- os/version: debian trixie/sid (64 bit)
- os/kernel: 6.7.12-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
e.g. WebDAV, sftp/scp, but should affect all
The command you were trying to run (eg rclone copy /tmp remote:tmp)
/usr/bin/rclone \
--syslog \
--log-level INFO \
--vfs-cache-mode full \
--vfs-cache-max-size 100G \
--cache-dir /bak/rclone \
--use-mmap \
mount \
--daemon \
--daemon-wait 10m \
"$@"
AFAICT there are just no cache quota checks on the call path of WriteFileHandle and Write/WriteAt.
Could this be added? If yes, could someone describe the required changes, so that maybe someone else can implement them?
To hopefully illustrate better what I'm referring to, here's some pseudo code (most likely garbage, I'm unfamiliar with the code base and haven't even tried to compile this):
---
vfs/read_write.go | 3 +++
vfs/vfscache/cache.go | 6 ++++++
vfs/vfscache/item.go | 4 ++++
3 files changed, 13 insertions(+)
diff --git a/vfs/read_write.go b/vfs/read_write.go
index 4130e68b0..adba673e7 100644
--- a/vfs/read_write.go
+++ b/vfs/read_write.go
@@ -363,6 +363,9 @@ func (fh *RWFileHandle) _writeAt(b []byte, off int64, release bool) (n int, err
// WriteAt bytes to the file at off
func (fh *RWFileHandle) WriteAt(b []byte, off int64) (n int, err error) {
+ for !fh.d.vfs.cache.QuotasOK() {
+ fh.item.Wait()
+ }
fh.mu.Lock()
n, err = fh._writeAt(b, off, true)
if fh.flags&os.O_APPEND != 0 {
diff --git a/vfs/vfscache/cache.go b/vfs/vfscache/cache.go
index a0af31bd1..8d7fc9b7a 100644
--- a/vfs/vfscache/cache.go
+++ b/vfs/vfscache/cache.go
@@ -740,6 +740,12 @@ func (c *Cache) quotasOK() bool {
return c.maxSizeQuotaOK() && c.minFreeSpaceQuotaOK()
}
+func (c *Cache) QuotasOK() bool {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ return c.quotasOK()
+}
+
// Return true if any quotas set
func (c *Cache) haveQuotas() bool {
return c.opt.CacheMaxSize > 0 || c.opt.CacheMinFreeSpace > 0
diff --git a/vfs/vfscache/item.go b/vfs/vfscache/item.go
index b76688db1..87908ab32 100644
--- a/vfs/vfscache/item.go
+++ b/vfs/vfscache/item.go
@@ -1416,6 +1416,10 @@ func (item *Item) Sync() (err error) {
return nil
}
+func (item *Item) Wait() {
+ item.cond.Wait()
+}
+
// rename the item
func (item *Item) rename(name string, newName string, newObj fs.Object) (err error) {
item.preAccess()
