Transparent compression with rclone mount

What is the problem you are having with rclone?

I am trying to mount a folder with transparent compression. It seems that rclone has all the bits to do this, but I am running into caching issues.

I am using the Local remote for actual storage. I use the Compress remote for the actual compression (gzip) and the Chunker remote to split large files in multiple chunks, which I thought would help with random access.

This is the command I am using to mount at /tmp/mount, with the actual data (gzipped chunks) in /tmp/chunks:

rclone mount --vfs-cache-mode minimal ':chunker,remote=":compress,remote="":local:/tmp/chunks"":",chunk_size=1Mi,hash_type=sha1:' /tmp/mount

This works well with sequential read and write, but random access fails with this message:

ERROR : dump.sql: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal

Using --vfs-cache-mode minimal or --vfs-cache-mode full copies the ENTIRE big file to ~/.cache/rclone, rather than the affected chunks. This makes the entire scheme useless for my purposes.

Is there a way to have rclone read or write only specific chunks when doing random access?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.19.0-41-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

None (local)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount ':chunker,remote=":compress,remote="":local:/tmp/chunks"":",chunk_size=1Mi,hash_type=sha1:' /tmp/mount

The rclone config contents with secrets removed.

(none, using connection string syntax)

A log from the command with the -vv flag

$ rclone mount -vv ':chunker,remote=":compress,remote="":local:/tmp/chunks"":",chunk_size=1Mi,hash_type=sha1:' /tmp/mount
2023/05/13 14:26:45 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "mount" "-vv" ":chunker,remote=\":compress,remote=\"\":local:/tmp/chunks\"\":\",chunk_size=1Mi,hash_type=sha1:" "/tmp/mount"]
2023/05/13 14:26:45 DEBUG : Creating backend with remote ":chunker,remote=\":compress,remote=\"\":local:/tmp/chunks\"\":\",chunk_size=1Mi,hash_type=sha1:"
2023/05/13 14:26:45 DEBUG : Using config file from "/home/remram/.config/rclone/rclone.conf"
2023/05/13 14:26:45 DEBUG : :chunker: detected overridden config - adding "{ccjOo}" suffix to name
2023/05/13 14:26:45 DEBUG : Creating backend with remote ":compress,remote=\":local:/tmp/chunks\":"
2023/05/13 14:26:45 DEBUG : :compress: detected overridden config - adding "{GIbBX}" suffix to name
2023/05/13 14:26:45 DEBUG : fs cache: renaming cache item ":compress,remote=\":local:/tmp/chunks\":" to be canonical ":compress{GIbBX}:"
2023/05/13 14:26:45 DEBUG : Reset feature "ListR"
2023/05/13 14:26:45 DEBUG : fs cache: renaming cache item ":chunker,remote=\":compress,remote=\"\":local:/tmp/chunks\"\":\",chunk_size=1Mi,hash_type=sha1:" to be canonical ":chunker{ccjOo}:"
2023/05/13 14:26:45 INFO  : Chunked ':chunker{ccjOo}:': poll-interval is not supported by this remote
2023/05/13 14:26:45 DEBUG : Chunked ':chunker{ccjOo}:': Mounting on "/tmp/mount"
2023/05/13 14:26:45 DEBUG : : Root: 
2023/05/13 14:26:45 DEBUG : : >Root: node=/, err=<nil>
2023/05/13 14:26:45 DEBUG : /: Lookup: name=".Trash"
2023/05/13 14:26:46 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2023/05/13 14:26:46 DEBUG : /: Lookup: name=".Trash-1000"
2023/05/13 14:26:46 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2023/05/13 14:26:53 DEBUG : /: Lookup: name="dump.sql"
2023/05/13 14:26:53 DEBUG : /: >Lookup: node=dump.sql, err=<nil>
2023/05/13 14:26:53 DEBUG : dump.sql: Attr: 
2023/05/13 14:26:53 DEBUG : dump.sql: >Attr: a=valid=1s ino=0 size=5263851520 mode=-rw-rw-r--, err=<nil>
2023/05/13 14:26:53 DEBUG : dump.sql: Open: flags=OpenReadWrite
2023/05/13 14:26:53 DEBUG : dump.sql: Open: flags=O_RDWR
2023/05/13 14:26:53 DEBUG : dump.sql: >Open: fd=dump.sql (w), err=<nil>
2023/05/13 14:26:53 DEBUG : dump.sql: >Open: fh=&{dump.sql (w)}, err=<nil>
2023/05/13 14:27:03 DEBUG : &{dump.sql (w)}: Write: len=10, offset=10000000
2023/05/13 14:27:03 DEBUG : dump.sql: waiting for in-sequence write to 10000000 for 1s
2023/05/13 14:27:04 DEBUG : dump.sql: aborting in-sequence write wait, off=10000000
2023/05/13 14:27:04 DEBUG : dump.sql: failed to wait for in-sequence write to 10000000
2023/05/13 14:27:04 ERROR : dump.sql: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/05/13 14:27:04 DEBUG : &{dump.sql (w)}: >Write: written=0, err=illegal seek
2023/05/13 14:27:04 DEBUG : &{dump.sql (w)}: Flush: 
2023/05/13 14:27:04 DEBUG : dump.sql: WriteFileHandle.Flush unwritten handle, writing 0 bytes to avoid race conditions
2023/05/13 14:27:04 ERROR : dump.sql: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2023/05/13 14:27:04 DEBUG : &{dump.sql (w)}: >Flush: err=operation not permitted
2023/05/13 14:27:04 DEBUG : &{dump.sql (w)}: Release: 
2023/05/13 14:27:04 DEBUG : dump.sql: WriteFileHandle.Release closing
2023/05/13 14:27:04 DEBUG : &{dump.sql (w)}: >Release: err=<nil>
$ rclone mount -vv --vfs-cache-mode writes ':chunker,remote=":compress,remote="":local:/tmp/chunks"":",chunk_size=1Mi,hash_type=sha1:' /tmp/mount
2023/05/13 14:28:11 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "mount" "-vv" "--vfs-cache-mode" "writes" ":chunker,remote=\":compress,remote=\"\":local:/tmp/chunks\"\":\",chunk_size=1Mi,hash_type=sha1:" "/tmp/mount"]
2023/05/13 14:28:11 DEBUG : Creating backend with remote ":chunker,remote=\":compress,remote=\"\":local:/tmp/chunks\"\":\",chunk_size=1Mi,hash_type=sha1:"
2023/05/13 14:28:11 DEBUG : Using config file from "/home/remram/.config/rclone/rclone.conf"
2023/05/13 14:28:11 DEBUG : :chunker: detected overridden config - adding "{ccjOo}" suffix to name
2023/05/13 14:28:11 DEBUG : Creating backend with remote ":compress,remote=\":local:/tmp/chunks\":"
2023/05/13 14:28:11 DEBUG : :compress: detected overridden config - adding "{GIbBX}" suffix to name
2023/05/13 14:28:11 DEBUG : fs cache: renaming cache item ":compress,remote=\":local:/tmp/chunks\":" to be canonical ":compress{GIbBX}:"
2023/05/13 14:28:11 DEBUG : Reset feature "ListR"
2023/05/13 14:28:11 DEBUG : fs cache: renaming cache item ":chunker,remote=\":compress,remote=\"\":local:/tmp/chunks\"\":\",chunk_size=1Mi,hash_type=sha1:" to be canonical ":chunker{ccjOo}:"
2023/05/13 14:28:11 INFO  : Chunked ':chunker{ccjOo}:': poll-interval is not supported by this remote
2023/05/13 14:28:11 DEBUG : vfs cache: root is "/home/remram/.cache/rclone"
2023/05/13 14:28:11 DEBUG : vfs cache: data root is "/home/remram/.cache/rclone/vfs/:chunker{ccjOo}"
2023/05/13 14:28:11 DEBUG : vfs cache: metadata root is "/home/remram/.cache/rclone/vfsMeta/:chunker{ccjOo}"
2023/05/13 14:28:11 DEBUG : Creating backend with remote "/home/remram/.cache/rclone/vfs/:chunker{ccjOo}/"
2023/05/13 14:28:11 DEBUG : fs cache: renaming cache item "/home/remram/.cache/rclone/vfs/:chunker{ccjOo}/" to be canonical "/home/remram/.cache/rclone/vfs/:chunker{ccjOo}"
2023/05/13 14:28:11 DEBUG : Creating backend with remote "/home/remram/.cache/rclone/vfsMeta/:chunker{ccjOo}/"
2023/05/13 14:28:11 DEBUG : fs cache: renaming cache item "/home/remram/.cache/rclone/vfsMeta/:chunker{ccjOo}/" to be canonical "/home/remram/.cache/rclone/vfsMeta/:chunker{ccjOo}"
2023/05/13 14:28:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/05/13 14:28:11 DEBUG : Chunked ':chunker{ccjOo}:': Mounting on "/tmp/mount"
2023/05/13 14:28:11 DEBUG : : Root: 
2023/05/13 14:28:11 DEBUG : : >Root: node=/, err=<nil>
2023/05/13 14:28:11 DEBUG : /: Lookup: name=".Trash"
2023/05/13 14:28:12 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2023/05/13 14:28:12 DEBUG : /: Lookup: name=".Trash-1000"
2023/05/13 14:28:12 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2023/05/13 14:28:16 DEBUG : /: Lookup: name="dump.sql"
2023/05/13 14:28:16 DEBUG : /: >Lookup: node=dump.sql, err=<nil>
2023/05/13 14:28:16 DEBUG : dump.sql: Attr: 
2023/05/13 14:28:16 DEBUG : dump.sql: >Attr: a=valid=1s ino=0 size=5263851520 mode=-rw-rw-r--, err=<nil>
2023/05/13 14:28:16 DEBUG : dump.sql: Open: flags=OpenReadWrite
2023/05/13 14:28:16 DEBUG : dump.sql: Open: flags=O_RDWR
2023/05/13 14:28:16 DEBUG : dump.sql: newRWFileHandle: 
2023/05/13 14:28:16 DEBUG : dump.sql: >newRWFileHandle: err=<nil>
2023/05/13 14:28:16 DEBUG : dump.sql: >Open: fd=dump.sql (rw), err=<nil>
2023/05/13 14:28:16 DEBUG : dump.sql: >Open: fh=&{dump.sql (rw)}, err=<nil>
2023/05/13 14:28:22 DEBUG : &{dump.sql (rw)}: Write: len=10, offset=10000000
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): _writeAt: size=10, off=10000000
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): openPending: 
2023/05/13 14:28:22 DEBUG : dump.sql: vfs cache: checking remote fingerprint "5263851520,2023-05-13 18:17:32.357656915 +0000 UTC" against cached fingerprint ""
2023/05/13 14:28:22 DEBUG : dump.sql: vfs cache: truncate to size=5263851520
2023/05/13 14:28:22 DEBUG : : Added virtual directory entry vAddFile: "dump.sql"
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): >openPending: err=<nil>
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): >_writeAt: n=10, err=<nil>
2023/05/13 14:28:22 DEBUG : &{dump.sql (rw)}: >Write: written=10, err=<nil>
2023/05/13 14:28:22 DEBUG : &{dump.sql (rw)}: Flush: 
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): RWFileHandle.Flush
2023/05/13 14:28:22 DEBUG : &{dump.sql (rw)}: >Flush: err=<nil>
2023/05/13 14:28:22 DEBUG : &{dump.sql (rw)}: Release: 
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): RWFileHandle.Release
2023/05/13 14:28:22 DEBUG : dump.sql(0xc000066040): close: 
2023/05/13 14:28:22 DEBUG : vfs cache: looking for range={Pos:0 Size:5263851520} in [{Pos:10000000 Size:10}] - present false
2023/05/13 14:28:22 DEBUG : dump.sql: ChunkedReader.RangeSeek from -1 to 0 length -1
2023/05/13 14:28:22 DEBUG : dump.sql: ChunkedReader.Read at -1 length 4096 chunkOffset 0 chunkSize 134217728
2023/05/13 14:28:22 DEBUG : dump.sql: ChunkedReader.openRange at 0 length 134217728
2023/05/13 14:28:22 DEBUG : dump.sql.XAAAAAAAAAA.gz: ChunkedReader.Read at -1 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.XAAAAAAAAAA.gz: ChunkedReader.openRange at 0 length 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at -1 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.openRange at 0 length 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 4096 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 8192 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 12288 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 16384 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 20480 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 24576 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 28672 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 32768 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 36864 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 40960 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 45056 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 49152 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 53248 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 57344 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 61440 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 65536 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 69632 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 73728 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 77824 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 81920 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 86016 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 90112 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 94208 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 98304 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 102400 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 106496 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 110592 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 114688 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 118784 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 122880 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 126976 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 131072 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 135168 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 139264 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 143360 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 147456 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 151552 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 155648 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 159744 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 163840 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 167936 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 172032 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 176128 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 180224 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 184320 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 188416 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 192512 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 196608 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 200704 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 204800 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 208896 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 212992 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 217088 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 221184 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 225280 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 229376 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 233472 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 237568 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 241664 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 245760 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 249856 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 253952 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 258048 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 262144 length 4096 chunkOffset 0 chunkSize 262144
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.openRange at 262144 length 524288
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 266240 length 4096 chunkOffset 262144 chunkSize 524288
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 270336 length 4096 chunkOffset 262144 chunkSize 524288
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 274432 length 4096 chunkOffset 262144 chunkSize 524288
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 278528 length 4096 chunkOffset 262144 chunkSize 524288
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 282624 length 4096 chunkOffset 262144 chunkSize 524288
2023/05/13 14:28:22 DEBUG : dump.sql.rclone_chunk.001.AAAQAAAAAAA.gz: ChunkedReader.Read at 286720 length 4096 chunkOffset 262144 chunkSize 524288
...
$ du -hs ~/.cache/rclone
5.0G    /home/remram/.cache/rclone

Not currently no. If you want to do random writes to the middle of a file, rclone will need to have it in the VFS cache. Chunker is working at a level below the VFS cache so using it doesn't help here.

That's a shame, it reduces the usefulness of the Chunker remote a lot. I understand the unwillingness to have random-access writes, since most cloud providers don't support it, however you could use Chunker with all of them and get random-access writes...

It's probably a big change to the code though.

We may do that someday... But probably not any time soon!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.