Rclone doesn't seem to do make multithread downloads and uploads to R2

Help and Support
1

What is the problem you are having with rclone?

rclone doesn't seem to do make multithread downloads and uploads to R2

Run the command 'rclone version' and share the full output of the command.

rclone version

rclone v1.64.2

  • os/version: centos 7.9.2009 (64 bit)
  • os/kernel: 5.4.260-1.el7.elrepo.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

R2 (Cloudflare) - compatible with S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone mount r2-storage:models /mnt/downloads/models --vfs-cache-mode full --buffer-size 64M --cache-dir=/pf/r2/cache --log-file /var/log/rclone-models.log  --no-modtime --multi-thread-cutoff 100M --multi-thread-streams 10  --transfers=10  -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[r2-storage]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://XXX.r2.cloudflarestorage.com
acl = bucket-owner-full-control
bucket_acl = public-read-write
upload_cutoff = 100M
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2023/11/11 14:19:19 DEBUG : /: Lookup: name="test.tar.gz"
2023/11/11 14:19:19 DEBUG : /: >Lookup: node=test.tar.gz, err=<nil>
2023/11/11 14:19:19 DEBUG : test.tar.gz: Attr: 
2023/11/11 14:19:19 DEBUG : test.tar.gz: >Attr: a=valid=1s ino=0 size=1965530670 mode=-rw-r--r--, err=<nil>
2023/11/11 14:19:20 DEBUG : /: Attr: 
2023/11/11 14:19:20 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/11/11 14:19:20 DEBUG : : Statfs: 
2023/11/11 14:19:20 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2023/11/11 14:19:20 DEBUG : : Statfs: 
2023/11/11 14:19:20 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2023/11/11 14:19:20 DEBUG : test.tar.gz: Open: flags=OpenReadOnly
2023/11/11 14:19:20 DEBUG : test.tar.gz: Open: flags=O_RDONLY
2023/11/11 14:19:20 DEBUG : test.tar.gz: newRWFileHandle: 
2023/11/11 14:19:20 DEBUG : test.tar.gz: >newRWFileHandle: err=<nil>
2023/11/11 14:19:20 DEBUG : test.tar.gz: >Open: fd=test.tar.gz (rw), err=<nil>
2023/11/11 14:19:20 DEBUG : test.tar.gz: >Open: fh=&{test.tar.gz (rw)}, err=<nil>
2023/11/11 14:19:20 DEBUG : &{test.tar.gz (rw)}: Read: len=16384, offset=0
2023/11/11 14:19:20 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=16384, off=0
2023/11/11 14:19:20 DEBUG : test.tar.gz(0xc000b84000): openPending: 
2023/11/11 14:19:20 DEBUG : test.tar.gz: vfs cache: checking remote fingerprint "1965530670,2023-07-16 11:25:33.917 +0000 UTC," against cached fingerprint ""
2023/11/11 14:19:20 DEBUG : test.tar.gz: vfs cache: truncate to size=1965530670
2023/11/11 14:19:20 DEBUG : : Added virtual directory entry vAddFile: "test.tar.gz"
2023/11/11 14:19:20 DEBUG : test.tar.gz(0xc000b84000): >openPending: err=<nil>
2023/11/11 14:19:20 DEBUG : vfs cache: looking for range={Pos:0 Size:16384} in [] - present false
2023/11/11 14:19:20 DEBUG : test.tar.gz: ChunkedReader.RangeSeek from -1 to 0 length -1
2023/11/11 14:19:20 DEBUG : test.tar.gz: ChunkedReader.Read at -1 length 4096 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:20 DEBUG : test.tar.gz: ChunkedReader.openRange at 0 length 134217728
2023/11/11 14:19:21 DEBUG : test.tar.gz: ChunkedReader.Read at 4096 length 8192 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:21 DEBUG : test.tar.gz: ChunkedReader.Read at 12288 length 16384 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:21 DEBUG : test.tar.gz: ChunkedReader.Read at 28672 length 32768 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=16384, err=<nil>
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: >Read: read=16384, err=<nil>
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: Read: len=65536, offset=49152
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=65536, off=49152
2023/11/11 14:19:21 DEBUG : vfs cache: looking for range={Pos:49152 Size:65536} in [{Pos:0 Size:28672}] - present false
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: Read: len=32768, offset=16384
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=32768, off=16384
2023/11/11 14:19:21 DEBUG : vfs cache: looking for range={Pos:16384 Size:32768} in [{Pos:0 Size:28672}] - present false
2023/11/11 14:19:21 DEBUG : test.tar.gz: ChunkedReader.Read at 61440 length 65536 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=32768, err=<nil>
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: >Read: read=32768, err=<nil>
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=114688
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=114688
2023/11/11 14:19:21 DEBUG : vfs cache: looking for range={Pos:114688 Size:131072} in [{Pos:0 Size:61440}] - present false
2023/11/11 14:19:21 DEBUG : test.tar.gz: ChunkedReader.Read at 126976 length 131072 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=65536, err=<nil>
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: >Read: read=65536, err=<nil>
2023/11/11 14:19:21 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=245760
2023/11/11 14:19:21 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=245760
2023/11/11 14:19:21 DEBUG : vfs cache: looking for range={Pos:245760 Size:131072} in [{Pos:0 Size:126976}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 258048 length 262144 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=376832
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=376832
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:376832 Size:131072} in [{Pos:0 Size:258048}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 520192 length 524288 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=507904
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=507904
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:507904 Size:131072} in [{Pos:0 Size:520192}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=638976
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=638976
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:638976 Size:131072} in [{Pos:0 Size:520192}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 1044480 length 1048576 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=770048
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=770048
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:770048 Size:131072} in [{Pos:0 Size:1044480}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=901120
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=901120
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:901120 Size:131072} in [{Pos:0 Size:1044480}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1032192
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1032192
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1032192 Size:131072} in [{Pos:0 Size:1044480}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1163264
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1163264
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1163264 Size:131072} in [{Pos:0 Size:1044480}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 2093056 length 1048576 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1294336
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1294336
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1294336 Size:131072} in [{Pos:0 Size:2093056}] - present true
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1425408
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1425408
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1425408 Size:131072} in [{Pos:0 Size:2093056}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1556480
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1556480
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1556480 Size:131072} in [{Pos:0 Size:2093056}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1687552
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1687552
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1687552 Size:131072} in [{Pos:0 Size:2093056}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1818624
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1818624
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1818624 Size:131072} in [{Pos:0 Size:2093056}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1949696
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1949696
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:1949696 Size:131072} in [{Pos:0 Size:2093056}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2080768
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2080768
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2080768 Size:131072} in [{Pos:0 Size:2093056}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2211840
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2211840
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2211840 Size:131072} in [{Pos:0 Size:2093056}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 3141632 length 1048576 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2342912
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2342912
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2342912 Size:131072} in [{Pos:0 Size:3141632}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2473984
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2473984
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2473984 Size:131072} in [{Pos:0 Size:3141632}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2605056
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2605056
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2605056 Size:131072} in [{Pos:0 Size:3141632}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2736128
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2736128
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2736128 Size:131072} in [{Pos:0 Size:3141632}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2867200
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2867200
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2867200 Size:131072} in [{Pos:0 Size:3141632}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=2998272
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=2998272
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:2998272 Size:131072} in [{Pos:0 Size:3141632}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3129344
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3129344
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3129344 Size:131072} in [{Pos:0 Size:3141632}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3260416
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3260416
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3260416 Size:131072} in [{Pos:0 Size:3141632}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 4190208 length 1048576 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3391488
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3391488
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3391488 Size:131072} in [{Pos:0 Size:4190208}] - present true
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3522560
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3522560
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3522560 Size:131072} in [{Pos:0 Size:4190208}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3653632
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3653632
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3653632 Size:131072} in [{Pos:0 Size:4190208}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3784704
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3784704
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3784704 Size:131072} in [{Pos:0 Size:4190208}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=3915776
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=3915776
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:3915776 Size:131072} in [{Pos:0 Size:4190208}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4046848
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4046848
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4046848 Size:131072} in [{Pos:0 Size:4190208}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4177920
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4177920
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4177920 Size:131072} in [{Pos:0 Size:4190208}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4308992
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4308992
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4308992 Size:131072} in [{Pos:0 Size:4190208}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 5238784 length 1048576 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4440064
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4440064
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4440064 Size:131072} in [{Pos:0 Size:5238784}] - present true
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4571136
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4571136
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4571136 Size:131072} in [{Pos:0 Size:5238784}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4702208
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4702208
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4702208 Size:131072} in [{Pos:0 Size:5238784}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4833280
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4833280
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4833280 Size:131072} in [{Pos:0 Size:5238784}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=4964352
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=4964352
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:4964352 Size:131072} in [{Pos:0 Size:5238784}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5095424
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5095424
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5095424 Size:131072} in [{Pos:0 Size:5238784}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5226496
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5226496
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5226496 Size:131072} in [{Pos:0 Size:5238784}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5357568
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5357568
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5357568 Size:131072} in [{Pos:0 Size:5238784}] - present false
2023/11/11 14:19:22 DEBUG : test.tar.gz: ChunkedReader.Read at 6287360 length 1048576 chunkOffset 0 chunkSize 134217728
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5488640
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5488640
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5488640 Size:131072} in [{Pos:0 Size:6287360}] - present true
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5619712
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5619712
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5619712 Size:131072} in [{Pos:0 Size:6287360}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5750784
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5750784
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5750784 Size:131072} in [{Pos:0 Size:6287360}] - present true
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=5881856
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=5881856
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:5881856 Size:131072} in [{Pos:0 Size:6287360}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=6012928
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=6012928
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:6012928 Size:131072} in [{Pos:0 Size:6287360}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=6144000
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=6144000
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:6144000 Size:131072} in [{Pos:0 Size:6287360}] - present true
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=6275072
2023/11/11 14:19:22 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=6275072
2023/11/11 14:19:22 DEBUG : vfs cache: looking for range={Pos:6275072 Size:131072} in [{Pos:0 Size:6287360}] - present false
2023/11/11 14:19:22 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=6406144


redacted because was too long
......
......
......
......
......

2023/11/11 14:20:35 DEBUG : vfs cache: looking for range={Pos:1965146112 Size:131072} in [{Pos:0 Size:1965530670}] - present true
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: Read: len=131072, offset=1965277184
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=131072, off=1965277184
2023/11/11 14:20:35 DEBUG : vfs cache: looking for range={Pos:1965277184 Size:131072} in [{Pos:0 Size:1965530670}] - present true
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=131072, err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: >Read: read=131072, err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: Read: len=122880, offset=1965408256
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): _readAt: size=122880, off=1965408256
2023/11/11 14:20:35 DEBUG : vfs cache: looking for range={Pos:1965408256 Size:122414} in [{Pos:0 Size:1965530670}] - present true
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): >_readAt: n=122414, err=EOF
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: >Read: read=122414, err=<nil>
2023/11/11 14:20:35 DEBUG : test.tar.gz: Attr: 
2023/11/11 14:20:35 DEBUG : test.tar.gz: >Attr: a=valid=1s ino=0 size=1965530670 mode=-rw-r--r--, err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: Flush: 
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): RWFileHandle.Flush
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: >Flush: err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: Release: 
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): RWFileHandle.Release
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): close: 
2023/11/11 14:20:35 DEBUG : test.tar.gz: vfs cache: setting modification time to 2023-07-16 11:25:33.917 +0000 UTC
2023/11/11 14:20:35 DEBUG : test.tar.gz(0xc000b84000): >close: err=<nil>
2023/11/11 14:20:35 DEBUG : &{test.tar.gz (rw)}: >Release: err=<nil>
2

welcome to the forum,

Note that mounts usually only do single threaded download whereas rclone copy will do multithreaded downloads.

Mounts are not multi threaded

https://forum.rclone.org/t/multi-threaded-downloads-comments-and-testers-needed/9721/17

3

I see in the documentation this

Multi-thread transfers will be used with rclone mount and rclone serve if --vfs-cache-mode is set to writes or above.

as I am using --vfs-cache-mode full I would expect this to work. Also, I think I saw on the same forum some people mentioning that was working on version 1.47-1.48, so for whatever reason was either removed or there is a bug.

Is there a solution that I could download my large files from R2 as a mounted drive?

4

Where do you see that? What link?

You are comparing to an ancient version and one that the cache was different. You can't compare them.

Don't use a mount. If you want multi threaded, you have to not use a mount until the feature is implemented.

Open issue here -> Support multi-threaded downloads when downloading a file to the cache · Issue #4760 · rclone/rclone (github.com)

5

Where do you see that? What link?

here you can see in the documentation: Documentation

You are comparing to an ancient version and one that the cache was different. You can't compare them.

I was just mentioning that I saw something in an earlier version and as I was expecting this to be still available in the current version

Don't use a mount. If you want multi-threaded, you have to not use a mount until the feature is implemented.

I need to use mount because there is another application that is expecting to find the files in a specific folder. I have 500TB of data, I cannot keep all of it local.

Is there an option to pay one of the contributors to (re)implement this into Rclone?

6

That's not for a mount.

Sponsor (rclone.org)

7

It was never in an earlier version for a mount.

8

[Sponsor (rclone.org) ](https://rclone.org/sponsor/)

I am open to sponsoring it, but would be great if this implementation could have like a timeline. Is there anybody who could coordinate this with me?

9

Just send @ncw a message here or follow the links on the sponsorship area. Either or would be fine I'd imagine.

10

Do you mean that documentation is incorrect? As it is not small typo or old reference to cache backend.

Multi thread transfers will be used with rclone mount and rclone serve if --vfs-cache-mode is set to writes or above.

I do not mind correcting it - after we clarify it.