What is the problem you are having with rclone?
I'm using rclone serve to serve images from a b2 backend over http. This worked fine until now, but now there is a new "folder" with more than 100k images and for images in this folder the responses are ridiculously slow.
Run the command 'rclone version' and share the full output of the command.
rclone v1.69.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-48-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Backblaze B2
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone serve http b2:images --baseurl "/img" --addr 10.5.1.22:8888 --read-only --fast-list --transfers 24 --no-modtime --no-checksum
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[b2]
type = b2
account = XXX
key = XXX
A log from the command that you were trying to run with the -vv
flag
Apr 10 11:08:33 web01 bash[126803]: DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "serve" "http" "b2:images" "-vv" "--baseurl" "/img" "--addr" "10.5.1.22:8888" "--read-only" "--fast-list" "--transfers" "24" "--no-modtime" "--no-checksum"]
Apr 10 11:08:33 web01 bash[126803]: DEBUG : rclone: systemd logging support activated
Apr 10 11:08:33 web01 bash[126803]: DEBUG : Creating backend with remote "b2:images"
Apr 10 11:08:33 web01 bash[126803]: DEBUG : Using RCLONE_CONFIG_PASS password.
Apr 10 11:08:33 web01 bash[126803]: DEBUG : Using config file from "/home/rclone/.config/rclone/rclone.conf"
Apr 10 11:08:34 web01 bash[126803]: INFO : B2 bucket images: poll-interval is not supported by this remote
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75915.webp: Open: flags=O_RDONLY
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75915.webp: >Open: fd=515266479078/s/image-75915.webp (r), err=<nil>
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75915.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75911.webp: Open: flags=O_RDONLY
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75911.webp: >Open: fd=515266479078/s/image-75911.webp (r), err=<nil>
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75911.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75916.webp: Open: flags=O_RDONLY
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75916.webp: >Open: fd=515266479078/s/image-75916.webp (r), err=<nil>
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75916.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75913.webp: Open: flags=O_RDONLY
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75913.webp: >Open: fd=515266479078/s/image-75913.webp (r), err=<nil>
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75913.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75912.webp: Open: flags=O_RDONLY
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75912.webp: >Open: fd=515266479078/s/image-75912.webp (r), err=<nil>
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75912.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75914.webp: Open: flags=O_RDONLY
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75914.webp: >Open: fd=515266479078/s/image-75914.webp (r), err=<nil>
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75914.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75911.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75911.webp: ChunkedReader.Read at 512 length 13536 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75912.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:27 web01 bash[126803]: DEBUG : 515266479078/s/image-75912.webp: ChunkedReader.Read at 512 length 16372 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75915.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75915.webp: ChunkedReader.Read at 512 length 32768 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75915.webp: ChunkedReader.Read at 33280 length 13752 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75916.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75916.webp: ChunkedReader.Read at 512 length 17278 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75913.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75913.webp: ChunkedReader.Read at 512 length 13722 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75914.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:28 web01 bash[126803]: DEBUG : 515266479078/s/image-75914.webp: ChunkedReader.Read at 512 length 15156 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:34 web01 bash[126803]: INFO :
Apr 10 11:09:34 web01 bash[126803]: Transferred: 122.711 KiB / 122.711 KiB, 100%, 20.451 KiB/s, ETA 0s
Apr 10 11:09:34 web01 bash[126803]: Transferred: 6 / 6, 100%
Apr 10 11:09:34 web01 bash[126803]: Elapsed time: 6.4s
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75916.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75916.webp: >Open: fd=515266479078/t/image-75916.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75916.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75915.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75915.webp: >Open: fd=515266479078/t/image-75915.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75915.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75914.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75914.webp: >Open: fd=515266479078/t/image-75914.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75914.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75913.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75913.webp: >Open: fd=515266479078/t/image-75913.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75913.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75912.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75912.webp: >Open: fd=515266479078/t/image-75912.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75912.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75911.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75911.webp: >Open: fd=515266479078/t/image-75911.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75911.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75916.webp: ChunkedReader.Read at 0 length 436 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75913.webp: ChunkedReader.Read at 0 length 368 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75910.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75910.webp: >Open: fd=515266479078/t/image-75910.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75910.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75911.webp: ChunkedReader.Read at 0 length 418 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75914.webp: ChunkedReader.Read at 0 length 394 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75909.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75909.webp: >Open: fd=515266479078/t/image-75909.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75909.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75915.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75915.webp: ChunkedReader.Read at 512 length 256 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75908.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75908.webp: >Open: fd=515266479078/t/image-75908.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75908.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75907.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75907.webp: >Open: fd=515266479078/t/image-75907.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75907.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75906.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75906.webp: >Open: fd=515266479078/t/image-75906.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75906.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75909.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75909.webp: ChunkedReader.Read at 512 length 168 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75912.webp: ChunkedReader.Read at 0 length 402 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75910.webp: ChunkedReader.Read at 0 length 414 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75904.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75904.webp: >Open: fd=515266479078/t/image-75904.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75904.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75903.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75903.webp: >Open: fd=515266479078/t/image-75903.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75903.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75905.webp: Open: flags=O_RDONLY
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75905.webp: >Open: fd=515266479078/t/image-75905.webp (r), err=<nil>
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75905.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:38 web01 bash[126803]: DEBUG : 515266479078/t/image-75906.webp: ChunkedReader.Read at 0 length 360 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75902.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75902.webp: >Open: fd=515266479078/t/image-75902.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75902.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75903.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75903.webp: ChunkedReader.Read at 512 length 220 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75904.webp: ChunkedReader.Read at 0 length 440 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75901.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75901.webp: >Open: fd=515266479078/t/image-75901.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75901.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75905.webp: ChunkedReader.Read at 0 length 350 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75900.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75900.webp: >Open: fd=515266479078/t/image-75900.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75900.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75899.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75899.webp: >Open: fd=515266479078/t/image-75899.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75899.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75902.webp: ChunkedReader.Read at 0 length 442 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75907.webp: ChunkedReader.Read at 0 length 382 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75908.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75908.webp: ChunkedReader.Read at 512 length 48 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75901.webp: ChunkedReader.Read at 0 length 496 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75898.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75898.webp: >Open: fd=515266479078/t/image-75898.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75898.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75897.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75897.webp: >Open: fd=515266479078/t/image-75897.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75897.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75896.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75896.webp: >Open: fd=515266479078/t/image-75896.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75896.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75910.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75910.webp: >Open: fd=515266479078/s/image-75910.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75910.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75896.webp: ChunkedReader.Read at 0 length 390 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75909.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75909.webp: >Open: fd=515266479078/s/image-75909.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75909.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75899.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75899.webp: ChunkedReader.Read at 512 length 266 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75897.webp: ChunkedReader.Read at 0 length 438 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75908.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75908.webp: >Open: fd=515266479078/s/image-75908.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75908.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75909.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75909.webp: ChunkedReader.Read at 512 length 24734 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75907.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75907.webp: >Open: fd=515266479078/s/image-75907.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75907.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75910.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75910.webp: ChunkedReader.Read at 512 length 13186 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75908.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75908.webp: ChunkedReader.Read at 512 length 17812 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75898.webp: ChunkedReader.Read at 0 length 370 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75906.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75906.webp: >Open: fd=515266479078/s/image-75906.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75906.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75905.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75905.webp: >Open: fd=515266479078/s/image-75905.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75905.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75906.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75906.webp: ChunkedReader.Read at 512 length 14252 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75907.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75907.webp: ChunkedReader.Read at 512 length 17324 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75895.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75895.webp: >Open: fd=515266479078/t/image-75895.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75895.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75900.webp: ChunkedReader.Read at 0 length 398 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75894.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75894.webp: >Open: fd=515266479078/t/image-75894.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75894.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75893.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75893.webp: >Open: fd=515266479078/t/image-75893.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75893.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75892.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75892.webp: >Open: fd=515266479078/t/image-75892.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75892.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75894.webp: ChunkedReader.Read at 0 length 390 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75895.webp: ChunkedReader.Read at 0 length 344 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75905.webp: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/s/image-75905.webp: ChunkedReader.Read at 512 length 13444 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75891.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75891.webp: >Open: fd=515266479078/t/image-75891.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75891.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75893.webp: ChunkedReader.Read at 0 length 374 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75890.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75890.webp: >Open: fd=515266479078/t/image-75890.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75890.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75889.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75889.webp: >Open: fd=515266479078/t/image-75889.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75889.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75888.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75888.webp: >Open: fd=515266479078/t/image-75888.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75888.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75887.webp: Open: flags=O_RDONLY
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75887.webp: >Open: fd=515266479078/t/image-75887.webp (r), err=<nil>
Apr 10 11:09:39 web01 bash[126803]: DEBUG : 515266479078/t/image-75887.webp: ChunkedReader.openRange at 0 length 134217728
Apr 10 11:09:40 web01 bash[126803]: DEBUG : 515266479078/t/image-75890.webp: ChunkedReader.Read at 0 length 358 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:40 web01 bash[126803]: DEBUG : 515266479078/t/image-75891.webp: ChunkedReader.Read at 0 length 360 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:40 web01 bash[126803]: DEBUG : 515266479078/t/image-75889.webp: ChunkedReader.Read at 0 length 336 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:40 web01 bash[126803]: DEBUG : 515266479078/t/image-75888.webp: ChunkedReader.Read at 0 length 396 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:40 web01 bash[126803]: DEBUG : 515266479078/t/image-75887.webp: ChunkedReader.Read at 0 length 370 chunkOffset 0 chunkSize 134217728
Apr 10 11:09:40 web01 bash[126803]: DEBUG : 515266479078/t/image-75892.webp: ChunkedReader.Read at 0 length 398 chunkOffset 0 chunkSize 134217728
Please note that the requests for the images arrived at the webserver at Apr 10 11:08:59, but it takes rclone around 30sec before even trying to download the images from the B2 backend.
I suspect rclone is trying to get a list of the directory content first, but since there's nothing in the logs during that time, I can't verify that.
And once rclone starts downloading the files, it still takes more than 10sec to get them all.
I also tried with caching enabled:
--vfs-cache-mode full --vfs-cache-size 50G
And while this has a positive impact on consecutive request, it still has several issues:
- the initial request is still unacceptably slow
- since the backend doesn't support polling the data can't be cached for long or I start receiving 404's whenever I try to retrieve a newly added image.
So this is not a solution.
When I load the images via a Cloudflare worker, the images load almost instantly so it seems related to rclone (cfr listing dir content before retrieval?).
Unfortunately the Cloudflare "workaround" isn't a solution either in my case because of other issues.