What is the problem you are having with rclone?
I have an S3 bucket with huge number of small files (~10 millions) and I try to create a single archive file of them using rclone archive create. I need this to speed up downloading the whole bucket locally.
The problem is that it lists objects very slow ~100 items per second. For now the command have been working for 24 hours and it listed only 5 million files.
rclone size s3:bucket-name takes only 9 minutes to count all 12 million files.
Run the command 'rclone version' and share the full output of the command.
rclone v1.72.1
- os/version: freebsd 13.1-release-p9 (64 bit)
- os/kernel: 13.1-release-p9 (amd64)
- os/type: freebsd
- os/arch: amd64
- go/version: go1.25.5
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3 (other provider)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone archive create s3:bucket-name /mnt/zfs/copy-of-bucket.tar.gz --transfers 16 --checkers 16 --use-mmap --s3-chunk-size 128M --fast-list --
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[s3]
type = s3
provider = other
access_key_id = XXX
secret_access_key = XXX
endpoint = other.cloud.provider:443
A log from the command that you were trying to run with the -vv flag
2026/01/10 23:08:50 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "archive" "create" "s3:bucket-name" "/mnt/zfs/copy-of-bucket.tar.gz" "--transfers" "16" "--checkers" "16" "--use-mmap" "--s3-chunk-size" "128M" "--fast-list" "--s3-list-chunk" "1000" "-vvv" "--progress"]
2026/01/10 23:08:50 DEBUG : Creating backend with remote "s3:bucket-name"
2026/01/10 23:08:50 DEBUG : Using config file from "/mnt/zfs/scripts/rclone.conf"
2026/01/10 23:08:50 DEBUG : s3: detected overridden config - adding "{OgfZc}" suffix to name
2026/01/10 23:08:50 DEBUG : fs cache: renaming cache item "s3:bucket-name" to be canonical "s3{OgfZc}:bucket-name"
2026/01/10 23:08:50 DEBUG : Creating backend with remote "/mnt/zfs/"
2026/01/10 23:08:50 DEBUG : fs cache: renaming cache item "/mnt/zfs/scripts/" to be canonical "/mnt/zfs/scripts"
dst=Local file system at /mnt/zfs/scripts, dstFile="copy-of-bucket.tar.gz", src=S3 bucket bucket-name, format="", prefix=""
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 0 / 0, -, Listed 600
Elapsed time: 5.8s