Cut S3/B2 listing time and transaction cost with --fast-list (when it helps, when it hurts)

Howto Guides
1

A tip that saved me real time (and transaction cost) on large object-store remotes — S3, B2, GCS, etc.

By default rclone lists a bucket directory-by-directory: roughly one API call per prefix. On a remote with thousands of prefixes that's thousands of List calls — slow, and on B2/S3 each one is a billable class-B/list transaction.

--fast-list flips this to a single recursive listing: rclone pulls the object list in as few calls as the backend allows, builds the tree in memory, then works from that. On a bucket I rclone size nightly this dropped the run from ~40s and well over a thousand list transactions to a few seconds and a handful of calls.

When it helps

  • Bucket / object-store backends (s3, b2, gcs, swift, azureblob) that have many directories.
  • Operations that need the whole tree anyway: sync, size, dedupe, check, lsf -R.

When it hurts (or does nothing)

  • Backends that can't do a cheap recursive list — rclone effectively ignores it and you see no speedup.
  • Huge flat buckets on a small box: it holds the entire listing in RAM. Millions of objects = real memory; watch RSS before you bake it into a cron job.

Gotchas

  • It changes listing, not transfer — pair it with --checkers / --transfers tuning if throughput (not enumeration) is your bottleneck.
  • To make it permanent without editing every command, set RCLONE_FAST_LIST=true in the environment — rclone maps every global flag to an RCLONE_* env var, which keeps cron/systemd units clean.

Quick before/after to measure the win on your own remote — run each with -vv and compare the transaction/time lines in the log:

rclone size remote:bucket -vv
rclone size remote:bucket --fast-list -vv
2

welcome to the forum,

depending on use-case another option that does not rely on env var
add override.fast_list = false to remote in the config file.

rclone config redacted remote:
[remote]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
override.fast_list = true

rclone lsd remote: -vv
DEBUG : rclone: Version "v1.74.4" starting with parameters ["rclone" "lsd" "remote:" "-vv"]
DEBUG : Creating backend with remote "remote:"
DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
DEBUG : remote: Set overridden config ["fast_list"] for backend startup

some providers do not charge for api calls.
for example, wasabi and idrive

3

On s3-like backends this is always cheaper but isn't always faster. Depending on the exact directory structure rclone can list with --checkers threads at once which can be much quicker. This will likely use more transactions though. --fast-list does use extra memory though which is why it isn't the default.