# S3 ListObjects\[V2\] performance optimization for a flat structures

**URL:** <https://forum.rclone.org/t/s3-listobjects-v2-performance-optimization-for-a-flat-structures/49860>\
**Category:** Feature\
**Created:** [February 4, 2025, 6:07pm UTC](https://forum.rclone.org/t/s3-listobjects-v2-performance-optimization-for-a-flat-structures/49860 "2025-02-04T18:07:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![alxfv](https://forum.rclone.org/user_avatar/forum.rclone.org/alxfv/32/19304_2.png) [@alxfv](https://forum.rclone.org/u/alxfv)\
**Post date:** [February 4, 2025, 6:07pm UTC](https://forum.rclone.org/t/s3-listobjects-v2-performance-optimization-for-a-flat-structures/49860/1 "2025-02-04T18:07:41Z")

</div>

**Motivation**

Imagine you have a bucket in s3 with millions keys under the same prefix:

```plaintext
source-bucket/a/b/c/2020-01-01-10-00-00.log
source-bucket/a/b/c/2020-01-01-10-00-01.log
source-bucket/a/b/c/2020-01-01-10-00-02.log
... 
source-bucket/a/b/c/2022-12-31-10-00-00.log
...
source-bucket/a/b/c/2024-12-31-10-00-00.log

```

You want to copy all the objects for Dec 2022. You need to filter keys by prefix "2022-12-".

Rclone doesn't support a path as a prefix for keys:

```auto
rclone copy my-s3-source-remote:source-bucket/a/b/c/2022-12- my-destination-remote:destination-bucket/d/e/f

```

We can use files matching pattern as a possible solution:

```auto
rclone copy my-s3-remote:bucket/prefix my-destination-remote:destination-bucket/d/e/f --include "2022-12-*"

```

With this approach Rclone sends [ListObjectsV2](https://github.com/rclone/rclone/blob/v1.69.0/backend/s3/s3.go#L4180) with `a/b/c` as a prefix. In the result Rclone will **list all millions of keys under the prefix** and do the filtering later.

Optimized way is to send ListObjectsV2 request with a `prefix` equals to [directory]/[key-prefix]: `a/b/c/2022-12-`. With this approach **S3 returns only keys starting with `a/b/c/2022-12-` bypassing unnecessary keys**.

"--fast-list" flag doesn't help with flat structure.

**Proposal**

Add a new flag `--s3-list-key-prefix` that modifies the ListObjectsV2 (and ListObjects) request to include a server-side filter for keys. This significantly improve performance for a flat structures.

**Implementation**

[draft] [Objects list performance optimization for a flat structures in S3 by alxfv · Pull Request #1 · alxfv/rclone · GitHub](https://github.com/alxfv/rclone/pull/1/files)

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [February 4, 2025, 6:22pm UTC](https://forum.rclone.org/t/s3-listobjects-v2-performance-optimization-for-a-flat-structures/49860/2 "2025-02-04T18:22:06Z")

</div>

I actually am in the process of fixing this using an out of memory sync routine for rclone which should be seamless.

Check out this issue: [Excess memory use when syncing millions of files in one directory · Issue #7974 · rclone/rclone · GitHub](https://github.com/rclone/rclone/issues/7974)

I just posted a binary for people to try.

---

<div class="post-metadata">

**Author:** ![alxfv](https://forum.rclone.org/user_avatar/forum.rclone.org/alxfv/32/19304_2.png) [@alxfv](https://forum.rclone.org/u/alxfv)\
**Post date:** [February 4, 2025, 8:56pm UTC](https://forum.rclone.org/t/s3-listobjects-v2-performance-optimization-for-a-flat-structures/49860/3 "2025-02-04T20:56:47Z")

</div>

@ncw thank you for the quick reply! My case is similar in the nature of data ("100 million files at the root. They are mostly small files \< 1mb."), but instead of copying all of the files I need to copy \<1% of them. To do this fast I believe there is only one option: filter keys in the ListObjects[V2] method. Using this method we filter files by prefix almost instantly: it's incredibly fast operation for S3/Ceph/...

[I've created an draft implementation](https://github.com/alxfv/rclone/pull/2/files) based on your branch as an example.

I've tested your branch on a bucket with millions of files in a single directory. Copying of 5000 files takes ~10 sec using filtering in ListObjectsV2 and minutes without it (99% of times spent on ListObjectsV2).

---

<div class="post-metadata">

**Author:** ![system](https://forum.rclone.org/uploads/default/original/2X/3/39ad3235f6ffec76ccd0c81616bd55697b4c300d.png) [@system](https://forum.rclone.org/u/system)\
**Post date:** [April 5, 2025, 8:57pm UTC](https://forum.rclone.org/t/s3-listobjects-v2-performance-optimization-for-a-flat-structures/49860/4 "2025-04-05T20:57:21Z")

</div>

Closed on April 5, 2025, 8:57pm UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
