Rclone S3 Sync - Only 1000 folders synced

Help and Support
1

What is the problem you are having with rclone?

I have to migrate a large S3 bucket (150 TB) from a storage appliance that uses MinIO as their server (I cann tell from the Server header). The destination is a self-built MinIO S3 server pool with 4 nodes.
The problem I see is that the sync completes successfully, but only a subset of the data is migrated (16.5 TiB - 7.3M objects). The folder I suspect the most of the data in has around 10K subfolders.
However, when I list the content of this folder, I receive only 1000 results, on source, and after the sync on the destination.

rclone lsf --dirs-only source-host:/path/to/folder/ | wc -l
1000

rclone lsf --dirs-only dest-host:bucket/path/to/folder/ | wc -l
1000

(The bucket is set automatically at the source as configured for the user, therefore not defined)

I read about a S3 API limit of 1000 objects, could this be the reason? Is there no paging?

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: debian 12.6 (64 bit)
  • os/kernel: 6.1.0-22-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Both MinIO, source is an closed appliance, destination is a self managed MinIO server pool with 4 nodes.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -P --no-check-certificate sync source-host: dest-host:bucket --transfers=30 --checkers=40 --size-only

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[dest-host]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://dest-host
acl = private

[source-host]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://source-host:1234
acl = private

A log from the command that you were trying to run with the -vv flag

Ne error shown. The command completes successfully after no changes needed. I started for a moment with -vv but this is much output and shows for all files two lines:
1: Sizes identical
2: Unchanged skipping

Thanks in advance for your help!

2

Add the following parameters (it fetches up to 8M objects):
--s3-list-chunk 8000000

3

Thanks.
I tried that with the lsf command, but I still receive only the exact 1000 objects.

I verified the request with --dump-headers and see that &max-keys=8000000 is part of the request.

4

I found that the ListObjects v1 API was used that apparently does not support Continuation.
Once I add --s3-list-version 2 I receive more than 1000 objects. So far this response is still incomplete, but dependent on --s3-list-chunk size I receive a different amount.
From what I see, that is an issue with the S3 server, the rclone Client side looks fine.

5

This is the problem - set this correctly to "Minio" and it should all work properly.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.