What is the problem you are having with rclone?
I have to migrate a large S3 bucket (150 TB) from a storage appliance that uses MinIO as their server (I cann tell from the Server header). The destination is a self-built MinIO S3 server pool with 4 nodes.
The problem I see is that the sync completes successfully, but only a subset of the data is migrated (16.5 TiB - 7.3M objects). The folder I suspect the most of the data in has around 10K subfolders.
However, when I list the content of this folder, I receive only 1000 results, on source, and after the sync on the destination.
rclone lsf --dirs-only source-host:/path/to/folder/ | wc -l
1000
rclone lsf --dirs-only dest-host:bucket/path/to/folder/ | wc -l
1000
(The bucket is set automatically at the source as configured for the user, therefore not defined)
I read about a S3 API limit of 1000 objects, could this be the reason? Is there no paging?
Run the command 'rclone version' and share the full output of the command.
rclone v1.67.0
- os/version: debian 12.6 (64 bit)
- os/kernel: 6.1.0-22-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Both MinIO, source is an closed appliance, destination is a self managed MinIO server pool with 4 nodes.
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone -P --no-check-certificate sync source-host: dest-host:bucket --transfers=30 --checkers=40 --size-only
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[dest-host]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://dest-host
acl = private
[source-host]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://source-host:1234
acl = private
A log from the command that you were trying to run with the -vv
flag
Ne error shown. The command completes successfully after no changes needed. I started for a moment with -vv but this is much output and shows for all files two lines:
1: Sizes identical
2: Unchanged skipping
Thanks in advance for your help!