Cannot sync AWS S3 with GCP

What is the problem you are having with rclone?

I'm trying to move an s3 bucket to GCP with Rclone. Unfortunately my command always fails due to the OOM killer.

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.0

  • os/version: debian 11.11 (64 bit)
  • os/kernel: 5.10.0-33-cloud-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

The source storage is AWS S3 and the destination bucket is in GCP.

The S3 bucket has Total objects: 4.825M (4825249) and Total size: 97.543 GiB (104736162700 Byte).

I tried to make the sync work by allocating 8G to the VM but it wasn't enough. I also tried the same command with 16 and 32G to no avail.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo -u rclone rclone --config /home/rclone/rclone.conf sync --combined /tmp/output.txt --checksum --progress --transfers=32 --use-mmap s3:source gcs:dest

Please run 'rclone config redacted' and share the full output.

[gcs]
type = google cloud storage
project_number = XXX

[s3]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = eu-central-1
endpoint = bucket.vpce-XXX-XXX.s3.eu-central-1.vpce.amazonaws.com

A log from the command that you were trying to run with the -vv flag

(I will add the log once the command is finished)

Paste  log here

Do you have any recommendation on how to sync 2 buckets without rclone blowing up its memory?

I also tried with a single transfer but it's too slow.

Good timing:) Potential fix for such scenario has been being tested right now.

Try it yourself and provide feedback if it works:

PS.

You can also try to lower this value, even to the default 4.

The new version is working well, it uses a lot less memory than the previous one!

But it's not very fast, it needed 1 hour to sync 450k files even with --transfers=32. For now it uses only 3G of memory and almost no CPU (files have already been copied).

After 1 hour it started to sync some new files and the CPU usage is up but not the memory usage.

It looks good!

Good to hear that it works.

Please provide your feedback on github issue. It is very valuable to have some real life tests.

You could try to increase number of checkers: --checkers 32 (??). Default is 8.