S3 remote -> S3 remote copy is slow

Help and Support
1

Hi

rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5

I’m copying (not syncing) 200Gb (3.6M files) from AWS to Wasabi. The data is directories of images - images are between 80K and 300K, and about 10-30 images per directory.

rclone copy -c -u -P S3:rover.images/sefrets Wasabi:rover.images/sefrets

It’s taken 40 hours so far, and it’s only copied 1/4 of the files. I suspect that I’m being throttled - perhaps by AWS or maybe Wasabi.

My question is - is there a way to tell who’s slowing me down?

Thank you
Steve

Ps - I have not tried --s3-upload-concurrency and --transfers . Does --transfers 8 yet because it feels like rclone is pretty well tuned by default.

2

-c is a good choice here to avoid metadata reads.

I would use a larger --transfers though - that should run quicker.

That will only win you on multipart transfers over the upload cutoff which is 200MB I think so probably won’t help.

Try 32 here - that shouldn’t be a problem.

3

Thank you - the support I’ve received on this forum is exceptional!

1 Like
4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.