AWS S3 to Azure transfer speeds low (30mb)

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

  • I have 60,000 video files in S3 that I have to copy ot Azure.
  • They range in size form a few MB to a hundred GB.
  • I have split them into 4 lists according to size and feed those list to rclone using --files-from.
  • I am currently using an an AWS t3.medium instance (2 vcpu and 4GB RAM).
  • The plan is to move up to larger instance size as I saturate the CPU and RAM.

No matter what combinations of ---transfers, --azureblob-chunk-size and --azureblob-upload-concurrency I use the transfer always seems to settle at around 29 MiB/s. With a high transer rate it may start at 230 MiB/s to 250 MiB/s, but it will eventually dip to 29 MiB/s.

If I copy from S3 to local, then local to Aure, the download speed from S3 is about 124 MiB/s and my upload to Azure is 170 MiB/s.

So I'd expect S3 to Azure direct to be around the 120 MiB/s mark per file. For testing purposes I'm using 10 GB files and larger and doing a single transfer.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: debian 11.7 (64 bit)
- os/kernel: 5.10.0-23-cloud-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3 --> Azure Blob

The command you were trying to run (eg rclone copy /tmp remote:tmp)

According to this post 4Mi seems to be the sweet spot. I have tried 8, 16, 32 etc with minimal changes to the transfer speed.

Likewise I have tried several values for --azureblob-upload-concurrency up to 1024 with little difference

	rclone copy \
		--progress \
		--no-traverse \
		--use-mmap \
		--transfers 1 \
		--max-backlog=800 \
		--azureblob-chunk-size 4Mi \
		--azureblob-upload-concurrency 64 \
		--buffer-size=1G \
		--checksum \
		--fast-list \
		--log-file=/home/admin/xl.log \
		--log-level DEBUG \
		--files-from \
		/home/admin/transfer_lists/size_greater_than_10_GB.txt aws:video-sources  az:sources

RAM and CPU usage will start high and quickly trail off. On a single upload I can't get anywhere near the capacity of the VM

The rclone config contents with secrets removed.

[aws]
type = s3
provider = AWS
env_auth = true
region = eu-west-1
location_constraint = eu-west-1
acl = private

[az]
type = azureblob
account = mediablob
env_auth = true

A log from the command with the -vv flag

There is nothing untoward in the logs at DEBUG level

2

I wont comment on your rclone copy command as I have limited experience with Azure but I think that issue is more general. Remote to remote at the moment does not utilize available parallelism to its full potential.

There is some activity to rectify it:

but these are early days.

3

OK good to know. I'll keep an eye on that MR. In the meantime I installed azcopy and it's hosing through it. For the record I really wanted azcopy not to work!

4

I'd guess this is some limitation on a per transfer speed somewhere.

The reason the transfer to local is quicker is that rclone is likely using multithreaded downloads so 4 download threads per file.

However if you've got lots of files you should be able to get that performance back by transferring multiple files at once.

Try setting --transfers 32 and also using --disable-http2 (there is a performance problem with Go and multiple streams of HTTP2) and transferring 32 1GB files at once.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.