Rclone copy is very slow when copying small files

Help and Support
1

I'm attempting to copy data from Alibaba Cloud OSS to AWS S3.
Most files are around 1KB in size
I'm dealing with around 10 billion objects, and the transfer speed is very slow — approximately 1 hour or more per GB. The intermediate server I'm using is an AWS c6in.4xlarge instance. Can anyone help explain why this is happening, and what settings or configurations can be adjusted to improve the transfer efficiency?

rclone config：
[oss]
type = s3
provider = Alibaba
access_key_id = xxx
secret_access_key = xxx
endpoint = oss-ap-southeast-1.aliyuncs.com
#acl = private
storage_class = STANDARD

#bucket_acl = public-read
#upload_cutoff = 1Ki
[s3]
type = s3
provider = AWS
access_key_id = xx
secret_access_key = xxx
region = ap-southeast-1
endpoint = s3.ap-southeast-1.amazonaws.com
#acl = private
storage_class = STANDARD

rclone commond

rclone copt oss:xxx s3:xxx --transfers 128 --checkers 256 --fast-list --size-only --use-server-modtime --buffer-size 1M --s3-upload-concurrency 32 --s3-chunk-size 1M --log-level INFO --progress

2

Adjusting the number of transfer or check operations directly has not resulted in a noticeable improvement. The instance utilization remains low, and the outbound network throughput is only about 1MB/s

3

There are no error messages in the logs. I'm running a test with approximately 1 million objects totaling 1GB in size. The transfer speed has been slower than expected, and I’m unsure what is limiting the overall sync performance. Any advice or assistance would be greatly appreciated.

4

root@ip-172-31-44-63:/home/ubuntu# rclone version
rclone v1.69.2

  • os/version: ubuntu 24.04 (64 bit)
  • os/kernel: 6.8.0-1024-aws (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.24.2
  • go/linking: static
  • go/tags: none
5

often, the answer can be found in the debug log.

is the machine running rclone in the same region as the bucket?

as a test, use default values.

rclone copy oss:xxx s3:xxx --fast-list --size-only --use-server-modtime  --log-level DEBUG --progress
6

The debug logs do not reveal any useful information. Apart from the program running normally, not specifying the number of transfers using this transfer parameter makes the transfer speed slower.