What is the problem you are having with rclone?
I see too low usage of resources and too low speed when using rclone.
I have a big EC2 instance and I'd like to use all resources that RClone can to provide maximum eff. I have c6in.32xlarge
with 200 Gigabit network bandwidth, 256.0 GiB of RAM and 128 vCPUs. So I'd like to utilize almost all of them and receive at less few gigabytes per second transfer speed. Currently I have speed ~400 MiB/s and top
utility says that rclone loads about 3-4 full CPUs (300-400 CPUs %). I'd like more!
We're transferring different files starting from 20-50-200 MBs and up to 40 GBs.
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1
- os/version: amazon 2 (64 bit)
- os/kernel: 4.14.301-224.520.amzn2.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3 => S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
NPROC="${_RCLONE_NPROC:-$(nproc --all)}" # 128 in my instance
BWLIMIT=${_RCLONE_BWLIMIT:-0}
TRANSFERS=${_RCLONE_TRANSFERS:-$((NPROC * 4))}
CHECKERS=${_RCLONE_CHECKERS:-$((NPROC * 4))}
UPLOAD_CONCURRENCY=${_RCLONE_UPLOAD_CONCURRENCY:-$((NPROC * 4))}
CHUNK_SIZE=${_RCLONE_CHUNK_SIZE:-256M}
CUTOFF_SIZE=${_RCLONE_CUTOFF_SIZE:-256M}
LOG_LEVEL=${_RCLONE_LOG_LEVEL:-NOTICE}
BUFFER_SIZE=${_RCLONE_BUFFER_SIZE:-256M}
MULTI_THREAD_CUTOFF=${_RCLONE_MULTI_THREAD_CUTOFF:-${CUTOFF_SIZE}}
MULTI_THREAD_STREAMS=${_RCLONE_MULTI_THREAD_STREAMS:-${NPROC}}
rclone sync \
"source:${SOURCE_BUCKET}/${SOURCE_OBJECT}" \
"target:${TARGET_BUCKET}/${TARGET_OBJECT}" \
--auto-confirm \
--create-empty-src-dirs \
--use-json-log \
--s3-disable-checksum \
--use-mmap \
--s3-memory-pool-use-mmap \
--order-by size,mixed,75 \
--max-backlog 10000 \
--buffer-size "${BUFFER_SIZE}" \
--bwlimit "${BWLIMIT}" \
--transfers "${TRANSFERS}" \
--checkers "${CHECKERS}" \
--s3-upload-concurrency "${UPLOAD_CONCURRENCY}" \
--s3-chunk-size "${CHUNK_SIZE}" \
--s3-upload-cutoff "${CUTOFF_SIZE}" \
--multi-thread-cutoff "${MULTI_THREAD_CUTOFF}" \
--multi-thread-streams "${MULTI_THREAD_STREAMS}" \
--config ./rclone.conf \
--log-level "${LOG_LEVEL}" \
--stats-log-level "${LOG_LEVEL}" \
--stats 10s
The rclone config contents with secrets removed.
[source]
type = s3
provider = AWS
env_auth = false
access_key_id = ${SOURCE_ACCESS_KEY_ID}
secret_access_key = ${SOURCE_SECRET_ACCESS_KEY}
session_token = ${SOURCE_SESSION_TOKEN}
region = ${SOURCE_REGION}
location_constraint = ${SOURCE_REGION}
sse_kms_key_id = ${SOURCE_BUCKET_SSE_KMS_KEY_ID}
server_side_encryption = ${SOURCE_BUCKET_SSE}
[target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${TARGET_ACCESS_KEY_ID}
secret_access_key = ${TARGET_SECRET_ACCESS_KEY}
session_token = ${TARGET_SESSION_TOKEN}
region = ${TARGET_REGION}
location_constraint = ${TARGET_REGION}
sse_kms_key_id = ${TARGET_BUCKET_SSE_KMS_KEY_ID}
server_side_encryption = ${TARGET_BUCKET_SSE}
A log from the command with the -vv
flag
"msg":"\nTransferred: \t 293.994 GiB / 365.499 GiB, 80%, 416.831 MiB/s, ETA 2m55s\nChecks: 62 / 62, 100%\nTransferred: 1 / 13, 8%