What is the problem you are having with rclone?
In a 1Gbps network environment, I'm using rclone copy files from a NFS-mounted directory to a Ceph S3 bucket. The speed is only around 35MB/s. However, when I mount the Ceph S3 bucket to the same client machine as an NFS directory and use cp directly to copy files, the speed is approximately 90MB/s. And use rclone copy a local dir file to Ceph S3 bucket, the speed is also approximately 70MB/s.
Run the command 'rclone version' and share the full output of the command.
rclone v1.66.0
- os/version: CentOS 8 (64 bit)
- os/kernel: 4.19.90-25.10.aarch64 (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: static
- go/tags: none
Which cloud storage system are you using?
Ceph S3
The command you were trying to run
rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso ceph37:mybucket/ --progress --s3-disable-checksum -vv
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[ceph37]
type = s3
access_key_id = XXX
endpoint = http://X.X.X.X:8090
provider = Ceph
secret_access_key = XXX
A log from the command that you were trying to run with the -vv flag
rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso ceph37:mybucket/ --progress --s3-disable-checksum -vv
2024/05/08 09:28:01 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso" "ceph37:mybucket/" "--progress" "--s3-disable-checksum" "-vv"]
2024/05/08 09:28:01 DEBUG : Creating backend with remote "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso"
2024/05/08 09:28:01 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/05/08 09:28:01 DEBUG : fs cache: adding new entry for parent of "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso", "/remote/ns1"
2024/05/08 09:28:01 DEBUG : Creating backend with remote "ceph37:mybucket/"
2024/05/08 09:28:01 DEBUG : ceph37: detected overridden config - adding "{1SSjr}" suffix to name
2024/05/08 09:28:01 DEBUG : Resolving service "s3" region "us-east-1"
2024/05/08 09:28:01 DEBUG : fs cache: renaming cache item "ceph37:mybucket/" to be canonical "ceph37{1SSjr}:mybucket"
2024/05/08 09:28:01 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Need to transfer - File not found at Destination
2024/05/08 09:28:01 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: disabling buffering because source is local disk
2024/05/08 09:28:02 INFO : S3 bucket mybucket: Bucket "mybucket" created with ACL ""
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: open chunk writer: started multipart upload: 2~7PlrmybwyWPoTKRkUmkMJAgKK4NUJAC
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Starting multi-thread copy with 2151 chunks of size 5Mi with 4 parallel streams
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 4/2151 (15728640-20971520) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 3/2151 (10485760-15728640) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 1/2151 (0-5242880) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 2/2151 (5242880-10485760) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multipart upload wrote chunk 4 with 5242880 bytes and etag "0d895e9eaa21e337bca9a3fa3e14ba48"
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 4/2151 (15728640-20971520) size 5Mi finished
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 5/2151 (20971520-26214400) size 5Mi starting
.......
2024/05/08 09:30:47 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 116/2151 (602931200-608174080) size 5Mi finished
Transferred: 585 MiB / 10.502 GiB, 5%, 35.626 MiB/s, ETA 4m45s
Transferred: 0 / 1, 0%
Elapsed time: 16.6s
Transferring:
* CentOS-Stream-8-x86_64-20220328-dvd1.iso: 5% /10.502Gi, 35.937Mi/s, 4m4
