Poor copy/sync speed from a NFS-mounted directory to Ceph S3 bucket

What is the problem you are having with rclone?

In a 1Gbps network environment, I'm using rclone copy files from a NFS-mounted directory to a Ceph S3 bucket. The speed is only around 35MB/s. However, when I mount the Ceph S3 bucket to the same client machine as an NFS directory and use cp directly to copy files, the speed is approximately 90MB/s. And use rclone copy a local dir file to Ceph S3 bucket, the speed is also approximately 70MB/s.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: CentOS 8 (64 bit)
  • os/kernel: 4.19.90-25.10.aarch64 (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using?

Ceph S3

The command you were trying to run

rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso ceph37:mybucket/ --progress --s3-disable-checksum -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[ceph37]
type = s3
access_key_id = XXX
endpoint = http://X.X.X.X:8090
provider = Ceph
secret_access_key = XXX

A log from the command that you were trying to run with the -vv flag

rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso ceph37:mybucket/ --progress --s3-disable-checksum -vv
2024/05/08 09:28:01 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso" "ceph37:mybucket/" "--progress" "--s3-disable-checksum" "-vv"]
2024/05/08 09:28:01 DEBUG : Creating backend with remote "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso"
2024/05/08 09:28:01 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/05/08 09:28:01 DEBUG : fs cache: adding new entry for parent of "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso", "/remote/ns1"
2024/05/08 09:28:01 DEBUG : Creating backend with remote "ceph37:mybucket/"
2024/05/08 09:28:01 DEBUG : ceph37: detected overridden config - adding "{1SSjr}" suffix to name
2024/05/08 09:28:01 DEBUG : Resolving service "s3" region "us-east-1"
2024/05/08 09:28:01 DEBUG : fs cache: renaming cache item "ceph37:mybucket/" to be canonical "ceph37{1SSjr}:mybucket"
2024/05/08 09:28:01 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Need to transfer - File not found at Destination
2024/05/08 09:28:01 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: disabling buffering because source is local disk
2024/05/08 09:28:02 INFO  : S3 bucket mybucket: Bucket "mybucket" created with ACL ""
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: open chunk writer: started multipart upload: 2~7PlrmybwyWPoTKRkUmkMJAgKK4NUJAC
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Starting multi-thread copy with 2151 chunks of size 5Mi with 4 parallel streams
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 4/2151 (15728640-20971520) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 3/2151 (10485760-15728640) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 1/2151 (0-5242880) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 2/2151 (5242880-10485760) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multipart upload wrote chunk 4 with 5242880 bytes and etag "0d895e9eaa21e337bca9a3fa3e14ba48"
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 4/2151 (15728640-20971520) size 5Mi finished
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 5/2151 (20971520-26214400) size 5Mi starting
.......
2024/05/08 09:30:47 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 116/2151 (602931200-608174080) size 5Mi finished
Transferred:          585 MiB / 10.502 GiB, 5%, 35.626 MiB/s, ETA 4m45s
Transferred:            0 / 1, 0%
Elapsed time:        16.6s
Transferring:
 * CentOS-Stream-8-x86_64-20220328-dvd1.iso:  5% /10.502Gi, 35.937Mi/s, 4m4

welcome to the forum,

so, can we conclude that the issue is not with

  • NFS server
  • Ceph

that the issue is slow transfer between the remote NFS server and the machine rclone is running on

can test the speed between the remote NFS server and the machine that rclone is running on.
cp /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso /root/test
rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso /root/test

and might want to check out
https://rclone.org/commands/rclone_nfsmount/
https://rclone.org/commands/rclone_serve_nfs/

Thanks about your suggessions. I had already done some test of copying from nfs-mounted to local filesystem. The speed is about 85MB/s which seems ok.