What is the problem you are having with rclone?
When using rclone with a remote SFTP backend, there is a significant disadvantage in both CPU load and performance compared to the local backend.
The node running the rclone program, as well as the specified source and destination nodes, are all localhost.
| Soruce | Dest | Elapsed time |
|---|---|---|
| local | local | 2m35s |
| local | sftp | 6m46s |
| sftp | local | 9m6.0s |
| sftp | sftp | 17m21.3s |
Despite various attempts, the gap remains significantly large when source and dest both are sftp.
The most effective parameter I tried was --sftp-concurrency 1, which reduced the time to 11m10s.
Run the command 'rclone version' and share the full output of the command.
rclone v1.70.0-beta.8598.6560ea9bd
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-193-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
sftp
The command you were trying to run (eg rclone copy /tmp remote:tmp)
The source data is generated with the configuration: depth=1, width=6000, files=200, and size=400k
# rclone size /mnt/test/src
Total objects: 1.200M (1200001)
Total size: 457.764 GiB (491520012200 Byte)
# df -h | grep mnt
/dev/nvme0n1 7.0T 509G 6.5T 8% /mnt/test
/dev/nvme1n1 7.0T 459G 6.2T 7% /mnt/test2
rclone copy /mnt/test/src /mnt/test2/test/dir1/ --checkers 128 --transfers 64 -P
rclone copy /mnt/test/src sftp:/mnt/test2/test/dir1/ --checkers 128 --transfers 64 -P
rclone copy sftp:/mnt/test/src /mnt/test2/test/dir1/ --checkers 128 --transfers 64 -P
rclone copy sftp:/mnt/test/src sftp:/mnt/test2/test/dir1/ --checkers 128 --transfers 64 -P
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[sftp]
type = sftp
host = localhost_ip_address
user = root
key_file = /root/.ssh/id_rsa
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
A log from the command that you were trying to run with the -vv flag
Copy local to local
* vdb.1_990.dir/vdb_f197928.file: 0% /400Ki, 0/s, -Transferred: 457.764 GiB / 457.764 GiB, 100%, 3.068 GiB/s, ETA 0s
Transferred: 1200001 / 1200001, 100%
Elapsed time: 2m34.9s
Copy sftp to sftp
Elapsed time: 17m20.9sTransferred: 457.764 GiB / 457.764 GiB, 100%, 530.662 MiB/s, ETA 0s
Transferred: 1200001 / 1200001, 100%
Elapsed time: 17m21.3s