What is the problem you are having with rclone?
Hello, I've currently subscribed to a BX31 plan with Hetzner Storage Box https://www.hetzner.com/storage/storage-box/bx31, as you maybe aware, storage boxes allow for rclone via sftp on port 23
The problem seems to be that when I perform a speed test using rsync, scp, and even 'sftp put' commands via ssh, the speeds are 50x faster than when i use rclone copy and when it is mounted (via sftp).
I've been searching today on the forum for others who use hetzner storage boxes, but couldn't find any copy or mount parameters that could possibly help me increase speed or see others with slow speeds as mine (upload seems to be limited to 1.5-2 MByte/s). I am looking for suggestions on how I can potentially improve speeds via rclone, or am I somehow dealing with encryption overhead via rclone sftp? that doesn't quite make sense to me since I can use 'sftp put' with fast upload speeds. rsync seems to work the fastest for me with hetzner storage box.
Run the command 'rclone version' and share the full output of the command.
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18.1
Which cloud storage system are you using? (eg Google Drive)
Hetzner Storage Box BX31 (Germany, FSN1)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone --progress --sftp-disable-hashcheck=true copy 1G storagebox:test/
Transferred: 1G / 1 GBytes, 100%, 2.187 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 7m48.5s
vs
sftp> put 1G
Uploading 1G to /home/test/1G
1G 100% 1024MB 142.6MB/s 00:07
scp -P 23 1G uxxxxxx@uxxxxxx.your-storagebox.de:test/
1G 100% 1024MB 109.6MB/s 00:09
rsync 1G -vrplogDtH --progress -e 'ssh -p 23' "uxxxxxx@uxxxxxx.your-storagebox.de:test/"
sending incremental file list
1G
1,073,741,824 100% 142.24MB/s 0:00:07 (xfr#1, to-chk=0/1)
sent 1,074,004,064 bytes received 35 bytes 143,200,546.53 bytes/sec
total size is 1,073,741,824 speedup is 1.00
[below is are example speeds of downloading via sftp, scp, rsync. but i am not focusing on troubleshooting download speeds atm]
sftp> get 1G
Fetching /home/test/1G to 1G
1G 100% 1024MB 33.8MB/s 00:30
scp -P 23 uxxxxxx@uxxxxxx.your-storagebox.de:test/1G .
1G 100% 1024MB 30.4MB/s 00:33
rsync -vrplogDtH --progress -e 'ssh -p 23' "uxxxxxx@uxxxxxx.your-storagebox.de:test/1G" .
receiving incremental file list
1G
1,073,741,824 100% 664.94MB/s 0:00:01 (xfr#1, to-chk=0/1)
sent 229,431 bytes received 131,187 bytes 37,959.79 bytes/sec
total size is 1,073,741,824 speedup is 2,977.50
The rclone config contents with secrets removed.
[storagebox]
type = sftp
host = uxxxxxx.your-storagebox.de
user = uxxxxxx
port = 23
shell_type = unix
key_file = ~/.ssh/id_rsa
md5sum_command = md5sum
sha1sum_command = sha1sum
rclone mount storagebox: /home/hd31/xxxx/mnt/sb/ --allow-other
A log from the command with the -vv flag
rclone -vv --progress --sftp-disable-hashcheck=true copy 1G storagebox:test/
2023/03/13 21:27:40 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vv" "--progress" "--sftp-disable-hashcheck=true" "copy" "1G" "storagebox:test/"]
2023/03/13 21:27:40 DEBUG : Creating backend with remote "1G"
2023/03/13 21:27:40 DEBUG : Using config file from "/home/hd31/xxxx/.config/rclone/rclone.conf"
2023/03/13 21:27:40 DEBUG : fs cache: adding new entry for parent of "1G", "/home/hd31/xxxx/xxxx"
2023/03/13 21:27:40 DEBUG : Creating backend with remote "storagebox:test/"
2023/03/13 21:27:40 DEBUG : sftp://uxxxxxx@uxxxxxx.your-storagebox.de:23/test/: New connection 185.162.x.x:54608->162.55.x.x:23 to "SSH-2.0-OpenSSH_8.8 FreeBSD-20211221"
2023/03/13 21:27:40 DEBUG : sftp://uxxxxxx@uxxxxxx.your-storagebox.de:23/test/: Using absolute root directory "/home/test"
2023-03-13 21:27:40 DEBUG : 1G: Need to transfer - File not found at Destination
2023-03-13 21:36:20 INFO : 1G: Copied (new)
Transferred: 1G / 1 GBytes, 100%, 1.970 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 8m40.0s
2023/03/13 21:36:20 INFO :
Transferred: 1G / 1 GBytes, 100%, 1.970 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 8m40.0s
2023/03/13 21:36:20 DEBUG : 12 go routines active