Tuning performance from M3 Mac to Hetzner StorageBox

Help and Support
1

What is the problem you are having with rclone?

Summary: Slow upload speeds to Hetzner StorageBox (Finland) from West Coast of the US compared to rsync.

Details: I have been backing up my M3 Macbook to my Hetzner StorageBox with rsync. I manage to get ~10MB/s from the West Coast of the US to the server in Finland. I can get ~500Mbps to Finnish speed test servers so I was looking for ways to get something faster.

I heard about rclone, installed it via Homebrew and have been working with it. The stats bar shows up to 30MBps for large files but for small files, it slows to well under 1Mbps.

I read the documentation, consulted websites, and tried to tune performance on my own. I have pasted an example in the template below. A 7.1GB folder of pictures (lots of small files) took 46 mins to send for ~2.5MBps which is a lot slower than rsync. I selected options that are said to help for lots of small files.

Overall, an rsync backup of ~130GB of my home folder takes ~3.5 hours. I tried with rclone (including tuning for small files) and it was still going after 8 hours.

Other:

  • I searched similar threads here and think I’ve enabled options that help speed up small files
  • I tried rcloneui (found while searching on my issue). It does not find my remote drive. The documentation that I could find is too sparse to troubleshoot further. I know this is a separate project - this is just for due diligence
  • I tried mounting the Storage Box with rclone mount but that isn’t supported on the Homebrew installation
    • I downloaded the suggested package but it is a folder of files and an executable with no installation instructions
  • I am aware that performance to Finland from the US is subject to the laws of physics. It seems that there may be some room for improvement within that bound.

What am I looking for?:

  • Tips to match or hopefully exceed rsync performance. There are so many options and combinations that it would take too long to find the right mix by chance
  • Honest advice if rclone may not be the tool for my needs. I’d rather not do a mix of rsync and rclone

Run the command 'rclone version' and share the full output of the command.


% rclone version
rclone v1.73.4
- os/version: darwin 15.7.5 (64 bit)
- os/kernel: 24.6.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.26.2
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Hetzner StorageBox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

```
time rclone copy -P ~/Pictures storagebox:/home/test/Pictures/  \
--transfers 32 \
--checkers 16 \
--tpslimit 12 \
--fast-list \ 
--stats-one-line \ 
--drive-chunk-size 128M 
```

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.


[storagebox]
type = sftp
host = XXX
user = XXX
port = 23
key_use_agent = true
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag
2

welcome to the forum,

rclone computes checksums for each file transferred.
to do this, rclone has to ask the hetzner server to run md5sum.
a rclone debug log would show that.

that flag does nothing, as sftp does not support ListR
https://rclone.org/overview/#optional-features

that flag does nothing as that is for google drive.
for sftp, chunk size is controlled by --sftp-chunk-size

i have been using storagebox sftp for 6+ years, never needed to limit rclone that way?

3

Thank you for the welcome and for your suggestions. I am not sure how to quote and excerpt your reply so I'll summarize as best as I can.

Switches: There is a lot of potentially wrong information out there. I got the small file optimization switches from
Why is rclone slow? | Rclone CLI as well as a couple of other sites whose URLs I've lost. Free ChatGPT (for whatever that's worth) gave me same invocations.

-vv log: I am rerunning my command for the 7.1GB transfer (copy not sync) with -vv and tee-ing it to a file. It will take a long time to finish and there are already over 26k lines. I have no idea how I’d sanitize the ultimate file when it is done. Filenames have a lot of personally identifiable information.

  • I do see md5 commands run on the Storage Box. I am sure this is slowing things down since the server is shared and the computation intensive. I am open to suggestions

Hetzner: Glad to know you are successfully using rclone with a Storage Box. Can you share your generic location information (e.g. from North America/Europe/Asia to Hetzner Germany/Finland), speeds achieved, and/or any switches you use or recommend?

4

fwiw, i tend to use default values and not add random flags with random values.
and read the debug log

the issue is the large amount of time spent by hetzner computing the md5 hash.

for sure, hetzner is rate-limiting cpu usage, disk usage, etc..
and slamming too many simultaneous md5sum commands is not going to help

5

I would appreciate any pointers to my options regarding the md5 hash. I am doing what I can to sift through documentation here and elsewhere. I’ll look for a switch to disable it but I’m sure there are tradeoffs to doing that.

flags/switches: I didn’t pick these at random. I searched for information on how to improve speed and a lot of sources gave this information.

debug log: I’ll definitely read it when it is done. 65k lines and growing. If there’s a pointer to what to look for and how to interpret it, please let me know.

Hetzner rate limiting: Just saw your update about Hetzner limiting the usage. May I ask what kind of speeds you achieve to your Storage Box and if you disable the md5?

6

i would never disable that.
i only use services that provide file verification using checksums. sftp, s3, etc...

if checksums are not important for your backups,
then might try webdav, which should have less round-trip latency.

i could be wrong but storagebox provides a rsync server?
if so, not fair to compare that to rclone and expect similar results.

for my backup solution, for linux and windows, in addition to rclone,restic and so on, i also use rsync without a rsync server running on the dest.
so i use --checksum --whole-file which is more comparable to rclone

7

Ok, since these are backups I see your point about not disabling the checksum.
FWIW, I mounted the Storage Box to Finder using both smb and https. I tried copy/pastes and they were very slow with little diagnostics. I did not try rclone with them though.

Storage Box utils: help from an ssh to the Storage Box gives (excerpt)

Available as server side backend:                                            |
|   borg                                                                       |
|   rsync                                                                      |
|   scp                                                                        |
|   sftp                                                                       |
|   rclone serve restic --stdio

I am not savvy enough to know if this a fair comparison or not. Hetzner said it supports rclone so I am trying it.

May I conclude that for my use case (laptop backup with a mix of large, medium, and small files) to a shared Hetzner resource that I should stick with rsync?

8

that is up to you.

let's see if any other forum members stop by and their input

9

as per hetzner documentation,
those are not the recommended values and can cause problems.

that is why you need to read the rclone debug log.

10

Thanks. I think I am not experienced enough to get the most out of rclone. I have collected logs as suggested but can’t find information on how to interpret them (earlier request).

I’ll stick with rsync.