Tuning performance from M3 Mac to Hetzner StorageBox

Help and Support
1

What is the problem you are having with rclone?

Summary: Slow upload speeds to Hetzner StorageBox (Finland) from West Coast of the US compared to rsync.

Details: I have been backing up my M3 Macbook to my Hetzner StorageBox with rsync. I manage to get ~10MB/s from the West Coast of the US to the server in Finland. I can get ~500Mbps to Finnish speed test servers so I was looking for ways to get something faster.

I heard about rclone, installed it via Homebrew and have been working with it. The stats bar shows up to 30MBps for large files but for small files, it slows to well under 1Mbps.

I read the documentation, consulted websites, and tried to tune performance on my own. I have pasted an example in the template below. A 7.1GB folder of pictures (lots of small files) took 46 mins to send for ~2.5MBps which is a lot slower than rsync. I selected options that are said to help for lots of small files.

Overall, an rsync backup of ~130GB of my home folder takes ~3.5 hours. I tried with rclone (including tuning for small files) and it was still going after 8 hours.

Other:

  • I searched similar threads here and think I’ve enabled options that help speed up small files
  • I tried rcloneui (found while searching on my issue). It does not find my remote drive. The documentation that I could find is too sparse to troubleshoot further. I know this is a separate project - this is just for due diligence
  • I tried mounting the Storage Box with rclone mount but that isn’t supported on the Homebrew installation
    • I downloaded the suggested package but it is a folder of files and an executable with no installation instructions
  • I am aware that performance to Finland from the US is subject to the laws of physics. It seems that there may be some room for improvement within that bound.

What am I looking for?:

  • Tips to match or hopefully exceed rsync performance. There are so many options and combinations that it would take too long to find the right mix by chance
  • Honest advice if rclone may not be the tool for my needs. I’d rather not do a mix of rsync and rclone

Run the command 'rclone version' and share the full output of the command.


% rclone version
rclone v1.73.4
- os/version: darwin 15.7.5 (64 bit)
- os/kernel: 24.6.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.26.2
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Hetzner StorageBox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

```
time rclone copy -P ~/Pictures storagebox:/home/test/Pictures/  \
--transfers 32 \
--checkers 16 \
--tpslimit 12 \
--fast-list \ 
--stats-one-line \ 
--drive-chunk-size 128M 
```

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.


[storagebox]
type = sftp
host = XXX
user = XXX
port = 23
key_use_agent = true
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag
2

welcome to the forum,

rclone computes checksums for each file transferred.
to do this, rclone has to ask the hetzner server to run md5sum.
a rclone debug log would show that.

that flag does nothing, as sftp does not support ListR
https://rclone.org/overview/#optional-features

that flag does nothing as that is for google drive.
for sftp, chunk size is controlled by --sftp-chunk-size

i have been using storagebox sftp for 6+ years, never needed to limit rclone that way?

3

Thank you for the welcome and for your suggestions. I am not sure how to quote and excerpt your reply so I'll summarize as best as I can.

Switches: There is a lot of potentially wrong information out there. I got the small file optimization switches from
Why is rclone slow? | Rclone CLI as well as a couple of other sites whose URLs I've lost. Free ChatGPT (for whatever that's worth) gave me same invocations.

-vv log: I am rerunning my command for the 7.1GB transfer (copy not sync) with -vv and tee-ing it to a file. It will take a long time to finish and there are already over 26k lines. I have no idea how I’d sanitize the ultimate file when it is done. Filenames have a lot of personally identifiable information.

  • I do see md5 commands run on the Storage Box. I am sure this is slowing things down since the server is shared and the computation intensive. I am open to suggestions

Hetzner: Glad to know you are successfully using rclone with a Storage Box. Can you share your generic location information (e.g. from North America/Europe/Asia to Hetzner Germany/Finland), speeds achieved, and/or any switches you use or recommend?

4

fwiw, i tend to use default values and not add random flags with random values.
and read the debug log

the issue is the large amount of time spent by hetzner computing the md5 hash.

for sure, hetzner is rate-limiting cpu usage, disk usage, etc..
and slamming too many simultaneous md5sum commands is not going to help

5

I would appreciate any pointers to my options regarding the md5 hash. I am doing what I can to sift through documentation here and elsewhere. I’ll look for a switch to disable it but I’m sure there are tradeoffs to doing that.

flags/switches: I didn’t pick these at random. I searched for information on how to improve speed and a lot of sources gave this information.

debug log: I’ll definitely read it when it is done. 65k lines and growing. If there’s a pointer to what to look for and how to interpret it, please let me know.

Hetzner rate limiting: Just saw your update about Hetzner limiting the usage. May I ask what kind of speeds you achieve to your Storage Box and if you disable the md5?

6

i would never disable that.
i only use services that provide file verification using checksums. sftp, s3, etc...

if checksums are not important for your backups,
then might try webdav, which should have less round-trip latency.

i could be wrong but storagebox provides a rsync server?
if so, not fair to compare that to rclone and expect similar results.

for my backup solution, for linux and windows, in addition to rclone,restic and so on, i also use rsync without a rsync server running on the dest.
so i use --checksum --whole-file which is more comparable to rclone

7

Ok, since these are backups I see your point about not disabling the checksum.
FWIW, I mounted the Storage Box to Finder using both smb and https. I tried copy/pastes and they were very slow with little diagnostics. I did not try rclone with them though.

Storage Box utils: help from an ssh to the Storage Box gives (excerpt)

Available as server side backend:                                            |
|   borg                                                                       |
|   rsync                                                                      |
|   scp                                                                        |
|   sftp                                                                       |
|   rclone serve restic --stdio

I am not savvy enough to know if this a fair comparison or not. Hetzner said it supports rclone so I am trying it.

May I conclude that for my use case (laptop backup with a mix of large, medium, and small files) to a shared Hetzner resource that I should stick with rsync?

8

that is up to you.

let's see if any other forum members stop by and their input

9

as per hetzner documentation,
those are not the recommended values and can cause problems.

that is why you need to read the rclone debug log.

10

Thanks. I think I am not experienced enough to get the most out of rclone. I have collected logs as suggested but can’t find information on how to interpret them (earlier request).

I’ll stick with rsync.

11

No, you are trying to much - just start with the default settings!

Otherwise, yes - rclone does not need to be used when you are confident with rsync!

I like it, because I can mostly use the same commands for all of the backends I have, and that allone is very convenient, esp. as I use rclone on all of Windows, Linux, macOS, iOS, iPadOS, Android and GrapheneOS.

12

Thanks.

My need: Backup my Macbook by copying critical folders and subfolders to remote storage at Hetzner. I don’t want any tools like TimeMachine or BorgBackup that put the backups in their own format. I was burned a long time ago by a different backup package that had its own format and it was an awful strain to recover. So, I just need fast-as-possible transfer of data.

I did try very barebones rclone copies of several typical folders. Overall, the rate was ~half of rsync. rclone handled large files very well. The status bar showed 30+MB/sec for these. It struggled with lots of small files and slowed to well under 300kB/sec.

rsync, on the other hand, reports ~10MB/sec consistently.

Even if I ignore the reported rates, rsync completes the transfers 2x faster than rclone.

I tried all those other rclone switches because web searches recommended them. ChatGPT sent me down many blind alleys as well. There’s a belief out there that rclone struggles transferring lots of small files and this does seem to be the case with me.

If I need to transfer a large file or two, I may reach for rclone again. But, for an unattended transfer of my user folder and its content to storage, I think rsync works better.

13

I don't understand the problem.

When you are happy with rsync, why don't you use it?

That's a bit confusing.

I use rclone as a convenience to copy, sync and mount stuff in the same way.

I some cases I could use rsync instead, but I prefer to stay with the same commands and user interface instead.

But as I am curious, I will test this tomorrow and report back!

14

I had heard that rclone, properly configured, could use more of my available upload bandwidth compared to rsync.

15

rsync of course should be able to use your full bandwidth - you could only expect rclone to match this, not to surpass this!
When rsync cannot use all of it, you already have a problem!

Anyways, I tested it with my Hetzner Storage Box:

Transfer of 1 GB of random data:

tja@studio:~$ dd if=/dev/urandom of=test.file bs=1024k count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 2.461716 secs (436176157 bytes/sec)

tja@studio:~$ ls -h test.file
-rw-r--r-- 1 tja staff 1.0G Apr 26 11:12 test.file

tja@studio:~$ rsync -av test.file x@x.your-storagebox.de:
1,073,741,824 100% 5.16MB/s 0:03:18 (xfr#1, to-chk=0/1)

That exactly matches my 40 Mb/s upload.

rclone:

tja@studio:~$ rclone copy test.file SB:test2.file
Transferred: 1 GiB / 1 GiB, 100%, 5.162 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3m18.3s

EXACTLY the same!

My settings should be unrelated to this:

tja@studio:~$ set | grep RCLONE
RCLONE_CACHE_DIR=/Volumes/S990PRO/CACHES/RCLONE/COPY
RCLONE_CHECKERS=8
RCLONE_CONTIMEOUT=1m
RCLONE_CREATE_EMPTY_SRC_DIRS=true
RCLONE_DISABLE_HTTP2=true
RCLONE_ERROR_ON_NO_TRANSFER=true
RCLONE_LOW_LEVEL_RETRIES=5
RCLONE_METADATA=true
RCLONE_NO_UPDATE_DIR_MODTIME=false
RCLONE_NO_UPDATE_MODTIME=false
RCLONE_PROGRESS=true
RCLONE_RETRIES=3
RCLONE_TRANSFERS=4
RCLONE_USE_SERVER_MODTIME=false
RCLONE_VFS_CACHE_MAX_AGE=24h
RCLONE_VFS_CACHE_MAX_SIZE=64G
RCLONE_VFS_CACHE_MODE=writes
RCLONE_VFS_READ_CHUNK_SIZE=128M
RCLONE_VFS_READ_CHUNK_SIZE_LIMIT=off
RCLONE_VFS_WRITE_BACK=20s

To verify this, I did unset all of them:

tja@studio:~$ set | grep RCLONE
_=RCLONE_VFS_WRITE_BACK

tja@studio:~$ export RCLONE_PROGRESS=true

tja@studio:~$ rclone copy test.file SB:test3.file
Transferred: 1 GiB / 1 GiB, 100%, 5.066 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3m20.0s

Basically the same, as I was doing other network stuff at the same time.

Ah, and yes, I used “copy” but “sync” does the same:

tja@studio:~$ rclone sync test.file SB:test4.file
Transferred: 1 GiB / 1 GiB, 100%, 5.163 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3m18.2s

Summary: Just use the rclone defaults MOST of the time and only change after testing and for a reason!

If rsync cannot saturate your line, you have a problem!
And then, it may well be true that rclone is suffering from the same problem.

16

I once had problems with the wrong MTU / MSS on my network and other problems when experimenting with Jumbo frames.
Just to list some things that MAY affect your upload speed.

17

It is not necessarily true. If you repeat your test with many small files instead of one big then you will see massive difference.

rclone ability to run parallel transfers is the main reason many people choose it over rsync.

18

That should not be needed!

You can upload file after file and don’t loose much!
I tested this many times and there are only minor differences for really small files.

rclone also only transfers 4 files in “parallel” by default!

As the OP wrote, rsync cannot saturate his uplink and it seems that this is the problem!
So the “parallel advantage” does not explain it.

But you are right, as a starter the OP should probably first test with a single large file and compare both rsync and rclone and their behavior. And only then start to compare with multiple files.

19

This is simply not true. Even when copying files locally. Try with 1000s of e.g. 1KB files to make this very apparent in any test.

One can easily bring the fastest SSD to crawl when copying sequentially some nasty datasets containing a lot of small files. And when doing it to a storage connected over high latency link the issue is only exacerbated.

Try your 1GB test but with 100k 10KiB files and you will see what I mean.

20

Yes, of course and I wrote that - with really small files :smiley:

But what kind of files shall that be?
I don’t have those.

My smallest files are images.
Even small images are often more in the “several MB” area of sizes and already here the benefit of parellel transfer dimishes!

For a test, I found screenshots to be ofter smaller than regular images:

tja@studio:~/Desktop$ du -hs SCREENSHOTS/
507M SCREENSHOTS/

With 418 files, they are just above 1 MB on average. Really small for me.

tja@studio:~/Desktop$ rclone sync SCREENSHOTS SB:SCREENSHOTS
Transferred: 506.048 MiB / 506.048 MiB, 100%, 5.063 MiB/s, ETA 0s
Checks: 0 / 0, -, Listed 420
Transferred: 418 / 418, 100%
Elapsed time: 1m39.4s

tja@studio:~/Desktop$ export RCLONE_TRANSFERS=1

tja@studio:~/Desktop$ rclone sync SCREENSHOTS SB:SCREENSHOTS2
Transferred: 506.048 MiB / 506.048 MiB, 100%, 91.895 MiB/s, ETA 0s
Checks: 0 / 0, -, Listed 420
Transferred: 418 / 418, 100%
Elapsed time: 1m51.8s

That is about 10% difference?
Uninteresting in my book!

And with larger images, there is nearly no difference at all - I test all of this intensively :wink: