Using rclone.exe causes large DHCP delays

What is the problem you are having with rclone?

When running a backup from Windows using rclone.exe, our DHCP server has a hell of a time handing out IP addresses. The server from which the backup is being run, is just a file server. Our DHCP server is separate from this.

What is your rclone version (output from rclone version)

v1.49.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 2012 R2

Which cloud storage system are you using? (eg Google Drive)

BackBlaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe copy D:\Source b2:\bucket\sub --bwlimit "06:45,10M 19:00,50M 21:00,off" -P --transfers=6 -u -vv

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Transferred:       85.142G / 379.066 GBytes, 22%, 9.951 MBytes/s, ETA 8h24m6s
Errors:                 2 (retrying may help)
Checks:             11645 / 11645, 100%
Transferred:         1383 / 5379, 26%
Elapsed time:   2h26m1.7s
Transferring:

The entire log is massive and would need to be whitewashed before I can post it. Suffice it to say there aren't any glaring issues.

Stopping the backup operation seems to resolve the issues, but we've been unable to pinpoint what would actually be causing this. Out internet is routed through our DHCP server it isn't our Gateway.

We're thinking maybe a packet storm of some sort, but wireshark hasn't been able to prove anything useful to this investigation.

Sounds like rclone is too effective and is performing too well and saturating your network.

You'd want to limit the bandwidth in rclone if that is the case as it seems your network cannot handle the values that you have or you can always look into packet shaping your network if that is available as that's what I do at home.

That prioritizes items and also puts any rclone traffic in a low priority bucket for me.

That is what I was expecting someone to say. :slight_smile: Yet these things are out of my control and the only thing I can do is scream at the wall..

You can reduce the bwlimit in rclone further though and that should help.

Yeah, really slowing down the backup speed isn't the issue, our internet connection is actually very good. I would rather the networking team just configure the switches such that this wasn't an issue.

Even slowing it down just a little could really help in this situation

Yep as @seuffert mentioned, if you bring it down a bit, it could help as you might be hitting a tipping point and the gear is handling at that capacity.

It is like trying to find a sweet spot as you want to push it as much as you can without breaking your network. A little trial and error is needed.

You are acheving this bandwith at the moment

So just knocking it down by 10% should leave enough spare on the network, eg --bwlimit 9M or --bwlimit 9.5M

Another thing you could do would be to apply some QoS in the network which is more difficult but will help with all programs.

I'll give the reduction a try, but am really more annoyed that the network is doing really weird things.

Without knowing much more about topology and makes of network equipment, I would normally start looking at layer 2 switches wrongly deprioritising broadcast traffic; But you're "only" doing 10MB/sec, which is a nominal 100Mb LAN - surely you use 1000Mb ports for your network nowadays?

Have you checked that the layer 2 hierarchy between rclone system and the external router is not crossing the collision domain of the DHCP server and/or the primary client systems showing problems?

Edit: Also, is anything using real jumbo packets along the same routes? Always a bad thing for blocking ports...

I'll just note you aren't the first rclone user to report this sort of problem. Reports like this are why --bwlimit exists!