Hi all, I am new to using rClone for backing up to an external site, and am running into some issues I hope you can help with, I am trying to sync to Backblaze B2, and every time I start the sync my internet connection drops immediately. I have a pfSense firewall in front of it, and have tried setting a bandwidth limit via the rClone CLI, I have a 5mpbs upload speed, and have set the flag --bwlimit 225, and this allows my connection to stay up at times, but when it starts a new file I have observed the VM running rClone (this is the only thing that VM does) jumping way up and using more bandwidth than I allocated via the bwlimit flag, and crashes my internet connection for a few minutes or so. I attempted to set a limit on the pfSense itself instead, but anytime I run rClone with that limiter in place it connects, uploads 3-4% of my files, then the speed dwindles down to 0 and never picks up again. Just trying to find some guidance, see if anybody has any idea what might be causing rClone to crash my internet, and why it still kicks up past my bwlimit on occasion, and brings down my connection again.
This isn’t the first report of rclone crashing people’s router I have seen. Rclone uses the internet connection quite aggressively. Hopefully you can get a firmware update for your router - that should fix it.
Or, actually, to be a bit more clear, it does not crash the router itself, that stays up and works fine, I can access my modem (in bridge mode, which is beyond my router) and see its status page, but I cannot connect to the internet, I did some reading, and am thinking it may be something to do with the bwlimit, that it is saturating my connection to the point that nothing can get in/out when it bursts, I am not sure though, this is a weird issue, I am at a bit of a loss on this issue
Try doing a transfer with rclone and the -v flag. rclone will say how many kByte/s it gets for the transfer. Then use a --bwlimit with half this value say - that should allow your network to function properly.
That appears to have resolved at least some of the issues, sadly not ideal, and getting occasional errors with backblaze as well, its uploading much slower than I would like, but it is at least working consistently now, thank you very much for your help ncw!
I suspect you are suffering from bufferbloat BTW - a common symptom is that when an big upload is in progress it kills the performance of downloads.
I used to use wondershaper for linux to help with this - I wonder if there is something similar for pfSense? What you do is limit the upload to a bit less than the link capacity (say 90%) - this stops the buffers getting full and hence the terrible performance.
I actually did give that a try before posting, I configured a bandwidth limit for the VM that is running rClone, and it would work for 15-20 minutes, then for some reason the upload will just dwindle down to 0b/s and never recover, I even tried setting a limit of (example) 250 KB/s upload on my pfsense firewall and 225 KB/s via --bwlimit, and it did the same thing. (I really appreciate your responsiveness, That is uncommon with a lot of Devs, and I love it!)
How does the shaper work? Does it drop lots of packets? If so that could explain the dwindling down to zero… Maybe…
I don’t think this is an rclone problem, I think it is something to do with your network but I don’t know what… Do you see similar problems when you use other programs to do big uploads?
Sorry for the late reply, now I am really confused by this one, it worked file all week at 1/2 speed, I stopped it to run some updates, started it up again and it started dwindling down to 0 even at half my upload speed (I am using the rClone built in traffic shaper, not the one built into pfSense) I don’t see anything getting blocked on the pfSense side (not to say that it couldn’t be that, it certainly could, just nothing obvious is showing up)
I recently updated an early version of rclone with cache to latest and I now have the same issue of internet becoming unresponsive or my router (orange box) rebooting.
Setting bwlimit is not an option on mount as it would also limit read.
Thanks for the tip @ncw I will have a look at wondershaper but it will complexity the setup.
