--bwlimit issue?


#1

I am having some issues when using the --bwlimit flag which have just started. I’ve been using the feature for months without issue but for the last week or so I’ve been having problems. What’s happening is that a couple of transfers will hog all of the available bandwidth and leave remaining transfers around 100 KB/s. The problem is that when the transfers that were hogging the bandwidth are finished the slower ones will not pick up speed and use the available bandwidth. So eventually, all of my transfers will get stuck around 100 KB/s. I’ve included a couple of screenshots to try to make this more clear. I’m not sure if this is a problem with rclone or with my setup. Has anyone experienced this before or have an idea of how to fix this?

There are 10 files being uploaded and --bwlimit is set to 40M. You can see with 4 transfers 2 are hogging all of the available bandwidth.

Here, all of the files have been uploaded except for the first two which have been hanging around 100 KB/s for the entirety of the transfer. These will never pick up speed. I’ve tried running transfers overnight and have seen them stall like this for hours at a time.


#2

I think his is unlikely to be anything rclone is doing but I could be wrong… Can you try without --bwlimit?

The fact that they all reach a fairly constant 10kBytes/s makes me suspect it might be your ISP doing traffic shaping - is that possible?


#3

Thanks for your reply. This is a server housed by OVH so there shouldn’t be any traffic shaping. I’ll try some transfers without --bwlimit and report back.


#4

Same issue without --bwlimit. Two downloads hog all the available bandwidth and two get left in the dust forever. I think you’re right that this is not an rclone issue. Especially considering that I have not changed the rclone version or made any changes to the config. I wonder what might be causing it on my system?

41 PM


#5

What is interesting is that it is always the first files to get “pushed back” that struggle. In the screen shot above the two files that are being uploaded fast finished and all subsequent uploads worked fine. That folder had about 20 files and they all uploaded fine except for the very first 2 files that show being uploaded slowly in the screen shot.


#6

What is odd about this is that the files at the end never pick up speed. If this is just TCP socket bandwidth contention in the OS then seeing uneven flow rates is probably only to be expected, however I would expect the slow files to pick up speed at the end.

Where are you uploading to? Have you tried with more or less --transfers? Which OS are you using?


#7

I’m uploading to Google Drive from Ubuntu 16.04 and I’ve tried with a number of different transfer options. It seems that no matter what I do 1-4 transfers will get left behind every batch upload. Would it help if I started keeping a log of every transfer?


#8

If you could keep a log with -vv and send it to me then I can take a look.


#9

Thanks a lot Nick! I really appreciate your help. I’ll run it nightly for a few nights and send you a log next week.