I am experiencing dramatic networking saturation issues links that are potentially correlated with "rclone copy <source_path> MyDrive:<dest_path>
This has apparently saturated my networking link and caused significant disruption to my other server that shares the same link.
I see your comment here about "rclone stealing the entire bandwidth" and just want to confirm with you whether the above command would cause a similar situation or not?
The rclone version used here was 1.47 but I see your report on even 1.53 and that makes me wonder if this is still the case. I also think that this also depends on the structure of the file trees that are getting copied over (Number of files/folders/size-per-file). The reason is that I haven't seen this event over the last several months and now this appears with a new set of potentially "big data ~ 100TB" or more
Can you clarify if I am thinking in the right direction or not? Also, any suggestions on how to control this situation to occur would be greatly appreciated. I have canceled a lot of rclone transfers so far...
Note, --bandwidth seems not to restrict the effective bandwidth. I can see specifying a much lower bandwidth for the copy command but still, bandwidth usage is on the roof.
Yes, bwlimit was not effective in that version. We are yet to try it with 1.53 but I am seeing other threads reporting difficulties controlling th bandwidth at 1.53 too. Should bandwidth per file and other options get tuned based on the files structure?
I just need more testing on this with the latest.
Can you test the current version, share the command you are using for testing, share your expectations and what are you trying to accomplish (I want to limit to 10MB/s) and report back.
Yes I need to find an off timehour to do this to avoid outages. 8MBps is what I would look into with bandwidth.
Also, is there a global settings that can enforce this bandwidth for all rclone executables? Speaking of multi-user environments.
thanks so much for your quick response
Rclone is configurable per user so each user's customizations would be for them. Just remember as bandwidth is usually measured in bits and rclone is doing this in bytes so 8MB would be ~64Mb/s.
You'd have to attack that from a different path and if you wanted to limit on a system, you'd look to QoS controls per user.