RClone bandwidth Issues

What is the problem you are having with rclone?

bandwidth is out-of-control, causing network congestion

What is your rclone version (output from rclone version)

1.47 (same issue appears in 1.53)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS7.9, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/// rclone copy <source> MyDrive:<destination> ///

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here

Related to # 4537

I am experiencing dramatic networking saturation issues links that are potentially correlated with "rclone copy <source_path> MyDrive:<dest_path>
This has apparently saturated my networking link and caused significant disruption to my other server that shares the same link.

I see your comment here about "rclone stealing the entire bandwidth" and just want to confirm with you whether the above command would cause a similar situation or not?

The rclone version used here was 1.47 but I see your report on even 1.53 and that makes me wonder if this is still the case. I also think that this also depends on the structure of the file trees that are getting copied over (Number of files/folders/size-per-file). The reason is that I haven't seen this event over the last several months and now this appears with a new set of potentially "big data ~ 100TB" or more

Can you clarify if I am thinking in the right direction or not? Also, any suggestions on how to control this situation to occur would be greatly appreciated. I have canceled a lot of rclone transfers so far...

Note, --bandwidth seems not to restrict the effective bandwidth. I can see specifying a much lower bandwidth for the copy command but still, bandwidth usage is on the roof.

1.47 is an ancient version so you really want to use the latest stable.

How much do you want to limit rclone?

https://rclone.org/docs/#bwlimit-bandwidth-spec is what you need to add to your command to limit the bandwidth in rclone.

Yes, bwlimit was not effective in that version. We are yet to try it with 1.53 but I am seeing other threads reporting difficulties controlling th bandwidth at 1.53 too. Should bandwidth per file and other options get tuned based on the files structure?
I just need more testing on this with the latest.

Can you test the current version, share the command you are using for testing, share your expectations and what are you trying to accomplish (I want to limit to 10MB/s) and report back.

Yes I need to find an off timehour to do this to avoid outages. 8MBps is what I would look into with bandwidth.
Also, is there a global settings that can enforce this bandwidth for all rclone executables? Speaking of multi-user environments.
thanks so much for your quick response

Rclone is configurable per user so each user's customizations would be for them. Just remember as bandwidth is usually measured in bits and rclone is doing this in bytes so 8MB would be ~64Mb/s.

You'd have to attack that from a different path and if you wanted to limit on a system, you'd look to QoS controls per user.

Note that you need --bwlimit 1M for this as rclone bandwidth limiter measures in MByte/s not MBit/s.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.