Bandwidth throttling logic

I was trying to find an alternative to rsync for copying data over wan circuit and found rclone. I am currently evaluating the use-case and hopefully it works for our requirement.

While going through the documentation, i found that, each sync can be set/started with '-bwlimit' option that will limit the amount of data that can be transferred.

I am curious to understand, how network bandwidth limit is achieved in rclone? I tried to look at the source code where i can find this logic but i failed. I wanted to understand the logic that was used to implement this feature.

Greatly appreciate if someone can help me understand the logic.

Regards
Maddy

Thank you for sharing the source paths. Code makes me believe its using token bucket but i am finding little difficult to understand how this is implemented for ALL storage systems (like SFTP, Dropbox etc)? Does rclone sends files as stream/packets to remote system? If true, how remote system understand the stream/packet its recieving? if rclone is sending files as single files (like copying one file using sftp client) how throttling is applied?

This is one of those projects which made me curious on many aspects and i am trying to understand the technical details of such aspects.

Regards

All data transferred comes through the accounting system as a stream, and we apply a token bucket rate limiter in the "middle" there.

We also apply token bucket rate limiters at the "edges" so for HTTP in and HTTP out. These are more accurate and stop excess buffering, but they don't work for non HTTP protocols like FTP/SFTP so these fall back to using the "middle" rate limiter.

thank you @ncw for providing more info to help me understand the logic.

Regards

1 Like