Bandwidth throttling logic

Maddy · November 2, 2021, 10:19am

I was trying to find an alternative to rsync for copying data over wan circuit and found rclone. I am currently evaluating the use-case and hopefully it works for our requirement.

While going through the documentation, i found that, each sync can be set/started with '-bwlimit' option that will limit the amount of data that can be transferred.

I am curious to understand, how network bandwidth limit is achieved in rclone? I tried to look at the source code where i can find this logic but i failed. I wanted to understand the logic that was used to implement this feature.

Greatly appreciate if someone can help me understand the logic.

Regards
Maddy

ivandeex · November 2, 2021, 12:34pm

github.com

rclone/rclone/blob/master/fs/accounting/token_bucket.go#L99

    
      
          			bandwidthAccounting = bandwidth.Rx
          		}
          	}
          	// Limit core bandwidth to max of Rx and Tx if both are limited
          	if bandwidth.Tx > 0 && bandwidth.Rx > 0 {
          		tbs[TokenBucketSlotAccounting] = newEmptyTokenBucket(bandwidthAccounting)
          	}
          	return tbs
          }
          
          
// StartTokenBucket starts the token bucket if necessary
          func (tb *tokenBucket) StartTokenBucket(ctx context.Context) {
          	tb.mu.Lock()
          	defer tb.mu.Unlock()
          	ci := fs.GetConfig(ctx)
          	tb.currLimit = ci.BwLimit.LimitAt(time.Now())
          	if tb.currLimit.Bandwidth.IsSet() {
          		tb.curr = newTokenBucket(tb.currLimit.Bandwidth)
          		fs.Infof(nil, "Starting bandwidth limiter at %v Byte/s", &tb.currLimit.Bandwidth)
          
          
		// Start the SIGUSR2 signal handler to toggle bandwidth.

github.com

rclone/rclone/blob/master/fs/fshttp/dialer.go#L114

    
      
          	n, err = c.Conn.Read(b)
          	accounting.TokenBucket.LimitBandwidth(accounting.TokenBucketSlotTransportRx, n)
          	if err == nil && n > 0 && c.timeout > 0 {
          		err = c.nudgeDeadline()
          	}
          	return n, err
          }
          
          
// Write bytes with rate limiting and idle timeouts
          func (c *timeoutConn) Write(b []byte) (n int, err error) {
          	accounting.TokenBucket.LimitBandwidth(accounting.TokenBucketSlotTransportTx, len(b))
          	n, err = c.Conn.Write(b)
          	if err == nil && n > 0 && c.timeout > 0 {
          		err = c.nudgeDeadline()
          	}
          	return n, err
          }

Maddy · November 3, 2021, 2:31am

Thank you for sharing the source paths. Code makes me believe its using token bucket but i am finding little difficult to understand how this is implemented for ALL storage systems (like SFTP, Dropbox etc)? Does rclone sends files as stream/packets to remote system? If true, how remote system understand the stream/packet its recieving? if rclone is sending files as single files (like copying one file using sftp client) how throttling is applied?

This is one of those projects which made me curious on many aspects and i am trying to understand the technical details of such aspects.

Regards

ncw · November 3, 2021, 10:09am

All data transferred comes through the accounting system as a stream, and we apply a token bucket rate limiter in the "middle" there.

We also apply token bucket rate limiters at the "edges" so for HTTP in and HTTP out. These are more accurate and stop excess buffering, but they don't work for non HTTP protocols like FTP/SFTP so these fall back to using the "middle" rate limiter.

Maddy · November 3, 2021, 12:20pm

thank you @ncw for providing more info to help me understand the logic.

Regards