Bwlimit seems to be restricted to approx. 2.5Gbps

Suspected Bug
1

Hi all, I love this tool and found what seems to be a small bug that I haven't seen reported by anyone else yet. I'm not yet sure if it's any source/destination pair, or specific to S3 to GCS.

I'm trying to coping data from a S3 compatible system to GCS with a rate limit of 500MBps (4Gbps) by using the --bwlimit 500M flag.
What I expected to happen is bw usage would be ~4Gbps.
What happens is bw usage is about 2.5Gbps. I have tried setting bwlimit even higher, e.g. 1G, with the same result. If I remove the --bwlimit flag then the entire internet pipe (10Gbs) is filled by the transfer, which seems to show that it's the rate limiting that is causing the issue.

rclone v1.56.0

  • os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
  • os/kernel: 10.0.17763.2061 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.16.5
  • go/linking: dynamic
  • go/tags: cmount

rclone command used

rclone sync sourceuser:sourcebucket destuser:destbucket --bwlimit 500M --transfers 400 --checkers 500 --max-delete 10000 --log-level INFO --stats 1m --stats-log-level INFO --use-mmap --fast-list --modify-window 1s

rclone config:

[sourceuser]
type = s3
env_auth = false
access_key_id = sourceuser
secret_access_key = deleted
region = other-v4-signature
endpoint = endpoint.dns.name
location_constraint = 
acl = private
server_side_encryption = 
storage_class = 

[destuser]
type = google cloud storage
service_account_file = ${RCLONE_CONFIG_DIR}\destuser.json
bucket_policy_only = true
location = us-central1

I haven't pasted any log files here since they are multi-GB log files with just INFO log level. Source bucket is large >1M objects totaling >250TB. Most objects are large.
I have the transfers number set so high since GCS seems to only do one transfer per object and I needed more than 200 transfers to get constantly over 4Gbps throughput.
I'm interested to see if anyone else has struck an upper limit to --bwlimit or has any ideas what to try. I am trying not to totally saturate our pipe, but 2Gbps isn't fast enough for this particular task since we have more than 1.5PB to upload. My next step will be to upload 2 buckets at once which I expect will go at 2Gbps each and be a good work around for my issue, so I'm not stuck at all, just curious and wanting to let others know what I'm seeing.

Thanks

2

The bwlimit is in Bytes not Bits so you have to convert.

      --bwlimit BwTimetable                  Bandwidth limit in KiByte/s, or use suffix B|K|M|G|T|P or a full timetable.
3

--bwlimit 500M/s should give about 4 Gbit/s you are right.

I have a feeling this problem is due to the granularity of the token bucket filter rclone uses for the bwlimit.

You could try increasing this constant and see if it makes any difference

This might also be to do with Windows - I think the timer on Windows runs slower so it is more likely the token bucket will empty (handwaving).

Trying this test on linux

$ rclone cat --bwlimit 500M /tmp/10G | pv >/dev/null
10.0GiB 0:00:20 [ 498MiB/s] [                                                       <=>                                                                              ]

I have a vague memory of this coming up before but I can't find the reference.
works fine at 500MiB/s. It maxes out at about 1.8GiB/s

5

Thanks Nick, it seems you got the issue in one. I increased maxButstSize by 4X and I can now throttle up to 10Gbps speeds. Tested with --bwlimit 500M and 1G and works perfectly with both with that change.

I guess this isn't an elegant solution, so is it worth putting in a pull request for this fix? Thoughts on if this will break anything else?
Thanks,

6

Great, glad that sorted it.

From what I remember there is a tension between making that number too high (which increases lumpiness in the bandwidth limit) and making it too small (which limits maximum bandwidth throughput).

What I really need to do is find the conversation I had about this previously.

As a first step could you open a new issue on Github about this and we can try and collect the info there.

It might be that a more sophisticated fix is needed or a different number for Windows / Linux I'm not sure at the moment.

7

Thanks Nick.
Issue created

Could maxBurstSize be a variable that is related to the bwlimit?

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.