Bwlimit is not really smooth

What is the problem you are having with rclone?

I'm using rclone to sync a local path to a remote directory on a mega account.
I noticed that rclone bandwith usage is not 'smooth'. It is bursting my dsl connection for 6-8 seconds; then there are some seconds without bandwidth usage 0kbps, probably in order to respect the requested bandwith.
my max dsl connection : 115k so I asked 70k in order to let some bandwidth to the others users.
I noticed the issue during a big file transfer (file size > 10MB) to mega, so at a moment I could expect a stable bandwith.

I also tried koofr storage which has a similar issue. S3 at scaleway looks good. and webdav looks good too.

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

linux debian stable/buster 64b

Which cloud storage system are you using? (eg Google Drive)

mega

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /mnt/local_path/ :mega:remote_dir/ -v --bwlimit 70k --max-delete 2 --transfers 1 --checkers 3 --stats 20m --size-only --delete-after --delete-excluded

The rclone config contents with secrets removed.

I'm running without config and I use the command line parameters

--mega-user 'xxx' --mega-pass xxxxxx

A log from the command with the -vv flag

2020/12/09 18:25:30 DEBUG : Creating backend with remote "hidden_local_path"
2020/12/09 18:25:30 DEBUG : Using config file from "/dev/null"
2020/12/09 18:25:30 INFO  : Starting bandwidth limiter at 70kBytes/s
2020/12/09 18:25:30 DEBUG : Creating backend with remote ":mega:hidden_remote_dir/"
2020/12/09 18:25:31 DEBUG : fs cache: renaming cache item ":mega:hidden_remote_dir/" to be canonical ":mega:hidden_remote_dir"
...
numerous files status
2020/12/09 18:25:31 DEBUG : *****: Excluded
2020/12/09 18:25:31 DEBUG : ****: Unchanged skipping
2020/12/09 18:25:31 DEBUG : ****: Sizes identical
...
2020/12/09 18:25:31 DEBUG : mega root 'hidden_remote_dir': Waiting for checks to finish
2020/12/09 18:25:31 DEBUG : mega root 'hidden_remote_dir': Waiting for transfers to finish
^C

Rclone is in Bytes not bits so when you set 70K, you are actually setting 70KB/s and if you are looking for 70Kb/s, it's not the same.

Also if you want really smooth bandwidth limits then try the beta from this issue: https://github.com/rclone/rclone/issues/4395

Hi,

Rclone is in Bytes not bits so when you set 70K, you are actually setting 70KB/s and if you are looking for 70Kb/s, it's not the same.

70KB/s is fine for me

if you want really smooth bandwidth limits then try the beta...

I tried these versions:
rclone-v1.54.0-beta.4959.d7c84b094.fix-4395-list-bwlimit-linux-amd64
rclone-v1.52.2-198-gaedc8623-fix-4395-list-bwlimit-beta-linux-amd64

--bwlimit 100k you can now write a pair --bwlimit 10M:1M which is upload:download

I didn't tried the upload /download. I tried using the old syntax
I only tried using the mega cloud storage.

my results are 8-10s at max dsl connection and then 0kbs for ~5s. The average value looks good but the interval in order to regulate is still too long.

Thank you for taking time in reading me (and that nice software)
Regards

That sounds like the http limiting isn't working for mega at least.

Can you try a different cloud storage provider and see if it works for them? I tried to replicate this with mega but I seem to be locked out of my account :frowning:

Hi,

I made some tests (using v1.53.3) with some other providers :
mega : bandwith not smooth (using v1.53.3 and v1.54.0-beta.4959.d7c84b094.fix-4395-list-bwlimit)
koofr : bandwith not smooth if using v1.53.3, but no issue using v1.54.0-beta.4959.d7c84b094.fix-4395-list-bwlimit
webdav (3 different providers) : no issue
s3 (from scaleway) : no issue
b2 : no issue
yandex : no issue
pcloud : no issue

Maybe you recover your mega account

Regards,

Thanks for your comprehensive testing. It sounds like the patch is working, but as I suspected the mega backend is evading the rate limiting somehow.

I see what is happening here. It is because the uploads for mega happens in quite small (1M) blocks. Mega then pauses for quite a while between each block. This pause allows the bandwidth limiter to recharge its bucket which I've set quite large, so there is a burst of very high speed traffic.

I think I've solved that here - can you give it a go?

v1.54.0-beta.4975.346fbde8c.fix-4395-list-bwlimit on branch fix-4395-list-bwlimit (uploaded in 15-30 mins)

Can you check the uploaded files ? because it looks like numerous target-os are missing (only 14 files instead of 38).

Sorry the build failed. Hopefully this one will work!

v1.54.0-beta.4974.293db48ce.fix-4395-list-bwlimit on branch fix-4395-list-bwlimit (uploaded in 15-30 mins)

I'm not sure, but it's possible that the new build has failed.

It did! Sorry. A stray commit got in there from the wrong branch.

This one has built!

v1.54.0-beta.4973.e562ed919.fix-4395-list-bwlimit on branch fix-4395-list-bwlimit (uploaded in 15-30 mins)

With the new build, I no longer observe the issue with mega cloud storage... Thank you

Great - thanks for testing :slight_smile: