Bwlimit is not really smooth

Suspected Bug
#1

What is the problem you are having with rclone?

I'm using rclone to sync a local path to a remote directory on a mega account.
I noticed that rclone bandwith usage is not 'smooth'. It is bursting my dsl connection for 6-8 seconds; then there are some seconds without bandwidth usage 0kbps, probably in order to respect the requested bandwith.
my max dsl connection : 115k so I asked 70k in order to let some bandwidth to the others users.
I noticed the issue during a big file transfer (file size > 10MB) to mega, so at a moment I could expect a stable bandwith.

I also tried koofr storage which has a similar issue. S3 at scaleway looks good. and webdav looks good too.

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

linux debian stable/buster 64b

Which cloud storage system are you using? (eg Google Drive)

mega

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /mnt/local_path/ :mega:remote_dir/ -v --bwlimit 70k --max-delete 2 --transfers 1 --checkers 3 --stats 20m --size-only --delete-after --delete-excluded

The rclone config contents with secrets removed.

I'm running without config and I use the command line parameters

--mega-user 'xxx' --mega-pass xxxxxx

A log from the command with the -vv flag

2020/12/09 18:25:30 DEBUG : Creating backend with remote "hidden_local_path"
2020/12/09 18:25:30 DEBUG : Using config file from "/dev/null"
2020/12/09 18:25:30 INFO  : Starting bandwidth limiter at 70kBytes/s
2020/12/09 18:25:30 DEBUG : Creating backend with remote ":mega:hidden_remote_dir/"
2020/12/09 18:25:31 DEBUG : fs cache: renaming cache item ":mega:hidden_remote_dir/" to be canonical ":mega:hidden_remote_dir"
...
numerous files status
2020/12/09 18:25:31 DEBUG : *****: Excluded
2020/12/09 18:25:31 DEBUG : ****: Unchanged skipping
2020/12/09 18:25:31 DEBUG : ****: Sizes identical
...
2020/12/09 18:25:31 DEBUG : mega root 'hidden_remote_dir': Waiting for checks to finish
2020/12/09 18:25:31 DEBUG : mega root 'hidden_remote_dir': Waiting for transfers to finish
^C
#2

Rclone is in Bytes not bits so when you set 70K, you are actually setting 70KB/s and if you are looking for 70Kb/s, it's not the same.

#3

Also if you want really smooth bandwidth limits then try the beta from this issue: https://github.com/rclone/rclone/issues/4395

#4

Hi,

Rclone is in Bytes not bits so when you set 70K, you are actually setting 70KB/s and if you are looking for 70Kb/s, it's not the same.

70KB/s is fine for me

if you want really smooth bandwidth limits then try the beta...

I tried these versions:
rclone-v1.54.0-beta.4959.d7c84b094.fix-4395-list-bwlimit-linux-amd64
rclone-v1.52.2-198-gaedc8623-fix-4395-list-bwlimit-beta-linux-amd64

--bwlimit 100k you can now write a pair --bwlimit 10M:1M which is upload:download

I didn't tried the upload /download. I tried using the old syntax
I only tried using the mega cloud storage.

my results are 8-10s at max dsl connection and then 0kbs for ~5s. The average value looks good but the interval in order to regulate is still too long.

Thank you for taking time in reading me (and that nice software)
Regards

#5

That sounds like the http limiting isn't working for mega at least.

Can you try a different cloud storage provider and see if it works for them? I tried to replicate this with mega but I seem to be locked out of my account :frowning:

#6

Hi,

I made some tests (using v1.53.3) with some other providers :
mega : bandwith not smooth (using v1.53.3 and v1.54.0-beta.4959.d7c84b094.fix-4395-list-bwlimit)
koofr : bandwith not smooth if using v1.53.3, but no issue using v1.54.0-beta.4959.d7c84b094.fix-4395-list-bwlimit
webdav (3 different providers) : no issue
s3 (from scaleway) : no issue
b2 : no issue
yandex : no issue
pcloud : no issue

Maybe you recover your mega account

Regards,

#7

Thanks for your comprehensive testing. It sounds like the patch is working, but as I suspected the mega backend is evading the rate limiting somehow.

I see what is happening here. It is because the uploads for mega happens in quite small (1M) blocks. Mega then pauses for quite a while between each block. This pause allows the bandwidth limiter to recharge its bucket which I've set quite large, so there is a burst of very high speed traffic.

I think I've solved that here - can you give it a go?

v1.54.0-beta.4975.346fbde8c.fix-4395-list-bwlimit on branch fix-4395-list-bwlimit (uploaded in 15-30 mins)

#8

Can you check the uploaded files ? because it looks like numerous target-os are missing (only 14 files instead of 38).

#9

Sorry the build failed. Hopefully this one will work!

v1.54.0-beta.4974.293db48ce.fix-4395-list-bwlimit on branch fix-4395-list-bwlimit (uploaded in 15-30 mins)

#10

I'm not sure, but it's possible that the new build has failed.

#11

It did! Sorry. A stray commit got in there from the wrong branch.

This one has built!

v1.54.0-beta.4973.e562ed919.fix-4395-list-bwlimit on branch fix-4395-list-bwlimit (uploaded in 15-30 mins)

#12

With the new build, I no longer observe the issue with mega cloud storage... Thank you

#13

Great - thanks for testing :slight_smile:

#14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.