How does it work --tpslimit-burst?

Help and Support
1

What is the problem you are having with rclone?

How exactly does --tpslimit-burst work?

After reading the documentation several times, I don't understand how this flag actually works.

The documentation details that:

--tpslimit-burst int
Max burst of transactions for --tpslimit (default 1).
Normally --tpslimit will do exactly the number of transactions per second specified. However if you supply --tps-burst then rclone can save up some transactions from when it was idle giving a burst of up to the parameter supplied.
For example if you provide --tpslimit-burst 10 then if rclone has been idle for more than 10*--tpslimit then it can do 10 transactions very quickly before they are limited again.
This may be used to increase performance of --tpslimit without changing the long term average number of transactions per second.

Ok, my flags for mounting the remote on Debian and working with Google Drive and Plex are:
--tpslimit 10
--tpslimit-burst 20

I understand that:
The maximum number of requests is 10 per second (--tpslimit 10). If Rclone was idle, then rclone can send 20 burst requests. It is right?
If this statement is correct then:
The value of --tpslimit-burst should always be greater than the value of --tpslimit, right?

What does it mean?:
However if you supply --tps-burst then rclone can save up some transactions from when it was idle giving a burst of up to the parameter supplied.

How does the value given in --tpslimit-burst work?
Is it the limit of burst requests being sent fast (in my case 20) or is it the number of times the --tpslimit is sent (in my case = 20 bursts X tpslimit 10 = 200 requests being sent fast)? ??

Please describe in detail how this works. I don't quite understand how the --tpslimit-burst flag works.
I don't speak English, I'm using Google's translator to translate from Spanish.

Thanks.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.0-beta.6610.beea4d511

  • os/version: debian 11.5 (64 bit)
  • os/kernel: 5.10.0-19-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive with my personal ID and Secret

The command you were trying to run (eg rclone copy /tmp remote:tmp)

--tpslimit 10
--tpslimit-burst 20 \

2

If you are using Google Drive, you want to use the items for that backend.

So you'd configure burst / pacer sleep time.

A burst is how many tractions it'll do before it sleeps.

If you have a burst of say 20 and on a limit of 10, it'll never burst past 10 since that's the limit.

3

Yes. Rclone is allowed to save up 20 requests and use them as quickly as it wants.

No - that is incorrect.

Here is some ASCII art assuming --tplimit 10 and --tpslimit-burst 20.

The numbers are transactions and the | marks are seconds


10    10    0     20    10    10    0     0     30    10
|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----

With --tpslimit-burst 0 this would be


10    10    0     10    10    10    0     0     10    10
|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----
1 Like
4

Hello!
Thank you very much for your answer.
Is it possible that the number 30 is wrong on the timeline, and the correct value is 20?
Remember my flags:
--tplimit 10 and --tpslimit-burst 20

10...10...0....20...10...10....0.....0...¡20!...10....0....0.....0....20...10...10...
|------|-----|-----|-----|-----|-----|-----|-----|------|-----|-----|-----|-----|-----|-----|-----

The outline with the timeline was really very useful.
Well done!

5

Hi Izan,

Nick is right, there is 20 "in the burst stock" and 10 new arriving - that is 30 available.

Here is another example:

Each morning you get 10 ride tokens that will expire by the end of the day - that is your Tokens per Day rate.

Burst is a purse where you can keep a limited number of unused tokens (e.g. 20) from expiring - that is for use another day.

1 Like
6

I'm not sure I've ever thought of it that way.

What would be the proper config then if I want to ensure I don't get more than 12 TPS per second.

I guess I want burst 0 and limit 12 so it never goes past that 12?

7

If you want to never make more than 12 requests in any given second then you want burst=0

If you want your long term average to stay below 12 and allow up to 20 in any given second then you want burst=8.

The technical details can be found here:
https://pkg.go.dev/golang.org/x/time/rate#Limiter
https://en.wikipedia.org/wiki/Token_bucket

8

I have to figure out how to test dropbox as more than 12 TPS 'seems' to cause an issue so I think by having 12/12 like my setup is now, I'm actually bursting to 24 potentially at some point as I sometimes get an issue and I never knew why.

I know why now.

I'm always amazed no matter how much I think I know, I learn something new. Love the forums.

1 Like
9

I guess you can test by adding --dump headers and then grepping for something like "DEBUG : HTTP REQUEST" in the log.

10

PS: You may want to add --log-format=time,microseconds to get all the details.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.