Dropbox size too_many_requests

What is the problem you are having with rclone?

Rclone size on a dropbox container guarantees a too_many_requests error

What is your rclone version (output from rclone version)

1.57 (error appeared somewhere in-between the 1.54 and 1.57 {I double checked} [I believe with the recent batch mode dropbox update in 1.56])

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone size dropbox:

The rclone config contents with secrets removed.

I don't feel comfortable doing this, don't feel like decrypting my config, and don't think it is relevant (the names of the containers are accurate to what they are)

A log from the command with the -vv flag

C:\rclone>rclone size dropbox: -vv
2021/12/08 02:22:03 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "size" "dropbox:" "-vv"]
2021/12/08 02:22:03 DEBUG : Creating backend with remote "dropbox:"
Enter configuration password:
password:
2021/12/08 02:22:10 DEBUG : Using config file from "C:\\rclone\\rclone.conf"
2021/12/08 02:23:50 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 02:23:50 DEBUG : pacer: low level retry 1/10 (error too_many_requests/..)
2021/12/08 02:23:50 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2021/12/08 02:23:50 DEBUG : pacer: Reducing sleep to 3m45s
2021/12/08 02:23:50 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 02:23:50 DEBUG : pacer: low level retry 1/10 (error too_many_requests/..)
2021/12/08 02:23:50 DEBUG : pacer: Reducing sleep to 2m48.75s
2021/12/08 02:23:50 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2021/12/08 02:23:50 DEBUG : pacer: Reducing sleep to 3m45s
2021/12/08 02:23:50 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 02:23:50 DEBUG : pacer: low level retry 1/10 (error too_many_requests/.)
2021/12/08 02:23:50 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2021/12/08 02:23:50 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 02:23:50 DEBUG : pacer: low level retry 1/10 (error too_many_requests/...)
2021/12/08 02:23:50 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 02:23:50 DEBUG : pacer: low level retry 2/10 (error too_many_requests/..)
2021/12/08 02:23:51 DEBUG : pacer: Reducing sleep to 3m45s
2021/12/08 02:24:16 INFO  : Signal received: interrupt
2021/12/08 02:24:16 INFO  : Dropbox root '': Commiting uploads - please wait...
2021/12/08 02:24:16 INFO  : Exiting...

I wanted to add as well that the total speed indicator and ETA (everything past the %) for -P progress on copy broke as well for dropbox in between these versions, but that's a very insignificant bug so it honestly doesn't seem worth making a whole report

Not sure that has much to do with the versions as more so just the API changes/rate limits on dropbox.

You'd want to add some limits and validate:

 --tpslimit 12 --tpslimit-burst 12

Would be a good starting point.

Not sure what that means as there isn't much detail on that item.

Here is the issue that has much more detail as well:

And some data from my side:

Without a limit

etexter@seraphite ~ % rclone size DB: -vvv
2021/12/08 07:24:40 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "size" "DB:" "-vvv"]
2021/12/08 07:24:40 DEBUG : Creating backend with remote "DB:"
2021/12/08 07:24:40 DEBUG : Using config file from "/Users/etexter/.config/rclone/rclone.conf"
2021/12/08 07:26:26 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 07:26:26 DEBUG : pacer: low level retry 1/10 (error too_many_requests/.)
2021/12/08 07:26:26 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2021/12/08 07:26:26 DEBUG : pacer: Reducing sleep to 3m45s
2021/12/08 07:26:26 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 07:26:26 DEBUG : pacer: low level retry 1/10 (error too_many_requests/.)
2021/12/08 07:26:26 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2021/12/08 07:26:26 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 07:26:26 DEBUG : pacer: low level retry 1/10 (error too_many_requests/..)
2021/12/08 07:26:26 DEBUG : pacer: Reducing sleep to 3m45s
2021/12/08 07:26:26 DEBUG : pacer: Reducing sleep to 2m48.75s
2021/12/08 07:26:26 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 07:26:26 DEBUG : pacer: low level retry 2/10 (error too_many_requests/.)
2021/12/08 07:26:26 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2021/12/08 07:26:26 DEBUG : pacer: Reducing sleep to 3m45s
2021/12/08 07:26:26 DEBUG : pacer: Reducing sleep to 2m48.75s
^C2021/12/08 07:26:34 INFO  : Signal received: interrupt
2021/12/08 07:26:34 INFO  : Dropbox root '': Commiting uploads - please wait...
2021/12/08 07:26:34 INFO  : Exiting...

With a limit

etexter@seraphite ~ % rclone size DB: -vvv --tpslimit 12 --tpslimit-burst 12
2021/12/08 07:48:33 INFO  : Starting transaction limiter: max 12 transactions/s with burst 12
2021/12/08 07:48:33 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "size" "DB:" "-vvv" "--tpslimit" "12" "--tpslimit-burst" "12"]
2021/12/08 07:48:33 DEBUG : Creating backend with remote "DB:"
2021/12/08 07:48:33 DEBUG : Using config file from "/Users/etexter/.config/rclone/rclone.conf"
Total objects: 55.699k (55699)
Total size: 180.908 TiB (198910637106946 Byte)
2021/12/08 07:56:26 DEBUG : 7 go routines active
2021/12/08 07:56:26 INFO  : Dropbox root '': Commiting uploads - please wait...
1 Like

@Animosity022

I am playing with some (throttling) improvements to the pacer and is therefore a curious to know if a (automatic) reduction of the checkers would work in this situation too (and perhaps give a result quicker than --tps-limit).

Are you able to get a result without too_many_requests if you do like this

rclone size DB: -vvv --checkers=X

where X is 4, 2 or 1?

4 fails as well. I can try 2/1 but they take quite some time:

etexter@seraphite ~ % rclone size DB: --checkers 4
2021/12/08 11:04:03 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 11:04:03 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 300 seconds.
2021/12/08 11:04:04 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 300 seconds.

Thanks! You can just try with 1 then we will certainly know if it helps to reduce checkers :slight_smile:

Yep. I started it as I had to wait 5 minutes and I'd imagine it'll take 10-30 minutes to run.

That worked:

etexter@seraphite ~ % rclone size DB: --checkers 1
Total objects: 55.682k (55682)
Total size: 180.909 TiB (198911822192069 Byte)

Perfect, thanks a lot!

The other piece that I commented the linked issue was confirming that the API as per app registration not per user as I was originally told from DB Support.

So you can easily test with making another app registration so you don't break your main use.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.