Dropbox Dir.ReadDirAll error: too_many_requests/

plotterx · February 6, 2022, 8:14pm

Hello,
Since 1 weeks, I got some errors. and I have some logs here.
I was able to use and mount all my directories before this time for 3 months.

my mount code is:

mounter="rclone mount dropbox1:/Party1 /dropbox1 --vfs-cache-mode off --multi-thread-streams 200 --low-level-retries 2 --retries 2 --vfs-read-chunk-size 32K --dropbox-chunk-size 32K --buffer-size off --max-backlog 20000 --contimeout 9s --no-traverse --no-modtime --read-only --log-level DEBUG --stats 1m --allow-non-empty --allow-other --dir-cache-time 10000h --vfs-cache-max-age 100000h --dropbox-batch-mode sync"

Also when I list the directory, I have "input/output error"

2022/02/06 21:13:00 DEBUG : /: >Lookup: node=plot2/, err=<nil>
2022/02/06 21:13:00 DEBUG : plot2/: Attr: 
2022/02/06 21:13:00 DEBUG : plot2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/06 21:13:00 DEBUG : : Statfs: 
2022/02/06 21:13:00 DEBUG : : >Statfs: stat={Blocks:200899821568 Bfree:57433520769 Bavail:57433520769 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2022/02/06 21:13:00 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 5 seconds.
2022/02/06 21:13:00 DEBUG : pacer: low level retry 1/2 (error too_many_requests/...)
2022/02/06 21:13:05 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 5 seconds.
2022/02/06 21:13:05 DEBUG : pacer: low level retry 2/2 (error too_many_requests/...)
2022/02/06 21:13:05 DEBUG : plot1: Dir.ReadDirAll error: too_many_requests/...
2022/02/06 21:13:05 DEBUG : plot1/: >ReadDirAll: item=-1, err=too_many_requests/...
2022/02/06 21:13:05 DEBUG : /: >Lookup: node=plot1/, err=<nil>
2022/02/06 21:13:05 DEBUG : plot1/: Attr: 
2022/02/06 21:13:05 DEBUG : plot1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/06 21:13:05 DEBUG : : Statfs: 
2022/02/06 21:13:05 DEBUG : : >Statfs: stat={Blocks:200899821568 Bfree:57433520769 Bavail:57433520769 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2022/02/06 21:13:09 DEBUG : /: Lookup: name="plot1"
2022/02/06 21:13:09 DEBUG : /: >Lookup: node=plot1/, err=<nil>
2022/02/06 21:13:09 DEBUG : plot1/: Attr: 
2022/02/06 21:13:09 DEBUG : plot1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/06 21:13:09 DEBUG : : Statfs: 
2022/02/06 21:13:09 DEBUG : : >Statfs: stat={Blocks:200899821568 Bfree:57433520769 Bavail:57433520769 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2022/02/06 21:13:09 DEBUG : /: Lookup: name="plot2"
2022/02/06 21:13:09 DEBUG : /: >Lookup: node=plot2/, err=<nil>
2022/02/06 21:13:09 DEBUG : plot2/: Attr: 
2022/02/06 21:13:09 DEBUG : plot2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/06 21:13:09 DEBUG : : Statfs: 
2022/02/06 21:13:09 DEBUG : : >Statfs: stat={Blocks:200899821568 Bfree:57433520769 Bavail:57433520769 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2022/02/06 21:13:22 DEBUG : /: Lookup: name="plot2"
2022/02/06 21:13:22 DEBUG : /: >Lookup: node=plot2/, err=<nil>
2022/02/06 21:13:22 DEBUG : plot2/: Attr: 
2022/02/06 21:13:22 DEBUG : plot2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/06 21:13:22 DEBUG : plot2/: ReadDirAll:

asdffdsa · February 6, 2022, 8:16pm

hello and welcome to the forum,

when you posted there was a template of questions, none of which you answered.
help us to help you and answer ALL the questions.

plotterx · February 7, 2022, 6:20am

Hello again,
When try only with rclone mount dropbox1:/Party1 /elma/tmp --log-level DEBUG
I get same results when I use ls command.

022/02/07 07:18:35 DEBUG : Creating backend with remote "dropbox1:/Party1"
2022/02/07 07:18:35 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/02/07 07:18:36 DEBUG : Dropbox root '': Using root namespace "9939456352"
2022/02/07 07:18:36 DEBUG : fs cache: renaming cache item "dropbox1:/Party1" to be canonical "dropbox1:Party1"
2022/02/07 07:18:36 DEBUG : Dropbox root 'Party1': Mounting on "/elma/tmp"
2022/02/07 07:18:36 DEBUG : : Root: 
2022/02/07 07:18:36 DEBUG : : >Root: node=/, err=<nil>
2022/02/07 07:18:50 DEBUG : /: Attr: 
2022/02/07 07:18:50 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/07 07:18:50 DEBUG : /: ReadDirAll: 
2022/02/07 07:18:51 DEBUG : /: >ReadDirAll: item=5, err=<nil>
2022/02/07 07:18:51 DEBUG : /: Lookup: name="plot1"
2022/02/07 07:18:51 DEBUG : /: >Lookup: node=plot1/, err=<nil>
2022/02/07 07:18:51 DEBUG : plot1/: Attr: 
2022/02/07 07:18:51 DEBUG : plot1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/07 07:18:51 DEBUG : /: Lookup: name="plot2"
2022/02/07 07:18:51 DEBUG : /: >Lookup: node=plot2/, err=<nil>
2022/02/07 07:18:51 DEBUG : plot2/: Attr: 
2022/02/07 07:18:51 DEBUG : plot2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/07 07:18:51 DEBUG : /: Lookup: name="plot3"
2022/02/07 07:18:51 DEBUG : /: >Lookup: node=plot3/, err=<nil>
2022/02/07 07:18:51 DEBUG : plot3/: Attr: 
2022/02/07 07:18:51 DEBUG : plot3/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/07 07:18:57 DEBUG : /: Attr: 
2022/02/07 07:18:57 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/07 07:18:58 DEBUG : /: ReadDirAll: 
2022/02/07 07:18:58 DEBUG : /: >ReadDirAll: item=5, err=<nil>
2022/02/07 07:18:59 DEBUG : /: Lookup: name="plot1"
2022/02/07 07:18:59 DEBUG : /: >Lookup: node=plot1/, err=<nil>
2022/02/07 07:18:59 DEBUG : plot1/: Attr: 
2022/02/07 07:18:59 DEBUG : plot1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2022/02/07 07:18:59 DEBUG : plot1/: ReadDirAll: 
2022/02/07 07:19:01 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:01 DEBUG : pacer: low level retry 1/10 (error too_many_requests/.)
2022/02/07 07:19:01 DEBUG : pacer: Rate limited, increasing sleep to 5s
2022/02/07 07:19:04 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:04 DEBUG : pacer: low level retry 2/10 (error too_many_requests/..)
2022/02/07 07:19:08 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:08 DEBUG : pacer: low level retry 3/10 (error too_many_requests/)
2022/02/07 07:19:14 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:14 DEBUG : pacer: low level retry 4/10 (error too_many_requests/...)
2022/02/07 07:19:18 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:18 DEBUG : pacer: low level retry 5/10 (error too_many_requests/...)
2022/02/07 07:19:23 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:23 DEBUG : pacer: low level retry 6/10 (error too_many_requests/.)
2022/02/07 07:19:28 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:28 DEBUG : pacer: low level retry 7/10 (error too_many_requests/.)
2022/02/07 07:19:33 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:33 DEBUG : pacer: low level retry 8/10 (error too_many_requests/)
2022/02/07 07:19:36 DEBUG : Dropbox root 'Party1': Checking for changes on remote
2022/02/07 07:19:38 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:38 DEBUG : pacer: low level retry 9/10 (error too_many_requests/..)
2022/02/07 07:19:48 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 07:19:48 DEBUG : pacer: low level retry 10/10 (error too_many_requests/.)
2022/02/07 07:19:48 DEBUG : plot1: Dir.ReadDirAll error: too_many_requests/.
2022/02/07 07:19:48 DEBUG : plot1/: >ReadDirAll: item=-1, err=too_many_requests/.

ncw · February 7, 2022, 10:36am

Try adding --tpslimit 12 to your command to limit the number of requests per second rclone makes. If that doesn't work, try making the number smaller until it does.

plotterx · February 7, 2022, 10:42am

Hello Nick,
I've already tried that and it doesn't work even with --tpslimit 4.
Also I've asked some friends and they have same issue. I guess that is a general issue with dropbox. Can you guys who are the project contributors or who can reach the project contributors please check this?

ncw · February 7, 2022, 10:53am

Dropbox does quite a lot of rate limiting, that is for sure.

I suspect your account has been marked somehow as when I try a test I don't get rate limited.

Are you using the same account elsewhere at the same time? I suspect the rate limits are on a per account basis, not on a per connection basis.

plotterx · February 7, 2022, 10:59am

Maybe you can be right,
I don't use the same account. I use only one time an account. But There are too many space in it. each file is 100 GB and there is 900 files on each folder.

Last update:
They've answered my ticket:

Please note that our specialized team took a look in your issue and it appears like the issue is related to a third party application.
In order to resolve this, you will need to reach out to the developer of that application and they can also reach out to us.

ncw · February 7, 2022, 11:24am

I don't think there is anything I can fix in rclone. It is dropbox giving back rate limit errors that is the problem.

plotterx · February 7, 2022, 12:12pm

If you want, I can provide a connection rclone key for debugging? I guess there is a general problem with this.

Animosity022 · February 7, 2022, 12:49pm

Dropbox rate limits per application you have registered.

If you have an application that is using more than ~12 tps, you'll get rate limited.

You need to reduce the tps to less than 12 and you shouldn't run into issues.

You have a lot of options in your mount that doesn't do anything as well.

Usually best to simplify the mount, start with nothing and only add what you need if you have know what it's doing and why you have it.

That error makes it \ obvious the TPS limit is too high.

I use 2 mounts and and an upload script with each of them using their own registered app in the console so I can track each and limit each without worrying about them stepping on each other.

It's a bit more config up front, but since I've done that, I've never had a pacer issue.

Hunny · February 7, 2022, 12:53pm

The same problem started for me in the last 2-3 days.

Animosity022 · February 7, 2022, 12:55pm

This is already answered with a solution and a root cause.

If the solution doesn't work for you, start a new post and use the help / support template and fill it out completely.

Ole · February 7, 2022, 2:12pm

Hi @plotterx and @Hunny,

The below post seems relevant in your situation. It was written about OneDrive but the same behaviour and logic applies for Dropbox:

This may also interest you:

Animosity022 · February 7, 2022, 2:30pm

I've done quite the load testing/validation on the TPS limits for Dropbox and engaged support (who doesn't quite answer much unfortunately).

The general consensus is 12 tps per app registration:

dropbox: uploading improvements · Issue #5156 · rclone/rclone (github.com)

It's very easy to test/generate the error as rclone itself is really fast and hits it very fast if you have a mount with many directories. There is a sweet spot you hit when have too many directories on a mount as I didn't hit it at first either.

Ole · February 7, 2022, 3:13pm

@Animosity022 Seems like you missed my point. I am not questioning your observations and knowledge in this area - on the contrary.

I am trying to explain why services like Dropbox, OneDrive, Google Drive etc. all have this kind of limitations; and give some food for thought on Chia farming in general.

plotterx · February 7, 2022, 3:42pm

Hello,

Your point is so meaningful but I hope that's not the reason. I've already tried over an app without direct mount over rclone. But I got the same errors..

Are there any other solutions from you guys?

Also it gets the same error even with tps limit 1...

rclone mount dropbox4:/Party1 /dropbox4 --tpslimit 1 --allow-non-empty --log-level DEBUG
2022/02/07 16:48:25 INFO : Starting transaction limiter: max 1 transactions/s with burst 1
2022/02/07 16:48:25 DEBUG : rclone: Version "v1.58.0-beta.5989.aa2d7f00c" starting with parameters ["rclone" "mount" "dropbox4:/Party1" "/dropbox4" "--tpslimit" "1" "--allow-non-empty" "--log-level" "DEBUG"]
2022/02/07 16:48:25 DEBUG : Creating backend with remote "dropbox4:/Party1"
2022/02/07 16:48:25 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/02/07 16:48:26 DEBUG : Dropbox root '': Using root namespace "2201984529"
2022/02/07 16:48:27 DEBUG : fs cache: renaming cache item "dropbox4:/Party1" to be canonical "dropbox4:Party1"
2022/02/07 16:48:28 DEBUG : Dropbox root 'Party1': Mounting on "/dropbox4"
2022/02/07 16:48:28 DEBUG : : Root:
2022/02/07 16:48:28 DEBUG : : >Root: node=/, err=
2022/02/07 16:48:30 DEBUG : /: Lookup: name="plot1"
2022/02/07 16:48:31 DEBUG : /: >Lookup: node=plot1/, err=
2022/02/07 16:48:31 DEBUG : plot1/: Attr:
2022/02/07 16:48:31 DEBUG : plot1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2022/02/07 16:48:31 DEBUG : plot1/: ReadDirAll:
2022/02/07 16:48:34 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 16:48:34 DEBUG : pacer: low level retry 1/10 (error too_many_requests/..)
2022/02/07 16:48:34 DEBUG : pacer: Rate limited, increasing sleep to 5s
2022/02/07 16:48:36 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 16:48:36 DEBUG : pacer: low level retry 2/10 (error too_many_requests/.)
2022/02/07 16:48:41 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 16:48:41 DEBUG : pacer: low level retry 3/10 (error too_many_requests/)
2022/02/07 16:48:46 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 5 seconds.
2022/02/07 16:48:46 DEBUG : pacer: low level retry 4/10 (error too_many_requests/..)

Ole · February 7, 2022, 4:07pm

I know that once OneDrive starts the hard rate limiting (too many requests), then it takes some time until you are allowed a normal rate again - up to 24 hours. The less activity you have in this period, the shorter. I guess you will see something similar on Dropbox.

Sorry, others have been there before:

Animosity022 · February 7, 2022, 4:26pm

My comment was meant to be additive like yours

We have the help template for filling in things and getting information.

Did you register your own app? Are you using one app for the mount like I suggested? Can you see what the API usage is for that app?

What's the output of your rclone version?
What's your rclone.conf look like?

plotterx · February 7, 2022, 5:39pm

Did you register your own app?

Yes

Are you using one app for the mount like I suggested?

Yes I've registered an app to use that.

Can you see what the API usage is for that app?

I don't know how to check this.

What's the output of your rclone version?

rclone v1.58.0-beta.5989.aa2d7f00c
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-97-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.6
- go/linking: static
- go/tags: none

Animosity022 · February 7, 2022, 5:52pm

Developers - Dropbox - Click on App Console. Click the App you registered, click on the analytics.

If it's working, you'll see data:

If that's the only app registered to that, you should see 1 user registered against it and the API usage. If you are generate an error with 1 TPS, something is very wrong.

Post a full debug log with that mount.

Why are you using a beta version as well? If there isn't a specific reason, please use the stable.