Dropbox: Too many requests or write operations. Trying again in 15 seconds

[arajin]

Even when using the latest rclone stable with --transfers 1 --checkers=20 --tpslimit 1 still get too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds. after just a few files have been transferred.
Really annoying

[m4n63 (Magnus)]

@ncw

@m4n63

Or would it give value if I were to chip in and help with some specific testing of it?

Yes please more testing would be great!

It is very much faster, and if you use --dropbox-batch-mode async even faster (with less guarantees). See this comment on the github issue for performance testing and pointers to the docs.

So... we have been using the v1.55.0-beta.5334.ac7acb9f0 binary for a few days now, uploading files to Dropbox via the "sync" command, and I thought I would share our experiences.
The flags we started with are:
--progress --log-file=rclone-batch.log --log-level DEBUG --fast-list --checksum --dropbox-batch-mode=async --dropbox-batch-size 1000 --dropbox-batch-timeout 10s --checkers 28 --transfers 28 --tpslimit 12 sync SRC DST

We directly observed a massive improvement in speed, but also that the command could simply error out and die on some cases without any real error message. In some cases rclone died right away, and in some cases after a bit of time. I think this was typically seen when memory was limited on the server where it was run, and when this happened, we reduced the amount of --checkers and --transfers, and the problem went away. (and sorry, no logfiles or dumps available to share on these cases).
So we ended up reducing the checkers and transfers to 4, but still saw the same huuuge improvement in speed.
So very positive in regard to this added option, it has worked well for us the last couple of days.

[arajin]

I've been using the latest version with --transfers 4 --checkers=4 --dropbox-batch-mode async. It sometimes freezes but seems to work fine. Tried it with --transfers 8 --checkers=8 --dropbox-batch-mode async but seems like it uses way more memory since my VPS froze

[ncw (Nick Craig-Wood)]

@m4n63

We directly observed a massive improvement in speed, but also that the command could simply error out and die on some cases without any real error message. In some cases rclone died right away, and in some cases after a bit of time. I think this was typically seen when memory was limited on the server where it was run,

Note that dropbox uploads use 48M of memory per large file transfer by default which can add up.

@m4n63

So very positive in regard to this added option, it has worked well for us the last couple of days.

Great

@arajin

I've been using the latest version with --transfers 4 --checkers=4 --dropbox-batch-mode async. It sometimes freezes but seems to work fine. Tried it with --transfers 8 --checkers=8 --dropbox-batch-mode async but seems like it uses way more memory

It would be nice to have some numbers on how much memory rclone is using. It is possible that the new code has a memory leak.

Check out the docs here on memory leaks: Remote Control / API

[PatriotM]

as soon as this message comes up "check failed: internal_error" it doesn't upload anything anymore, the upload speed counts down slowly but the transfer remains at 0mb per bmon. as soon as I kill the process and start rclone move again, it works wonderfully again.

2021-04-15 15:14:31 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Batch idle for 500ms so committing
2021-04-15 15:14:31 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Committing sync batch length 7
2021-04-15 15:14:31 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 100ms to wait for batch to complete: "in_progress": try 1/120
2021-04-15 15:14:32 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 200ms to wait for batch to complete: "in_progress": try 2/120
2021-04-15 15:14:32 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 400ms to wait for batch to complete: "in_progress": try 3/120
2021-04-15 15:14:33 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 800ms to wait for batch to complete: "in_progress": try 4/120
2021-04-15 15:14:34 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 5/120
2021-04-15 15:14:35 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 6/120
2021-04-15 15:14:36 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 7/120
2021-04-15 15:14:38 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 8/120
2021-04-15 15:14:39 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 9/120
2021-04-15 15:14:40 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 10/120
2021-04-15 15:14:42 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 11/120
2021-04-15 15:14:43 DEBUG : Dropbox root 'DROPBOX/MYFOLDER': Sleeping for 1s to wait for batch to complete: "in_progress": try 12/120
2021-04-15 15:14:44 ERROR : Dropbox root 'DROPBOX/MYFOLDER': sync batch commit: failed to commit batch length 7: wait for batch completion: check failed: internal_error/.
Transferred: 1.321T / 1.871 TBytes, 71%, 28.314 MBytes/s, ETA 5h39m37s

[ncw (Nick Craig-Wood)]

@PatriotM

as soon as this message comes up "check failed: internal_error" it doesn't upload anything anymore, the upload speed counts down slowly but the transfer remains at 0mb per bmon. as soon as I kill the process and start rclone move again, it works wonderfully again.

That message is from dropbox...

Can you try the latest beta from the branch which is a bit better at retrying failed batches

[PatriotM]

it looks like it's going well!

I had uploaded almost 4TB in 27 hours without interruption!

all that's missing now is a fix for cutting off the first characters
very good work you/they are doing here

[ncw (Nick Craig-Wood)]

Thanks for testing We'll fix the cutoff chars soon I'm sure! Here is the issue #5165 for anyone playing along at home.

[ncw (Nick Craig-Wood)]

I think I've managed to fix the changenotify problem - can you give this a try?

v1.56.0-beta.5412.bba3e3e4f.fix-5165-dropbox-changenotify on branch fix-5165-dropbox-changenotify (uploaded in 15-30 mins)

[PatriotM]

it work flawless! seems to be all fixed now

Transferred: 5.316T / 5.399 TBytes, 98%, 47.528 MBytes/s, ETA 30m15s
Errors: 153 (retrying may help)
Checks: 248540 / 248554, 100%
Deleted: 124197 (files), 0 (dirs)
Renamed: 123736
Transferred: 123736 / 127955, 97%
Elapsed time: 32h35m4.0s

[ncw (Nick Craig-Wood)]

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.55.1

[bpj (Benct Philip Jonsson)]

This thread is huge and I haven't read all of it. Apologies it this already has been addressed.

I still have this problem. I am on Termux so it may be that their package isn't up to date.
Does anyone know?

$ rclone --version
rclone 1.55.1-termux
- os/type: android
- os/arch: arm64                                - go/version: go1.16.3
- go/linking: dynamic
- go/tags: noselfupdat

[ncw (Nick Craig-Wood)]

@bpj the fix for the changenotify problem went into 1.55.1 - if you are talking about the too many requests problem then try the latest beta - you'll find an android version in the testbuilds directory.

[bpj (Benct Philip Jonsson)]

@ncw yes I'm talking about the too many requests problem. Which test build should I use? rclone --version says os/arch arm64 but none of the file names matches that exactly. Sorry, I'm a total idiot when it comes to modern hardware! Also, can I just extract it and move it to my ~/bin directory?

[m4n63 (Magnus)]

I use this one:
v1.55.0-beta.5334.ac7acb9f0.fix-dropbox-batch-sync
And yes, just extract and run it.

[ncw (Nick Craig-Wood)]

@bpj

I'm talking about the too many requests problem. Which test build should I use? rclone --version says os/arch arm64 but none of the file names matches that exactly. Sorry, I'm a total idiot when it comes to modern hardware!

This is the one you want: https://beta.rclone.org/v1.56.0-beta.5513.c0cda087a/testbuilds/rclone-android-21-armv8a.gz

@bpj

Also, can I just extract it and move it to my ~/bin directory?

You'll need to download it, gunzip it then it should be ready to run. You can move it into ~/bin and rename it if you want.

@m4n63

I use this one:
v1.55.0-beta.5334.ac7acb9f0.fix-dropbox-batch-sync
And yes, just extract and run it.

That code is now merged to the latest beta

[arajin]

Have there been changes in the beta compared to this v1.56.0-beta.5394.d16702566.fix-dropbox-batch-sync version? I've been using the beta for the last couple of days and it freezes way more and more often

[ncw (Nick Craig-Wood)]

@arajin

Have there been changes in the beta compared to this v1.56.0-beta.5394.d16702566.fix-dropbox-batch-sync version?

The dropbox backend code should be the same.

@arajin

I've been using the beta for the last couple of days and it freezes way more and more often

What sort of freezes?

[arajin]

It freezes while transferring, have to stop it or wait a very long time for it to keep transferring