Dropbox: Too many requests or write operations. Trying again in 15 seconds

Help and Support
1

What is the problem you are having with rclone?

I'm getting this message:
Too many requests or write operations. Trying again in 15 seconds. I started facing this problem a couple of days ago, happened too with the previous version of rclone but only seems to happen with small files (in this case several pictures). When I try to upload bigger files (a few hundred megabytes) I don't get the message.

What is your rclone version (output from rclone version)

1.55.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bits

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy gallery-dl/ {dropbox}: -P

The rclone config contents with secrets removed.

[dropbox]
type = dropbox
token = {"access_token":"xyz","token_type":"bearer","refresh_token":"xyz","expiry":"2021-04-05T23:29:02.0511076-05:00"}

A log from the command with the -vv flag

2021/04/06 00:53:36 DEBUG : Using config file from "/app/.config/rclone/rclone.conf"
2021/04/06 00:53:36 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "copy" "gallery-dl/" "dropbox:" "-P" "-vv"]
2021/04/06 00:53:36 DEBUG : Creating backend with remote "gallery-dl/"
2021/04/06 00:53:36 DEBUG : fs cache: renaming cache item "gallery-dl/" to be canonical "/app/gallery-dl"
2021/04/06 00:53:36 DEBUG : Creating backend with remote "dropbox:"
2021-04-06 00:53:37 DEBUG : Dropbox root '': Waiting for checks to finish
2021-04-06 00:53:37 DEBUG : Dropbox root '': Waiting for transfers to finish
2021-04-06 00:53:39 DEBUG : xyz.jpg: DropboxHash = xyz OK
2021-04-06 00:53:39 INFO  : zzz.jpg: Copied (new)
2021-04-06 00:53:39 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 15 seconds.
2021-04-06 00:53:39 DEBUG : pacer: low level retry 1/1 (error too_many_requests/...)
2021-04-06 00:53:39 DEBUG : pacer: Rate limited, increasing sleep to 15s
2021-04-06 00:53:39 DEBUG : asd.jpg: Received error: upload failed: too_many_requests/... - low level retry 1/10
2021-04-06 00:53:39 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
2021-04-06 00:53:39 DEBUG : pacer: low level retry 1/1 (error too_many_requests/.)
2021-04-06 00:53:39 DEBUG : cvf.jpg: Received error: upload failed: too_many_requests/. - low level retry 1/10
2021-04-06 00:53:39 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
2021-04-06 00:53:39 DEBUG : pacer: low level retry 1/1 (error too_many_requests/.)
2021-04-06 00:53:39 DEBUG : zyx.jpg: Received error: upload failed: too_many_requests/. - low level retry 1/10
2021-04-06 00:53:40 DEBUG : pacer: Reducing sleep to 11.25s
2021-04-06 00:53:40 DEBUG : opq.jpg: DropboxHash = xyz OK
2021-04-06 00:53:40 INFO  : lll.jpg: Copied (new)
2021-04-06 00:53:40 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 15 seconds.
2021-04-06 00:53:40 DEBUG : pacer: low level retry 1/1 (error too_many_requests/)
2021-04-06 00:53:40 DEBUG : pacer: Rate limited, increasing sleep to 15s
2021-04-06 00:53:40 DEBUG : eee.jpg: Received error: upload failed: too_many_requests/ - low level retry 1/10
2

hello and welcome to the forum,

make sure to do this
https://rclone.org/dropbox/#get-your-own-dropbox-app-id

might tweak
https://rclone.org/docs/#tpslimit-float

3

Is the App key supposed to be the client_id and the App secret the client_secret?

4

Hi,

I have since a few days exactly the same problems! in the hope that the app and secret key would bring something. unfortunately this is not the case. i have created the "id" on the url and added both to the rclone config. however without any success. in addition, I get since about the 03.04.2021 no more speed, now I do not know if it is the routing. I fluctuate between 2-10mb, before I always had around 60mb/s was there somehow a change internally at dropbox? since when is there the Dropbox App ID ?

Transferred: 842.982M / 1.140 GBytes, 72%, 1.998 MBytes/s, ETA 2m42s
Checks: 136 / 144, 94%
Deleted: 68 (files), 0 (dirs)
Renamed: 68
Transferred: 68 / 103, 66%
Elapsed time: 7m32.7s
Checking:

2021-04-06 03:32:56 DEBUG : pacer: Reducing sleep to 1.501693725s
2021-04-06 03:32:58 DEBUG : pacer: Reducing sleep to 1.126270293s
2021-04-06 03:32:59 DEBUG : pacer: Reducing sleep to 844.702719ms
2021-04-06 03:33:00 DEBUG : pacer: Reducing sleep to 633.527039ms
2021-04-06 03:33:01 DEBUG : pacer: Reducing sleep to 475.145279ms
2021-04-06 03:33:02 DEBUG : pacer: Reducing sleep to 356.358959ms
2021-04-06 03:33:02 DEBUG : pacer: Reducing sleep to 267.269219ms
2021-04-06 03:33:03 DEBUG : pacer: Reducing sleep to 200.451914ms
2021-04-06 03:33:03 DEBUG : pacer: Reducing sleep to 150.338935ms
2021-04-06 03:33:03 DEBUG : pacer: Reducing sleep to 112.754201ms
2021-04-06 03:33:03 DEBUG : pacer: Reducing sleep to 84.56565ms
2021-04-06 03:33:04 DEBUG : pacer: Reducing sleep to 63.424237ms
2021-04-06 03:33:04 DEBUG : pacer: Reducing sleep to 47.568177ms
2021-04-06 03:33:04 DEBUG : pacer: Reducing sleep to 35.676132ms
2021-04-06 03:33:04 DEBUG : pacer: Reducing sleep to 26.757099ms
2021-04-06 03:33:04 DEBUG : pacer: Reducing sleep to 20.067824ms
2021-04-06 03:33:04 DEBUG : pacer: Reducing sleep to 15.050868ms
2021-04-06 03:33:05 DEBUG : pacer: Reducing sleep to 11.288151ms

5

for the rclone config.

[YOURNAME]
type = dropbox
token = {"access_token":"YOUR.TOKEN","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}
app_key = YOUR.KEY
app_secret = YOUR.SECRET

but it looks like it doesn't quite work for me, it said "Attempt 3/3 failed with 1 errors and: missing_scope/..."

6

Does the "about" command work for you? I get Failed to about: About call failed: about failed: missing_scope/..

7

with the command "rclone move" i have renewed the token after the scope flags now the upload/move works again but just as slow as before.

i have tried all constellations with --transfers --checkers --drive-chunk-size it just doesn't get more than a maximum of 15mb/s

i have also been switching back and forth between versions. i have a feeling that the problem is on dropbox's side.

i have tried it from several servers with 1 gbit uplink everywhere is the same result

8

For what its worth, I have also been struggling for some time with getting more than ~15MB/s when uploading to Dropbox. Using --tpslimit 13 is generally recommended for Dropbox uploads I believe to try and work around the pacer. (I get a lot more speed downloading). I believe it is the Dropbox API that limits this. Interesting that you got ~60MB/s earlier, mind sharing your settings for that speed?

9

it is well noted no small files all files are over 20mb in size

rclone move /my/path dropbox:/ --ignore-existing -c --transfers=8 --checkers=8 --delete-after --min-age 1200m --tpslimit=16

otherwise I have only adjusted the linux. what I use for years and runs well

TCP BBR

Blockquote

Create backup.

if [ ! -e /mnt/speed_backup.sh ] ; then
echo -n "Creating backup (/mnt/speed_backup.sh)... "
echo "#!/bin/bash" > /mnt/speed_backup.sh
echo "" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/ipv4/tcp_sack" > /proc/sys/net/ipv4/tcp_sack" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/ipv4/tcp_timestamps" > /proc/sys/net/ipv4/tcp_timestamps" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/ipv4/tcp_mem" > /proc/sys/net/ipv4/tcp_mem" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/ipv4/tcp_rmem" > /proc/sys/net/ipv4/tcp_rmem" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/ipv4/tcp_wmem" > /proc/sys/net/ipv4/tcp_wmem" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/core/optmem_max" > /proc/sys/net/core/optmem_max" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/core/rmem_default" > /proc/sys/net/core/rmem_default" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/core/rmem_max" > /proc/sys/net/core/rmem_max" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/core/wmem_default" > /proc/sys/net/core/wmem_default" >> /mnt/speed_backup.sh
echo "echo "cat /proc/sys/net/core/wmem_max" > /proc/sys/net/core/wmem_max" >> /mnt/speed_backup.sh
chmod 744 /mnt/speed_backup.sh
echo -e "\tdone!"
else
echo "Backup found (pwd/speed_backup.sh). Skipping creation of one."
fi

Boost buffer settings

echo -n "Boosting... "
echo "1048576" > /proc/sys/net/core/optmem_max
echo "20971520" > /proc/sys/net/core/wmem_max
echo "20971520" > /proc/sys/net/core/rmem_max
echo "8192 2621440 10485760" > /proc/sys/net/ipv4/tcp_wmem
echo "8192 2621440 10485760" > /proc/sys/net/ipv4/tcp_rmem
echo "1" > /proc/sys/net/ipv4/tcp_timestamps
echo "1" > /proc/sys/net/ipv4/tcp_sack
echo "1" > /proc/sys/net/ipv4/tcp_window_scaling
echo "3129344 3137536 3145728" > /proc/sys/net/ipv4/tcp_mem
echo -e "\tdone!"

1 Like
10

that is a known issue here at the forum.
so search around for other posts.

11

that is for gdrive, not dropbox. so you can remove that.

12

to all posters, make sure you have done this
https://rclone.org/dropbox/#get-your-own-dropbox-app-id

13

We are currently trying to fix this here

which uses dropbox's batch mode which helps enormously with lots of small files.

On that issue there is a binary to test.

With this binary I can upload as many small files as I want.

1 Like
14

Tried the v1.55.0-beta.5332.d732cf2e9.fix-dropbox-batch-sync version with sync and asyc. Sync is as slow as the stable version, however async is like three of four times faster though it froze a couple of times for like three seconds

15

the speed has directly doubled with move! it also starts uploading the files much sooner compared to the 1.55 stable version but now i get this message from rclone every now and then back in debug -vv

2021/04/07 03:38:59 NOTICE: Encrypted drive 'dropbox:/': ChangeNotify was unable to decrypt "ocaecglelm1sajd2ivmce1u8vc5bu4meppnhf1ip3bqm19m7g/krmfmos20d7sav1b2qc2q1thik/q9fu66rbbakhselpf21557v1jk/a90n9pk4glged0298503trcko188fievu5vjte18jjn9tqu1mjtn4k8smia5v8nr8oc39oqjdjrb8od3s6rccta3582859h6v9ktvqg/v2ts4dd0ld68ahg98n5r9en7gqtfc8avnjukcr6gr1jm2bc5ep9c7b544jci2nkq8tr8op4i69lfel63t6vo4ofns47dmff24fm5ev0": illegal base32 data at input byte 49

note:

it looks like it can't find the file path because 2 characters are cut off at the beginning cutted: jc0634nr7j3ldsmqmtl23v5s vs org:

dropbox
dropbox2051×731 613 KB
1gjc0634nr7j3ldsmqmtl23v5s

16

It should be the same speed or faster but it will never give the retrying in 15 seconds warnings.

Async is blazing fast, yes! At the cost of a bit of possibly integrity but nothing that rclone check won't pick up.

Great

That is dropbox: Unable to decrypt returned paths from changeNotify · Issue #5165 · rclone/rclone · GitHub - your insight there would be appreciated.

17

I've been testing it and noticed that sometimes I get this message
2021/04/07 19:29:15 ERROR : Dropbox root '': async batch commit: failed to commit batch length 17: batch had 17 errors: last error: too_many_write_operations
And rclone says that all files have been transferred but I don't see them in my Dropbox. Tried it twice but nothing was copied, tried it again and this time the files were copied. The output with the -vv flag of the third (successful) try is:

I tried a new transfer this time I used the -vv flag on the first try and this was the output:

At that point rclone printed it had copied 90 / 126 files, 490.995M / 534.090 MBytes 92%, at a speed of 4.771 MBytes/s but again nothing appeared on my Dropbox

18

Hmm it looks like Dropbox returned too_many_write_operations when rclone was querying the batch status so it gave up trying to commit the batch. It probably should have carried on trying.

Does it do this if you use sync batch mode?

19

This is the output of using sync:

At that point the transfer was:
Transferred: 122.044M / 534.090 MBytes, 23%, 1.125 MBytes/s, ETA 6m6s
Errors: 15 (retrying may help)
Transferred: 29 / 111, 26%
Elapsed time: 1m49.4s

This time those 29 files were actually transferred to my dropbox

20

@arajin can you try this - it should try harder to complete the batch. If it fails, please send a debug log with -vv

v1.55.0-beta.5333.d65916f9a.fix-dropbox-batch-sync on branch fix-dropbox-batch-sync (uploaded in 15-30 mins)