Dropbox server side copy being rate limited - error too_many_write_operations

buifas · June 4, 2023, 12:46pm

What is the problem you are having with rclone?

I'm doing a server side copy from my old remote to new remote in the same dropbox account. This is to fix another issue related to long filenames.

The server side copy is being rate limited due to error too_many_write_operations/.

I tried reducing number of transfers to 1 and it seems to have fixed the issue, at the cost of extremely slow transfers. Since, I have a large number of small files, it's not really a solution.

(Note that I'm being rate limited with larger files as well >1Gb)

I also saw that there's an open issue to fix this with batch uploads, but I don't see any traction there since last 2 years. Are there any updates?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: debian 10.13 (64 bit)
- os/kernel: 4.19.0-22-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy --server-side-across-configs  oldcrypt: newcrypt: --tpslimit 5 --tpslimit-burst 5 --transfers 3

(well below the recommended TPS of 10, and default transfer of 5`

The rclone config contents with secrets removed.

[oldcrypt]
type = crypt
remote = dropbox:folder-old
password = <REDACTED>

[newcrypt]
type = crypt
remote = dropbox:folder-new
password = <REDACTED>
filename_encoding = base32768

A log from the command with the `-vv` flag

2023-06-04 14:42:14 INFO  : FILE1: Copied (server-side copy)
2023-06-04 14:42:16 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/...)
2023-06-04 14:42:16 DEBUG : pacer: Rate limited, increasing sleep to 91.227882ms
2023-06-04 14:42:17 DEBUG : pacer: Reducing sleep to 68.420911ms
2023-06-04 14:42:17 INFO  : FILE2: Copied (server-side copy)
2023-06-04 14:42:17 DEBUG : pacer: Reducing sleep to 51.315683ms
2023-06-04 14:42:17 INFO  : FILE3: Copied (server-side copy)
2023-06-04 14:42:19 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/...)
2023-06-04 14:42:19 DEBUG : pacer: Rate limited, increasing sleep to 102.631366ms
2023-06-04 14:42:19 DEBUG : pacer: Reducing sleep to 76.973524ms
2023-06-04 14:42:19 INFO  : FILE4: Copied (server-side copy)
2023-06-04 14:42:21 DEBUG : pacer: low level retry 2/10 (error too_many_write_operations/...)
2023-06-04 14:42:21 DEBUG : pacer: Rate limited, increasing sleep to 153.947048ms
2023-06-04 14:42:21 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/..)
2023-06-04 14:42:21 DEBUG : pacer: Rate limited, increasing sleep to 307.894096ms
2023-06-04 14:42:21 DEBUG : pacer: Reducing sleep to 230.920572ms
2023-06-04 14:42:21 INFO  : FILE5: Copied (server-side copy)
2023-06-04 14:42:24 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/...)
2023-06-04 14:42:24 DEBUG : pacer: Rate limited, increasing sleep to 461.841144ms
2023-06-04 14:42:24 DEBUG : pacer: low level retry 2/10 (error too_many_write_operations/..)
2023-06-04 14:42:24 DEBUG : pacer: Rate limited, increasing sleep to 923.682288ms
2023-06-04 14:42:24 DEBUG : pacer: Reducing sleep to 692.761716ms
2023-06-04 14:42:24 INFO  : FILE6: Copied (server-side copy)
2023-06-04 14:42:29 DEBUG : pacer: Reducing sleep to 519.571287ms
2023-06-04 14:42:29 INFO  : FILE7: Copied (server-side copy)
2023-06-04 14:42:29 DEBUG : pacer: Reducing sleep to 389.678465ms
2023-06-04 14:42:29 INFO  : FILE8: Copied (server-side copy)
2023-06-04 14:42:30 DEBUG : pacer: Reducing sleep to 292.258848ms
2023-06-04 14:42:30 INFO  : FILE9: Copied (server-side copy)
2023-06-04 14:42:32 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/)
2023-06-04 14:42:32 DEBUG : pacer: Rate limited, increasing sleep to 584.517696ms
2023-06-04 14:42:32 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/)
2023-06-04 14:42:32 DEBUG : pacer: Rate limited, increasing sleep to 1.169035392s
2023-06-04 14:42:32 DEBUG : pacer: Reducing sleep to 876.776544ms
2023-06-04 14:42:32 INFO  : FILE10: Copied (server-side copy)
2023-06-04 14:42:35 DEBUG : pacer: low level retry 2/10 (error too_many_write_operations/.)
2023-06-04 14:42:35 DEBUG : pacer: Rate limited, increasing sleep to 1.753553088s
2023-06-04 14:42:35 DEBUG : pacer: Reducing sleep to 1.315164816s
2023-06-04 14:42:35 INFO  : FILE11: Copied (server-side copy)
2023-06-04 14:42:37 DEBUG : pacer: Reducing sleep to 986.373612ms
2023-06-04 14:42:37 INFO  : FILE12: Copied (server-side copy)
2023-06-04 14:42:38 DEBUG : pacer: Reducing sleep to 739.780209ms
2023-06-04 14:42:38 INFO  : FILE13: Copied (server-side copy)

kapitainsky · June 4, 2023, 1:16pm

batch mode has been implemented long time ago:

interesting if it helps with server side operations.

kapitainsky · June 4, 2023, 1:24pm

Other thing if not in place yet is to get your own Dropbox App ID

nle · June 4, 2023, 2:24pm

I'm having the exact same issue. I've made separate app id's for the different crypts (or dropboxes that the crypt uses).

Tried a far bit, but nothing has changed.

This is what I settled on, and just let it run it's course.

rclone move db-crypt: db-crypt2: --tpslimit 12 --tpslimit-burst 0 --server-side-across-configs --log-level DEBUG --progress

Progress atm:

Transferred:        8.578 TiB / 10.012 TiB, 86%, 0 B/s, ETA -
Errors:                17 (retrying may help)
Checks:             27125 / 27129, 100%
Renamed:            27108
Transferred:            0 / 10008, 0%
Elapsed time:   15h9m34.5s
Checking:
 * <redacted>:  0% /6.876Ki, 0/s, -, moving
 * <redacted>:  0% /598, 0/s, -, moving
 * <redacted>:  0% /2.500Gi, 0/s, -, moving
 * <redacted>:  0% /50, 0/s, -, moving

Haven't checked out the errors yet, but it's most likely files that already exists.

This is my current log:

2023/06/04 16:20:37 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:37 DEBUG : pacer: Reducing sleep to 1.125s
2023/06/04 16:20:37 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:40 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/...)
2023/06/04 16:20:40 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/06/04 16:20:40 DEBUG : pacer: Reducing sleep to 1.5s
2023/06/04 16:20:40 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:40 DEBUG : pacer: Reducing sleep to 1.125s
2023/06/04 16:20:40 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:43 DEBUG : pacer: low level retry 2/10 (error too_many_write_operations/..)
2023/06/04 16:20:43 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/06/04 16:20:45 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/)
2023/06/04 16:20:46 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/...)
2023/06/04 16:20:48 DEBUG : pacer: low level retry 3/10 (error too_many_write_operations/.)
2023/06/04 16:20:49 DEBUG : pacer: low level retry 2/10 (error too_many_write_operations/...)
2023/06/04 16:20:50 DEBUG : pacer: Reducing sleep to 1.5s
2023/06/04 16:20:50 INFO  :<redacted>: Moved (server-side)
2023/06/04 16:20:52 DEBUG : pacer: low level retry 4/10 (error too_many_write_operations/..)
2023/06/04 16:20:52 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/06/04 16:20:52 DEBUG : pacer: Reducing sleep to 1.5s
2023/06/04 16:20:52 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:55 DEBUG : pacer: Reducing sleep to 1.125s
2023/06/04 16:20:55 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:57 DEBUG : pacer: low level retry 1/10 (error from_write/too_many_write_operations/...)
2023/06/04 16:20:57 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/06/04 16:20:58 DEBUG : pacer: Reducing sleep to 1.5s
2023/06/04 16:20:58 INFO  : <redacted>: Moved (server-side)
2023/06/04 16:20:58 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/..)
2023/06/04 16:20:58 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/06/04 16:21:00 DEBUG : pacer: low level retry 2/10 (error from_write/too_many_write_operations/)
2023/06/04 16:21:00 DEBUG : pacer: Reducing sleep to 1.5s
2023/06/04 16:21:00 INFO  : <redacted>: Moved (server-side)

Animosity022 · June 4, 2023, 4:09pm

Dropbox doesn't share any specifics on exactly how it does rate limiting so it's a true guessing game.

As noted, you do want a separate client ID/secret for each thing you have going on. I have one for each mount, one for when I do copies and one I just use for testing.

For copying/mass updates, they seem to throttle more as I never get any issues on my mounts when playing or letting things normally upload.

I do get some issues if I mass rename in Sonarr/Radarr, but they happen not that often.

Best bet is start too small and slowly work your way up to find the sweet spot for your use case.

buifas · June 4, 2023, 7:11pm

Thanks for the replies.

As suggested, I updated the API keys and enabled the batch-mode option. While the issue didn't go away entirely, it mitigated the error to the point where I was able to finish the server-side-copy in an hour.

Since, it was a one time thing, I'm okay with this.

nle · June 4, 2023, 8:26pm

@buifas What settings did you use?

buifas · June 4, 2023, 8:32pm

Updated API keys and,

rclone copy --server-side-across-configs  oldcrypt: newcrypt: --tpslimit 10 --tpslimit-burst 10 --transfers 5 --dropbox-batch-mode sync

I was still being rate limited but way less than before. I didn't have the patience to play around the settings as this was a one-time thing. However, I have a feeling I'll need to revisit this as I move my entire setup and scripts to dropbox.