Rclone does not seem to use the Dropbox move_batch_v2 command

Feature
1

What is the problem you are having with rclone?

Rclone does not seem to use the move_batch_v2 API command for Dropbox. At least when copying from one crypt to another.

Instead it seems to try to move each individual file, resulting in rate limiting and lots of retries due to too_many_write_operations. Without batch renaming/moving, the rate limit is around 1 operation/second. Enabling it could provide a 100x speed increase.

I'm not sure if it's possible to use this command, but it seems like it should be.

The use case here is re-encrypting existing files to a different format/using a different key.

This happens when using my own app ID too.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: centos 7.9.2009 (64 bit)
  • os/kernel: 3.10.0-1160.42.2.el7.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox + Crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy dropbox-crypt-1: dropbox-crypt-2: ----delete-empty-src-dirs --server-side-across-configs --transfers=5 --dropbox-batch-mode async  --dropbox-batch-size 1000 --dropbox-batch-timeout 1s --checkers 1 --tpslimit 12 --tpslimit-burst 0

The rclone config contents with secrets removed.

[dropbox]
type = dropbox
token = ***

[dropbox-crypt-1]
type = crypt
remote = dropbox:foldername
password = ***
password2 = ***

[dropbox-crypt-2]
type = crypt
remote = dropbox:foldername2
filename_encoding = base32768
password = ***
password2 = ***

A log from the command with the -vv flag

The only relevant part of the log is a lot of:

2023-07-29 07:16:47 DEBUG : file.ext: Need to transfer - File not found at Destination
2023-07-29 07:16:47 DEBUG : pacer: low level retry 1/10 (error too_many_write_operations/)
2023-07-29 07:16:47 DEBUG : pacer: Rate limited, increasing sleep to 1.062334034s
2023-07-29 07:16:48 DEBUG : pacer: Reducing sleep to 796.750525ms
2023-07-29 07:16:48 INFO  : file.ext: Moved (server-side)
1 Like
2

I also have found that dropbox is really bad with rclone for large folders even if you are just doing checks and not transfering anything at all

3

You are right, it doesn't. It only uses batches for uploading files at the moment.

The batcher seems to follow roughly the same form as the upload batcher but it isn't the same.

I also had a request for a delete batcher in github issues. Maybe you could put a comment on that issue about move batching?

I wonder if the current batching code can be made generic to work with all 3. The batching code is quite complicated, as the internal architecture of rclone wasn't really made for batching, so it would be nice not to duplicate it.

dropbox also seems to support batching for

  • copy
  • mkdir

So maybe those could be included too. Though mkdir needs to be done before other things so probably not that one.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.