Drobox batch mode behavior question

When using the async batch mode option (--dropbox-batch-mode async) on an rclone mount using --vfs-cache-mode minimal, does this mean that rclone will never check whether the file upload completed at all?

Background: I'm exploring backing a proxmox-backup-server repository (creates lots of <4MB objects) with an rclone mount (likely using --vfs-cache-mode minimal) and considering options for the mount with hopefully better performance characteristics than ~2 objects/sec.

I notice that the current performance blocker appears to be the rate at which dropbox will commit new objects. Async batch mode eliminates this blocker, so I'd love to use it provided there'd be some detection (and ideally retry) of failed uploads from the mount.

I have also considered using writethrough mode; however, writethrough mode doesn't seem to respect --vfs-cache-max-size and I don't have a way to manage a potentially unbounded cache size. Ideally, writethrough would buffer objects until the cache size limit, then behave more like writeback until back under the cache size limit.

Away from the correctness question on the async batch mode, any other suggestions for alternative approaches to improve the new object commit rate would be appreciated.

Thank you!

There will be an error in the log but rclone won't retry the file as the caller will have returned at that point.

You'll need sync mode to guarantee the uploads. You can experiment with the batch size and the delay - these were both optimized for rclone sync and the VFS cache will need different parameters.

I think with a bit of tweaking you should be able to get good performance.

Try --dropbox-batch-timeout 5s (note the docs are wrong (they've been fixed but the fix isn't live yet) the default for this in sync mode is 500ms)

Also you may want to increase --transfers as this is obeyed by the VFS and the Dropbox batch code.

Thank you very much. I'm still testing this, but your pointers were very helpful!

1 Like