Dear rclone community,
We consider to replace parallel rclone calls by a single call using the new combine feature. Let me briefly explain our setting:
We do have a DART security application called Dido. Dido splits new files into chunks and stores these chunks in different cloud storage systems. These chunks contain all necessary information to reconstruct the original files, the version history, type of encryption if used...
Say we have 3 new files created locally
blackforest.jpg
highlands.jpg
alps.jpg
Dido splits the files apart and uses rclone move to transfer the particle files to different cloud storage systems (particle file names uses hashes in the real Dido application and for demonstration purpose I use 2 digit file names here, particle files have different content in different remotes event though they are named equally when belonging to the same original file):
blackforest.jpg -> split -> rclone move
12.zip on gdrive
12.zip on hidrive
12.zip on dropbox
highlands.jpg -> split -> rclone move
77.zip on gdrive
77.zip on hidrive
77.zip on dropbox
It is important that all particles of the newest file (blackforest.jpg) are transferred first to gdrive, hidrive, dropbox, then all particle files of the second newest file highlands.jpg ...
Dido uses 3 DART isolates (like threads) to rclone move the particle files in parallel, using --order-by modtime.
This works fine. But when I've been reading in the release notes about how combine works, I've been wonder if I might replace these three parallel rclone move calls by just one.
I did some tests and calling rclone move with a combine setting respects the --order-by modtime flag. In my example
local/2gdrive/12.zip -> gdrive:
local/2gdrive/77.zip -> gdrive:
local/2hidrive/12.zip -> hidrive:
local/2hidrive/77.zip -> hidrive:
local/2dropbox/12.zip -> dropbox:
local/2dropbox/77.zip -> dropbox:
works as required
first
local/2gdrive/12.zip -> rclone move -> gdrive:
local/2hidrive/12.zip -> rclone move -> hidrive:
local/2dropbox/12.zip -> rclone move -> dropbox:
then
local/2gdrive/77.zip -> rclone move -> gdrive:
local/2hidrive/77.zip -> rclone move -> hidrive:
local/2dropbox/77.zip -> rclone move -> dropbox:
This is great and I consider to simply my DART code replacing rclone calls in isolates by a single call using the rclone combine feature. Thus letting Dido do the heavy lifting and keeping my own application as lightweight as possible.
I did not build a fully fledged test environment but I assume, that using combine would be less memory demanding and probably bit slower in transferring data. So rclone using isolates would be best in a server environment and rclone combine in the desktop use case.
Any comments on this from rclone insighters? Would be highly appreciated.
Harald
