Replacing parallel rclone calls to different remotes by new combine feature? Comments by rclone experts welcome

Dear rclone community,

We consider to replace parallel rclone calls by a single call using the new combine feature. Let me briefly explain our setting:

We do have a DART security application called Dido. Dido splits new files into chunks and stores these chunks in different cloud storage systems. These chunks contain all necessary information to reconstruct the original files, the version history, type of encryption if used...

Say we have 3 new files created locally

  blackforest.jpg
  highlands.jpg
  alps.jpg

Dido splits the files apart and uses rclone move to transfer the particle files to different cloud storage systems (particle file names uses hashes in the real Dido application and for demonstration purpose I use 2 digit file names here, particle files have different content in different remotes event though they are named equally when belonging to the same original file):

blackforest.jpg -> split -> rclone move

    12.zip on gdrive
    12.zip on hidrive
    12.zip on dropbox

highlands.jpg -> split -> rclone move

    77.zip on gdrive
    77.zip on hidrive
    77.zip on dropbox

It is important that all particles of the newest file (blackforest.jpg) are transferred first to gdrive, hidrive, dropbox, then all particle files of the second newest file highlands.jpg ...

Dido uses 3 DART isolates (like threads) to rclone move the particle files in parallel, using --order-by modtime.

This works fine. But when I've been reading in the release notes about how combine works, I've been wonder if I might replace these three parallel rclone move calls by just one.

I did some tests and calling rclone move with a combine setting respects the --order-by modtime flag. In my example

  local/2gdrive/12.zip -> gdrive:
  local/2gdrive/77.zip -> gdrive:

  local/2hidrive/12.zip -> hidrive:
  local/2hidrive/77.zip -> hidrive:

  local/2dropbox/12.zip -> dropbox:
  local/2dropbox/77.zip -> dropbox:

works as required

first

  local/2gdrive/12.zip -> rclone move -> gdrive:
  local/2hidrive/12.zip -> rclone move -> hidrive:
  local/2dropbox/12.zip -> rclone move -> dropbox:

then

  local/2gdrive/77.zip -> rclone move -> gdrive:
  local/2hidrive/77.zip -> rclone move -> hidrive:
  local/2dropbox/77.zip -> rclone move -> dropbox:

This is great and I consider to simply my DART code replacing rclone calls in isolates by a single call using the rclone combine feature. Thus letting Dido do the heavy lifting and keeping my own application as lightweight as possible.

I did not build a fully fledged test environment but I assume, that using combine would be less memory demanding and probably bit slower in transferring data. So rclone using isolates would be best in a server environment and rclone combine in the desktop use case.

Any comments on this from rclone insighters? Would be highly appreciated.

Harald

You are only running 1 rclone binary rather than 3 so yes, less memory demanding.

Rclone will transfer things in parallel using up to --transfers at once. I suspect it won't be any slower.

Sounds like a good plan to me :slight_smile:

Thank you Nick,
Harald

1 Like