Delta-transfer algorithm (rsync style) possible?

Hello everyone,

First off, I use rclone for years now, it's really a great tool, as you all know it : ) But since a few months, I use it to sync local smb shares too. I used rsync in the past, but I had to ditch it for some reasons. So you see my question coming, but would it be possible for rclone to implement a delta-transfer algorithm, so rclone only move the differences between 2 files, and not the whole (newer) file ? I don' doubt it was asked before, but I don't find another topic dedicated about that...

Thx :slight_smile:

IMHO it does not make much sense:

  1. most cloud providers do not allow to write in the middle of the file. So you could maybe do something for e.g. local remote - but then it is reinventing the wheel as there is rsync.

  2. When point 1 is not valid proper delta transfer requires server/client setup - it implies you can install your software on both ends. If you can rclone you can also rsync.

What rsync does (or not) that rclone could do better in your opinion?

The key rclone feature is "rsync" style connectivity to various cloud storage providers with icing on the cake like "virtual remotes" (crypt, chunker etc.) and some server style features (rclone serve/rclone rc). It does it very well but still there are rough edges to iron here and there. It would be waste of limited resources IMHO to try to replicate functionality of already existing an very mature tool like rsync itself.

Well, Rsync is not natively available for windows systems (or i'm wrong ?). I know you can use wsl but it has lmitations.

Next, it would be interesting IF rclone can avoid the client/server setup. Right now I'm using hasher, so I cache the md5 results, and the sync between 2 shares are pretty quick. I know you can kind of do remote / remote with rsync, but it's a hack more than anything. With rclone I can move the "compute" part on a machine with a decent cpu, which is not always the case with rsync if you use it between some arm or mips nas for exemple.

If it's not possible, I get it, I'm just asking if it is :slight_smile:

1 Like

Blame microsoft:) You have now WSL so you can use rsync.

Of course I am not saying I am 100% right - just my view. It is always good people look for better ways:)

As said, I know wsl is here, but it's not perfect, and rsync is not made for remote / remote sync (even if you can do it with some hack, it's not clean at all). So for me, it's not a solution, and rclone wouldn't be redondant for that.

you can do one side delta "transfers" but it requires maintaining database of chunks info. This is something backup tools do - restic, kopia, borg etc. Still I think it is asking for rclone to do something it was not designed to do - and I believe in UNIX motto of doing one thing well - and for this rclone is already dangerous hybrid. Yes maybe it does not apply to windows:)

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.