Rsync vs rclone verification (md5sum slow)

Help and Support
1

What is the problem you are having with rclone?

The md5sum verification is very slow. I am transferring from a remote host to a local drive. I had been using rsync, but have been trying rclone because it supports multiple simultaneous transfers. Based on observations and reading docs, I see that rclone computes an md5sum after a transfer is complete. This is extremely slow... and basically nullifies the overall transfer rate increase I get by running multiple transfers. Is there any configuration change I can make to improve this behavior? According to rsync's man page, it also performs some hash-based verification:

Note  that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check.

But rsync does not hang at 100% completion for several minutes as it generates a hash. I guess in light of the above (ie that rsync calculates hash in parallel with the transfer), and observing that rclone just calls md5sum at the end, is this just a fundamental limitation of rclone?

(Actually I don't even fully understand the excerpt from the rsync page, seemingly you'd have to wait until the file is fully transferred to compute a hash on the destination. Unless you're hashing individual chunks.)

Anyway I've tried with and without --checkers=1 with no real improvement. Curious as to why rsync has apparently requires no extra time for checking/hashing, and if I can make rclone behave similarly without just --ignore-checksum. Also note that in comparing to rsync, I'm using the exact same remote source and local destination (host and path).

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.1

  • os/version: arch (64 bit)
  • os/kernel: 6.13.6-arch1-1 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.24.0
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

linux host via sftp

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -Pvv --checkers=1 --log-file=log.log --files-from my-files.txt myremote:some/path /some/local/path

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[rev]
type = sftp
host = XXX
user = XXX
port = 12345
key_use_agent = true
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command that you were trying to run with the -vv flag

N/A

2

welcome to the forum,

imo, with sftp, that is the only reliable method to verify file transfers.

not an expert, but rsync has two modes, delta transfer or whole file transfer.
doing delta transfer, and only a single byte of a 1TiB file has changed, only one chunk needs to be transferred from source to dest, only need to checksum that chunk.
no logic to checksum the entire 1TiB file.

i run run rsync on windows dozens of times every day, on small files, i use --whole-file --checksum

3

The rsync protocol does hashing (MD4 if I remember correctly) as it goes along, sending hashes with each chunk. The SFTP protocol doesn't allow for that though hence having to run the md5sum command via the shell.

You should be able to improve things by increasing --transfers though.