Does rclone check the checksums of files as they get transferred from local to local?

What is the problem you are having with rclone?

Does rclone check the checksums of files as they get transferred from local to local (no remote)? To quote from the homepage, "Rclone really looks after your data. It preserves timestamps and verifies checksums at all times." Is that true for local file transfers as well? Because the release notes for rclone 1.49 state these changes:

Local ...

Don't calculate any hashes by default (Nick Craig-Wood)

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.4529 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local filesystem

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync D:\Notes\IT K:\D\Notes\IT --exclude-from D:\System\PowerShell\rclone_exclude_d.txt --check-first --order-by name --transfers 1 --create-empty-src-dirs --progress

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

2024/06/26 15:35:11 NOTICE: Config file "C:\\Users\\User\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults
; empty config

A log from the command that you were trying to run with the -vv flag

2024/06/26 15:37:59 DEBUG : rclone: Version "v1.67.0" starting with parameters ["C:\\Software\\rclone\\rclone.exe" "sync" "D:\\Notes\\IT" "K:\\D\\Notes\\IT" "--exclude-from" "D:\\System\\PowerShell\\rclone_exclude_d.txt" "--check-first" "--order-by" "name" "--transfers" "1" "--create-empty-src-dirs" "--progress" "-vv"]
2024/06/26 15:37:59 DEBUG : Creating backend with remote "D:\\Notes\\IT"
2024/06/26 15:37:59 NOTICE: Config file "C:\\Users\\User\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults
2024/06/26 15:37:59 DEBUG : fs cache: renaming cache item "D:\\Notes\\IT" to be canonical "//?/D:/Notes/IT"
2024/06/26 15:37:59 DEBUG : Creating backend with remote "K:\\D\\Notes\\IT"
2024/06/26 15:37:59 DEBUG : fs cache: renaming cache item "K:\\D\\Notes\\IT" to be canonical "//?/K:/D/Notes/IT"
2024/06/26 15:37:59 INFO  : Local file system at //?/K:/D/Notes/IT: Running all checks before starting transfers
2024/06/26 15:37:59 DEBUG : Backup: Directory modification time the same (differ by 0s, within tolerance 100ns)
2024/06/26 15:37:59 DEBUG : Added delayed dir = "Backup", newDst=Backup
...
2024/06/26 15:37:59 DEBUG : Windows/Windows Path Length.md: Unchanged skipping
2024/06/26 15:37:59 DEBUG : Windows/Windows Sandbox.md: Unchanged skipping
2024/06/26 15:37:59 DEBUG : Windows/cmd.exe.md: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/26 15:37:59 DEBUG : Windows/Windows Terminal.md: Unchanged skipping
2024/06/26 15:37:59 DEBUG : Windows/cmd.exe - Hide Terminal Window.md: Unchanged skipping
2024/06/26 15:37:59 DEBUG : Windows/cmd.exe.md: Unchanged skipping
2024/06/26 15:37:59 INFO  : Local file system at //?/K:/D/Notes/IT: Checks finished, now starting transfers
2024/06/26 15:37:59 DEBUG : Local file system at //?/K:/D/Notes/IT: Waiting for transfers to finish
2024/06/26 15:37:59 DEBUG : Backup/rclone.md: md5 = 85cfc0f4593a425c519f661601a5b0ec OK
2024/06/26 15:37:59 DEBUG : Backup/rclone.md.tiyomop7.partial: renamed to: Backup/rclone.md
2024/06/26 15:37:59 INFO  : Backup/rclone.md: Copied (replaced existing)
2024/06/26 15:37:59 DEBUG : Waiting for deletions to finish
2024/06/26 15:37:59 INFO  : Backup: Set directory modification time (using SetModTime)

welcome to the forum,

yes, rclone transferred Backup/rclone.md and verified/checked the checksum.
Backup/rclone.md: md5 = 85cfc0f4593a425c519f661601a5b0ec OK

OK, so it looks like it, thanks!

But then why does the changelog state, "Don't calculate any hashes by default (Nick Craig-Wood)"?

welcome

that is about comparing files, not transferring files.
be default, rclone compares files using modtime and size.
to compare files by checksum, use --checksum

1 Like

Yes, maybe that's the reason:

The rclone sync documentation states:

Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM.

So the not hashing from the changelog probably means that for local to local transfers, rclone doesn't do an MD5SUM by default to see if a file has changed. And the program only tests with size and modification time.

check out my post from 2019

It's funny, I actually read that post before, that's where I first saw the changelog line I referred to in the first post of this thread. But I don't quite understood it...

that is ok, as i love to quote myself ;wink

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.