Hasher Bolt Not Written After File Replacement + No Strict Checksum Mode

Aliases · April 3, 2026, 11:27am

What is the problem you are having with rclone?

I want every file on my remote to have a verified checksum matching the local source.
Running rclone sync --checksum does not achieve this reliably due to two problems:

Problem 1: When rclone sync replaces an existing file, the hasher bolt is not updated.
The log line where it fails: "Dst hash empty - aborting Src hash check". New uploads write
to the bolt correctly. Replacements do not. On the next run, the stale bolt entry masks the
missing checksum and the file passes hash validation undetected.

Problem 2: There is no strict checksum mode. rclone sync --checksum falls back to size
and modtime when a hash is unavailable, so files with no remote checksum pass undetected.

The hasher backend was added on top of the crypt layer specifically to capture and cache
checksums locally during transfers, so that rclone has access to remote checksums for
comparison during sync. Without it, the crypt layer does not expose the underlying
remote checksums to rclone sync.

Proposed Fixes:

New flag(s) on rclone sync. Example: rclone sync {source} {remote} --checksum-strict --upload-missing-hash

For providers that store checksums natively on upload, this would make the hasher backend
and unnecessary.

Alternatively, fix the hasher backend so the bolt is always written after a successful
file replacement, removing the condition that causes "Dst hash empty - aborting Src hash check" to abort the bolt write.

Happy to discuss on these forums. More details in the GitHub issue:

github.com/rclone/rclone

Fix Current Unreliable Data Integrity: hasher bolt not written after file replacement and no strict checksum mode

opened 11:03AM - 03 Apr 26 UTC

Subordinator

# Requested Fix for Current Unreliable Method of Data Integrity ## Environment … - **rclone version:** 1.73.1 (Windows AMD x64) - **Operating system:** Windows 10 (64 bit) --- ## Section 1 - The Goal: Guaranteed Data Integrity Between Local and Remote My goal is to be certain that every file on my remote storage is bit-for-bit identical to its local source. Size and modification time checks are not enough. A file can match on both and still have different or corrupted content. The only reliable way to confirm two files are identical is a checksum match. This means: 1. Every file on the remote must have a checksum stored against it. 2. That checksum must match the locally computed checksum of the source file. If even one file on the remote has no checksum, that file cannot be verified. The integrity guarantee is broken. --- ## Section 2 - How I Currently Attempt To Accomplish This ### Remote Setup I've tested rclone with a three-layer remote chain approach: ``` BaseRemote (type = filen) CryptRemote (type = crypt, remote = BaseRemote:, filename_encryption = off) HasherRemote (type = hasher, remote = CryptRemote:, hashes = {hash_algorithm}, max_age = off) ``` All sync and copy operations target `HasherRemote:`. The intent is that every file transfer writes a hash to the local bolt database during the transfer. ### Why This Setup is Needed When a file is uploaded to the cloud storage provider via rclone, the provider stores a checksum against it on the remote. The hasher backend sits on top of the crypt layer and is used to capture and cache those checksums locally in a bolt database during the transfer. This allows rclone to retrieve checksums from the local bolt on subsequent runs without needing to re-download files to recompute them. However, files uploaded to the provider outside of rclone (example: desktop app or browser client) do not have their checksum stored. Moreover, the provider has no way to retroactively generate checksums for these files through rclone (or their own tools). The only resolution is to re-upload those files through rclone. Additionally, hasher can compute and store the checksum during the transfer (to then be used at comparison stages). ### The Batch Script Because rclone alone does not have the flags needed to achieve full checksum coverage in a single command, I wrote a Windows batch script. The script logic (simplified) is as follows: 1. Detect unencrypted files on the remote and upload their encrypted counterpart. 2. Identify files on the remote that have no checksum and re-upload them to force checksum caching. 3. Run `rclone sync --checksum` to mirror source to remote. 4. Generate final checksum lists for both source and remote. 5. Compare the two lists to verify the remote matches the source exactly. ### The Problem Discovered During testing, I found that `rclone sync` does not write to the hasher bolt when it replaces an existing file on the remote. This means `rclone sync` completes successfully, but any file that was replaced during sync has no checksum in the bolt afterwards. The only workaround within the current rclone feature set is to upload every replaced file a second time after sync using a separate `rclone copy` pass. Uploading every changed file twice is inefficient and defeats the purpose of using `rclone sync`. Because of this, I started rewriting the solution as a PowerShell script that replaces `rclone sync` entirely with explicit `rclone copy`, `rclone deletefile`, and `rclone purge` calls. This guarantees the bolt is written on every transfer. However, this is a significant amount of engineering work to work around what should be a fix inside rclone. --- ## Section 3 - Problems With rclone ### Problem 1: Hasher Bolt Not Updated After File Replacement When `rclone sync` transfers a file that already exists on the remote (a replacement, not a new upload), the bolt database is not updated with the new hash. The log line where this fails: ``` "Dst hash empty - aborting Src hash check" ``` When the destination has no prior bolt entry, the post-transfer hash write is abandoned entirely. By contrast, when uploading a new file, the bolt is written correctly: ``` "cached {hash_algorithm} = ..." ``` **Step-by-step of what happens:** 1. A file exists on the remote with a valid bolt entry from a previous upload. 2. The local source file is modified. Its content, size, and checksum all change. 3. `rclone sync --checksum` runs and detects a size mismatch between source and remote. 4. rclone transfers the new version of the file to the remote successfully. 5. During the transfer, rclone attempts to write the new hash to the bolt. 6. The bolt lookup for the destination returns no record. 7. rclone logs `"Dst hash empty - aborting Src hash check"` and abandons the bolt write. 8. The file is marked as `"Copied (replaced existing)"` in the log. 9. The bolt still holds the old stale hash from the previous upload. 10. Running `rclone hashsum` against the remote returns a blank entry for this file. 11. The source and remote checksum lists do not match. The integrity check fails. **What happens on the next run:** 1. The pre-sync hash check reads the stale bolt entry from step 9 above. 2. The file is incorrectly classified as having a valid hash on the remote. 3. The remote file has no checksum stored against it, but the stale bolt entry masks this. --- ### Problem 2: No Flag to Enforce Hash Upload During Transfer There is no flag on `rclone sync` or `rclone copy` that forces a hash to be computed and stored in the bolt for every transferred file unconditionally. `--hasher-auto-size` applies only to hashsum read operations, not to transfers. The only way to reliably populate the bolt is `rclone copy --no-check-dest --inplace`. This works, but it means every corrective action requires uploading the file again in a separate pass after sync. --- ### Problem 3: No Flag to Treat a Missing Hash as a Mismatch `rclone sync --checksum` falls back to size and modification time when a hash is not available on one or both sides. There is no strict mode that treats a missing hash as a definitive mismatch and triggers a re-upload. This means a file with no remote hash can pass the sync check undetected, leaving the remote in an unverified state. --- ### Problem 4: No Way to Force the Remote to Generate Hashes for Existing Files Understandably, rclone has no command to instruct a cloud storage provider to generate checksums for files that were uploaded without one. The only resolution is to re-upload the file through rclone. Additionally, the hasher can compute and store the hash locally during the transfer. For providers that do not generate checksums natively, this means any file uploaded outside rclone will permanently have no checksum unless it is re-uploaded through rclone. --- ## Section 4 - Root Cause The root cause of Problem 1 is in the hasher backend (`backend/hasher/hasher.go`). After a successful file transfer that replaces an existing remote file, the bolt write is conditionally skipped when the destination had no prior hash entry. This condition should not exist. If a file was successfully transferred, its hash should always be written to the bolt. The hash was already computed during the transfer. Discarding it is incorrect behaviour. --- ## Section 5 - What I Am Requesting ### Ideal Solution: sync flags The ideal solution is a single command that handles everything: rclone sync {source} {remote} --checksum-strict --upload-missing-hash Where: - `--checksum-strict` uses only checksums for comparison with no fallback to size or modification time. If a hash is not available on either side, the file is treated as a mismatch and re-uploaded. - `--upload-missing-hash` re-uploads any file that exists on the remote but has no checksum stored against it, to force hash generation on the provider side during upload. Alternatively, combine those flags together into a single flag. However, a user might want strict checksum comparison without triggering re-uploads for missing hashes, or vice versa. Combining them removes that flexibility. For providers that store checksums natively on upload, this command alone would not require the hasher backend, a bolt database, or any external scripting. If this is not feasible, the fix below would resolve the core problem described in Section 3. --- ### Fix: hasher backend After any successful file transfer through the hasher backend, always write the computed hash to the bolt. Remove the condition that causes `"Dst hash empty - aborting Src hash check"` to abort the bolt write. This is the minimum fix needed. --- ## Section 6 - Impact The ideal solution would resolve all problems in Section 3 in a single sync command, with no need for the hasher backend or external scripting. Without the ideal solution, the hasher backend fix is the minimum viable fix. Without it, the hasher bolt becomes unreliable on every run that includes file replacements, and the hasher backend cannot be trusted to track checksums accurately.

Run the command 'rclone version' and share the full output of the command.

1.73.1 (Windows AMD x64)

rclone config contents with secrets removed.

With the proposed rclone sync flags, the [HasherRemote] can be removed.

[BaseRemote]
type = {type}
email = {redacted}
password = {redacted}
api_key = {redacted}

[CryptRemote]
type = crypt
remote = BaseRemote:
filename_encryption = off
directory_name_encryption = false
password = {redacted}

[HasherRemote]
type = hasher
remote = CryptRemote:
hashes = blake3
max_age = off

minesheep · April 4, 2026, 8:42am

I don't think hasher backend is needed at all in that scenario, crypt backend has authentication built in

Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.

That is 16bytes of authenticator (poly1305) for every 64KiB of data

Note however that this does not help at all if data is corrupted on upload and only discovered later on download and no backup exists (nor does locally stored hash help here).

Aliases · April 4, 2026, 9:05am

Hi @minesheep

Thank you for taking a look at this scenario. That’s much appreciated.

Currently, the reason of using Hasher is to capture hashes during upload so that I can
retrieve them in the same run via rclone hashsum, write them to a file, and compare
them against the locally computed source hashes to verify the remote matches the
source exactly without downloading any files.

Hasher is specifically needed here because files are encrypted via the crypt backend.
The hash stored on the remote is the hash of the encrypted bytes, which will never
match the locally computed hash of the plaintext source file. The hasher captures the
plaintext hash during upload before encryption occurs, which is the only way to get a
hash comparable to the source without downloading and decrypting every file to recompute it.