Unable to update original post due to "This topic was automatically closed 30 days after the last reply. New replies are no longer allowed."
Original Post: Hasher Bolt Not Written After File Replacement + No Strict Checksum Mode
I have updated fix-9308-hasher-update branch.
Batch Script Test: feat: add upload checksum no-fallback upload-missing include-modtime by Subordinator · Pull Request #9505 · rclone/rclone · GitHub
Note after eventually merging to main I will then focus my efforts on:
A crypt-backend command or flag, provisional name
`rclone backend encrypt-leftovers <crypt:>`, that walks the underlying remote,
identifies entries that do not parse as a valid encrypted name (rule differs per
filename_encryption mode), re-uploads via the crypt wrapper, then removes the
underlying entry. Until that lands, pre-existing unencrypted leftovers on the
underlying remote will persist; new uploads via this script always go through
crypt and are encrypted normally.
Post-sync equality check confirming every folder, file, name, modtime, and
checksum match between source and target.
Required upstream changes (shared edit site: fs/operations/check.go for
CheckOpt/checkMarch/CheckFn; flag binding in cmd/check/check.go via AddFlags,
inherited by cmd/cryptcheck/cryptcheck.go without further edits):
--check-include-modtime Compare modtime alongside the hash. Skipped when the
destination does not support modtime. Parallel to
--checksum-include-modtime on sync/copy.
--check-no-fallback Treat noHash result as an error (counted via
fs.CountError) instead of silently passing. Parallel
to --checksum-no-fallback on sync/copy.
--check-directories Extend the checkMarch walk to report missing
directories (including empty ones) via the existing
missing-on-src and missing-on-dst writers.
Once those land, this segment becomes a single `rclone check` (or `cryptcheck`)
invocation with --checksum --check-include-modtime --check-no-fallback
--check-directories plus --differ/--missing-on-src/--missing-on-dst/--error
writers, and the run can assert all four files are zero bytes.
Once all these are merged, we'll then have support to upload data matching source bit-for-bit, and to then validate this.
TLDR: I've added flags to upload data with strict data integrity mode. Todo: Work on a flag to encrypt unencrypted files (and then remove the unencrypted counterparts). Another flag to validate remote matches source exactly (with no fallbacks).
Why? See my original post (and the original ticket).