Rclone check - SUM

tb582 · July 25, 2021, 11:10pm

Looking at the new Rclone check SUM support - trying to get away from my own script that generates checksums... seems like I can use rclone check SUM to do this for me?

--checkfile HASH

Animosity022 · July 25, 2021, 11:19pm

I'm not sure what you are asking.

The check help page is here:

https://rclone.org/commands/rclone_check/

tb582 · July 26, 2021, 7:47pm

Is there a link somewhere to what the exact syntax is of the SUM format ?

If you supply the --checkfile HASH flag with a valid hash name, the source:path must point to a text file in the SUM format.

Animosity022 · July 26, 2021, 8:02pm

All written up here:

github.com/rclone/rclone

cmd: add hashSUM file support

rclone:master ← ivandeex:pr-checksum

opened 07:00PM - 22 May 21 UTC

ivandeex

+685 -41

### What is the purpose of this change? Currently `rclone check` supports mat…ching two file trees by sizes and hashes. However, rclone does not support so called hash-SUM files. These are produced by the de-facto standard family of GNU core utilities like [sha1sum](https://en.wikipedia.org/wiki/Sha1sum). ### Use Cases SUM files provide end-to-end checksums, an invaluable tool where storage sums don't exist. They serve the following use cases: - User-side "cache" of remote hashsums. - As a measure to detect [bit-rot](https://en.m.wikipedia.org/wiki/Data_corruption#Silent) timely. - Tracking integrity across multiple data migrations. - Migration of ancient data corpus to incompatible provider. - And more... ### File Naming There exist two widely used naming schemes for SUM files: 1. `MD5SUM`, `SHA1SUM`, etc - uppercase name without extension carrying the hash digest type right in the name, usually located right beside the files to check. 2. `something.md5`, `somethingelse.sha1`, etc - having hash type in extension. ### File Format The SUM file format is [very simple](https://en.wikipedia.org/wiki/Sha1sum#Hash_file_trees), has been stable for years and supported by a wide range of software. However, there is no official standard. GNU man pages ([md5sum](https://www.gnu.org/software/coreutils/manual/html_node/md5sum-invocation.html#md5sum-invocation) et al) is the _de-facto_ standard. SUM files are line oriented. They **don't record** the hash digest type - it should be known in advance. Each line contains: 1. Hash digest 2. Space character 3. Another space or `*` (asterisk, see below) 4. File name, up to the line end. The asterisk modifier is a legacy artifact of _GNU coreutils_ builds for DOS/Windows where it used to denote that the file is _binary_ (with Unix line endings). It means nothing on Unix. The proposed implementation of SUM file support simply ignores asterisks and treats all files as _binary_ (no DOS line feed translation), following the [SUM file parser](https://github.com/ericlagergren/go-coreutils/blob/master/md5sum/checksum_common/check_sum.go#L85) in the [Go coreutils port](https://github.com/ericlagergren/go-coreutils/). The "distance" between digest and file name **must** always be 2 characters, be it `" *"` for binary files or `" "` (two spaces) for **ex** text files. The 3rd space, if any, will belong to the file name. Note: I could not find any real world use of the `" |"` modifier mentioned by @klauspost in https://github.com/rclone/rclone/issues/157#issuecomment-144439399. ### Unimplemented Features Besides _ignoring asterisks_, this implementation will not: - perform _backslash unescaping_ (ancient hack to tackle extended unicode and whitespace in file names) - support DOS-related flags _--binary_ and _--text_ (we treat files as _binary_, use them as is and never translate EOLs) - handle logic of rarely used flags _--strict_ and _--zero_ from the [GNU manual](https://www.gnu.org/software/coreutils/manual/html_node/md5sum-invocation.html#md5sum-invocation) These features could be implemented by future PRs, but I guess nobody will ever need them. Note that integration with rclone `fs.Hash` API (e.g. using SUM files as a transparent hashsum source or cache) is also out of scope for this PR. ### Making Sum Files rclone can produce valid SUM files with command `rclone hashsum HASH REMOTE:PATH --output-file FILE [--download]` (one minor feature that I miss sometimes is sorting output by file name - one day I'm gonna submit a PR) Standard coreutils normally print SUM data to _STDOUT_ so the file can be **produced** by redirection `sha1sum /files > SHA1SUM`. To **check** file tree against a SUM file, use the `-c` flag like `sha1sum -c SHA1SUM /files`. The hash digest used is defined by the name of the executable but **not recorded anywhere** in the sum file. This pull request handles only the **check** case above. ### CLI: Constraints This patch tries to enroll the `-c` GNU convention into the rclone CLI flag family respecting the following constraints: 1. The `-c` short flag is already taken, so we use upper-case `-C` to feel as close as possible. 2. The `--checksum` long flag is also taken, so we use `--checkfile` (no dash since it's a noun). Probably `--sum-file` would be closer to the file format name, but the first letter `c` better correlates with the short flag. See a note below. ### CLI: Variants The patch introduces a single internal function `operations.CheckSum` and the following command-line interfaces for it: 1. The `hashsum` command gains new flag `--checkfile SUMFILE`, ex.: `rclone hashsum CRC32 -C sumfile.crc32 remote:path/to/files` 2. Commands `sha1sum` and `md5sum` also gain this flag, ex.: `rclone sha1sum --checkfile SHA1SUM remote:path/tofiles` 3. New command `checksum HASHNAME SUMFILE REMOTE:PATH` is added, ex.: `rclone checksum md5 remote:path/to/MD5SUM remote:path/to --exclude MD5SUM` 4. The `check` command gains a new form with **3** arguments, ex.: `rclone check HASHNAME SUMFILE REMOTE:PATH` 5. The `check` command in the current 2-argument form gains the new flag `--checkfile HASHNAME`. As the last case feels slightly out of sync, let me bring an example: ``` rclone check REMOTE1:PATH1 REMOTE2:PATH2 # match two file trees by hash/size rclone check -C sha1 SUMFILE REMOTE:PATH # check file tree against a SUM file by hash ``` All the variants are placed in separate commits within this PR so **reviewer can decide** which are good for final squash/merge and which are doomed. ### Notes 1. In all cases above the SUM file can point to a remote object using the `REMOTE:PATH` notation as well as be a local file (SUM files are frequently distributed within the same tree as the data). 2. The new `rclone checksum` command supports all extended reporting features from `rclone check`, like `--combined`, `--missing-...` and so on. 3. The `md5sum`, `sha1sum` and `hashsum` commands _do not_ support extended check reporting features. Instead, they work as if `--combined -` was supplied (i.e. _all sigils_ are reported to **stdout**, unless `--progress` is enabled). This behavior is compatible with `-c` in GNU utilities. 4. All proposed commands support filtering, which can be useful if a sum file is located beside the files to be checked. 5. All variants support the `--download` flag and respect `--checkers`. 6. The `--checkfile` option might be named as `--check-from` or `--check-by` (with short name `-C`), or just `--check` if not taken. I'm leaving the decision up to reviewer. ### Was the change discussed in an issue or in the forum before? Approved by https://github.com/rclone/rclone/issues/157#issuecomment-532762729 Fixes #1005 ### Checklist - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request). - [x] I have added tests for all changes in this PR if appropriate. - [x] I have added documentation for the changes if appropriate. - [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [x] I'm done, this Pull Request is ready for review :-Q

asdffdsa · July 26, 2021, 8:10pm

you can create a sum file using rclone

for example,
rclone hashsum md5 . --output-file=sum.txt -vv

rclone check sum.txt . --checkfile=md5 --one-way -vv

tb582 · August 20, 2021, 8:06pm

I'm working on verifying this but does anyone know if the filePath in the SUM file can be absolute? or just it just need to be relative? like it is in the output of

rclone hashsum md5 . --output-file=sum.txt -vv

ie the command above outputs:
e4cb2eafeaade82c723af3a058cec2b2 Pictures/2021-02-07 17-45-40.jpeg

where as I have in my SUM file:
E4CB2EAFEAADE82C723AF3A058CEC2B2 C:\\Users\\Tony\\Pictures\\2021-02-07\ 17-45-40.jpeg

ivandeex · August 20, 2021, 8:56pm

rclone does not support absolute paths in SUM files.
SUM paths are always treated as relative to the remote argument of the hashsum, checksum, or check command.
absolute paths will make rclone print a error.

please keep in mind that rclone is a cloud-first tool and does not have a notion of absolute paths. all paths are relative to some "remote".
even local paths iterated during operations are treated as relative to the local path supplied on command line which constitutes a per-command "local remote".

asdffdsa · August 20, 2021, 9:43pm

did some quick testing and this worked for me.

dir of the folder containing the files.

b:\test>dir /b
file.txt

sum.txt
d41d8cd98f00b204e9800998ecf8427e test/file.txt

rclone command
rclone check sum.txt b:\ --checkfile=md5 --one-way -vv

so i would test the following

change
C:\\Users\\Tony\\Pictures\\2021-02-07\ 17-45-40.jpeg
to
Users/Tony/Pictures/2021-02-07/17-45-40.jpeg
use a command such as
rclone check sum.txt c:\ --checkfile=md5 --one-way -vv

tb582 · August 22, 2021, 5:03pm

thanks - curious though if I'm running on windows why do I need to change my path slashes? I guess its part of the cloud first ?

asdffdsa · August 22, 2021, 5:47pm

in my testing, need to use /, not \
tho it seems that rclone silently ignores a path that uses \, no warning, no error...
the error.txt created is empty, a zero byte file

sum.txt

d41d8cd98f00b204e9800998ecf8427e  test\file01.txt
d41d8cd98f00b204e9800998ecf8427e  test/file02.txt

rclone.exe ls b:\test 
        0 file01.txt
        0 file02.txt

rclone.exe check sum.txt b:\ --checkfile=md5 --one-way -vv --include="/test/**" --error=error.txt 
DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "check" "sum.txt" "b:\\" "--checkfile=md5" "--one-way" "-vv" "--error=error.txt"]
DEBUG : test/file02.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK
NOTICE: Local file system at //?/b:/: 0 differences found
NOTICE: Local file system at //?/b:/: 1 matching files

ivandeex · August 22, 2021, 6:35pm

In many clouds, say S3, the file names can contain *?:\ and so on without limits.
The only ~~greatest common divisor~~ common path separator is /, so rclone follows the majority.

Probably in future we can add special options for check and checksum like --skip-prefix-dirs N that will bite off the given number of leading path directories in the SUM file like patch -p (and probably something to convert windows slashes). Fancy submitting a patch or at least a feature request?

ivandeex · August 22, 2021, 6:40pm

I oracle that if you omit --one-way then rclone will cry aloud that it can't find the test\file01.txt file with backslash in the name (if my memory doesn't fail me, --one-way means ignore files missing on one side of the check).

tb582 · August 24, 2021, 5:13pm

ok another question here ... I want to have rclone check my local sum file against my rclone remote - is that possible?

asdffdsa · August 24, 2021, 6:04pm

yes, that is possible

rclone hashsum md5 .\source --output-file=sum.txt

sum.txt
560f9e8172f5d359e6cb2205adbcc5df file.txt

rclone checksum md5 sum.txt remote:testfolder01 -vv 
DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "checksum" "md5" "sum.txt" "remote:testfolder01" "-vv"]
DEBUG : Creating backend with remote "sum.txt"
DEBUG : file.txt: md5 = 560f9e8172f5d359e6cb2205adbcc5df OK
NOTICE: S3 bucket testfolder01: 0 differences found
NOTICE: S3 bucket testfolder01: 1 matching files

asdffdsa · August 24, 2021, 6:07pm

well, i was taught in greek mythology high school class.
never trust an oracle that gives a correct answer the first time...

tb582 · August 24, 2021, 9:09pm

Alright so I had rclone generate a sample sum file:

94918a22d08f0af48da351e517ad9ae6  NTUSER.DAT{53b39e88-18c4-11ea-a811-000d3aa4692b}.TM.blf
                           ERROR  NTUSER.DAT
6f7d4c882b008493c20d4b590e548de3  _netrc
                           ERROR  ntuser.dat.LOG1
                           ERROR  ntuser.dat.LOG2
6fc234ad3752e1267b34fb12bcd6718b  ntuser.ini
b4f52fc75ea9380f652e0bb17055e1f4  NTUSER.DAT{53b39e88-18c4-11ea-a811-000d3aa4692b}.TMContainer00000000000000000001.regtrans-ms
c5373647232233e265896e88703b1ce2  .gitconfig
59071590099d21dd439896592338bf95  NTUSER.DAT{53b39e88-18c4-11ea-a811-000d3aa4692b}.TMContainer00000000000000000002.regtrans-ms
293e53c7631fce3a3ab033458aa85a2c  sum2.txt
c0aec59d3c84d2d7928e3bffc9999f7b  sum.txt
793bc044a231708c798287dc6a01924b  summmm.txt
9e36cc3537ee9ee1e3b10fa4e761045b  Desktop/desktop.ini

If I now use that file to try and run rclone check it complains that there's invalid lines??

rclone check C:\Users\Tony\sum2.txt remote:/ --checkfile=md5 --filter-from "C:\Program Files (x86)\rclone\filter-file.txt"
2021/08/24 17:06:15 NOTICE: sum2.txt: improperly formatted checksum line 1
2021/08/24 17:06:15 NOTICE: sum2.txt: improperly formatted checksum line 3
2021/08/24 17:06:15 NOTICE: sum2.txt: improperly formatted checksum line 4
2021/08/24 17:06:16 ERROR : My Videos/.DS_Store: sum not found
2021/08/24 17:06:16 ERROR : My Videos/100_1324.MOV: sum not found
2021/08/24 17:06:16 ERROR : My Videos/100_1325.MOV: sum not found

asdffdsa · August 24, 2021, 9:14pm

post the command used to generate the sum file.
post filter-file.txt

tb582 · August 24, 2021, 9:18pm

rclone hashsum md5 . --output-file=sum2.txt -vv

filterfile

- /My Pictures/Scans/**
- /My Pictures/Mom/**
- /My Videos/google_unsupported/**
+ /My Pictures/**
+ /My Videos/**
- /**
- **.xmp
- **.DS_Store
- **.ini

asdffdsa · August 24, 2021, 9:24pm

what is the purpose for creating the hashsum, what you do plan to with it?

tb582 · August 24, 2021, 9:26pm

I want to check the files on my remote against the SUM file. Any that exist on the remote but not the SUM file I'll copy locally for further processing.