# Rclone check - SUM **URL:** \ **Category:** Help and Support\ **Created:** [July 25, 2021, 11:10pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566 "2021-07-25T23:10:44Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [July 25, 2021, 11:10pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/1 "2021-07-25T23:10:44Z")
Looking at the new Rclone check SUM support - trying to get away from my own script that generates checksums... seems like I can use rclone check SUM to do this for me? `--checkfile HASH` ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [July 25, 2021, 11:19pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/2 "2021-07-25T23:19:19Z")
I'm not sure what you are asking. The check help page is here: [https://rclone.org/commands/rclone\_check/](https://rclone.org/commands/rclone_check/) ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [July 26, 2021, 7:47pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/3 "2021-07-26T19:47:08Z")
Is there a link somewhere to what the exact syntax is of the SUM format ? > If you supply the `--checkfile HASH` flag with a valid hash name, the `source:path` must point to a text file in the SUM format. ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [July 26, 2021, 8:02pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/4 "2021-07-26T20:02:00Z")
All written up here: > > > \### What is the purpose of this change? > > Currently \`rclone check\` supports mat…ching two file trees by sizes and hashes. > However, rclone does not support so called hash-SUM files. > These are produced by the de-facto standard family of GNU core utilities like \[sha1sum\](https://en.wikipedia.org/wiki/Sha1sum). > > \### Use Cases > > SUM files provide end-to-end checksums, an invaluable tool where storage sums don't exist. > They serve the following use cases: > \- User-side "cache" of remote hashsums. > \- As a measure to detect \[bit-rot\](https://en.m.wikipedia.org/wiki/Data\_corruption#Silent) timely. > \- Tracking integrity across multiple data migrations. > \- Migration of ancient data corpus to incompatible provider. > \- And more... > > \### File Naming > > There exist two widely used naming schemes for SUM files: > 1. \`MD5SUM\`, \`SHA1SUM\`, etc - uppercase name without extension carrying the hash digest type right in the name, > usually located right beside the files to check. > 2. \`something.md5\`, \`somethingelse.sha1\`, etc - having hash type in extension. > > \### File Format > > The SUM file format is \[very simple\](https://en.wikipedia.org/wiki/Sha1sum#Hash\_file\_trees), has been stable for years and supported by a wide range of software. However, there is no official standard. GNU man pages (\[md5sum\](https://www.gnu.org/software/coreutils/manual/html\_node/md5sum-invocation.html#md5sum-invocation) et al) is the \_de-facto\_ standard. > > SUM files are line oriented. They \*\*don't record\*\* the hash digest type - it should be known in advance. > Each line contains: > 1. Hash digest > 2. Space character > 3. Another space or \`\*\` (asterisk, see below) > 4. File name, up to the line end. > > The asterisk modifier is a legacy artifact of \_GNU coreutils\_ builds for DOS/Windows where it used to denote that the file is \_binary\_ (with Unix line endings). It means nothing on Unix. The proposed implementation of SUM file support simply ignores asterisks and treats all files as \_binary\_ (no DOS line feed translation), following the \[SUM file parser\](https://github.com/ericlagergren/go-coreutils/blob/master/md5sum/checksum\_common/check\_sum.go#L85) in the \[Go coreutils port\](https://github.com/ericlagergren/go-coreutils/). > > The "distance" between digest and file name \*\*must\*\* always be 2 characters, be it \`" \*"\` for binary files or \`" "\` (two spaces) for \*\*ex\*\* text files. The 3rd space, if any, will belong to the file name. > > Note: I could not find any real world use of the \`" |"\` modifier mentioned by @klauspost in https://github.com/rclone/rclone/issues/157#issuecomment-144439399. > > \### Unimplemented Features > > Besides \_ignoring asterisks\_, this implementation will not: > \- perform \_backslash unescaping\_ (ancient hack to tackle extended unicode and whitespace in file names) > \- support DOS-related flags \_--binary\_ and \_--text\_ (we treat files as \_binary\_, use them as is and never translate EOLs) > \- handle logic of rarely used flags \_--strict\_ and \_--zero\_ from the \[GNU manual\](https://www.gnu.org/software/coreutils/manual/html\_node/md5sum-invocation.html#md5sum-invocation) > > These features could be implemented by future PRs, but I guess nobody will ever need them. > > Note that integration with rclone \`fs.Hash\` API (e.g. using SUM files as a transparent hashsum source or cache) is also out of scope for this PR. > > \### Making Sum Files > > rclone can produce valid SUM files with command > \`rclone hashsum HASH REMOTE:PATH --output-file FILE \[--download\]\` > (one minor feature that I miss sometimes is sorting output by file name - one day I'm gonna submit a PR) > > Standard coreutils normally print SUM data to \_STDOUT\_ so the file can be \*\*produced\*\* by redirection \`sha1sum /files \> SHA1SUM\`. To \*\*check\*\* file tree against a SUM file, use the \`-c\` flag like \`sha1sum -c SHA1SUM /files\`. The hash digest used is defined by the name of the executable but \*\*not recorded anywhere\*\* in the sum file. This pull request handles only the \*\*check\*\* case above. > > \### CLI: Constraints > > This patch tries to enroll the \`-c\` GNU convention into the rclone CLI flag family respecting the following constraints: > 1. The \`-c\` short flag is already taken, so we use upper-case \`-C\` to feel as close as possible. > 2. The \`--checksum\` long flag is also taken, so we use \`--checkfile\` (no dash since it's a noun). > Probably \`--sum-file\` would be closer to the file format name, but the first letter \`c\` better correlates with the short flag. See a note below. > > \### CLI: Variants > > The patch introduces a single internal function \`operations.CheckSum\` and the following command-line interfaces for it: > > 1. The \`hashsum\` command gains new flag \`--checkfile SUMFILE\`, ex.: > \`rclone hashsum CRC32 -C sumfile.crc32 remote:path/to/files\` > 2. Commands \`sha1sum\` and \`md5sum\` also gain this flag, ex.: > \`rclone sha1sum --checkfile SHA1SUM remote:path/tofiles\` > 3. New command \`checksum HASHNAME SUMFILE REMOTE:PATH\` is added, ex.: > \`rclone checksum md5 remote:path/to/MD5SUM remote:path/to --exclude MD5SUM\` > 4. The \`check\` command gains a new form with \*\*3\*\* arguments, ex.: > \`rclone check HASHNAME SUMFILE REMOTE:PATH\` > 5. The \`check\` command in the current 2-argument form > gains the new flag \`--checkfile HASHNAME\`. > > As the last case feels slightly out of sync, let me bring an example: > \`\`\` > rclone check REMOTE1:PATH1 REMOTE2:PATH2 # match two file trees by hash/size > rclone check -C sha1 SUMFILE REMOTE:PATH # check file tree against a SUM file by hash > \`\`\` > > All the variants are placed in separate commits within this PR so \*\*reviewer can decide\*\* which are good for final squash/merge and which are doomed. > > \### Notes > > 1. In all cases above the SUM file can point to a remote object using the \`REMOTE:PATH\` notation as well as be a local file > (SUM files are frequently distributed within the same tree as the data). > 2. The new \`rclone checksum\` command supports all extended reporting features from \`rclone check\`, > like \`--combined\`, \`--missing-...\` and so on. > 3. The \`md5sum\`, \`sha1sum\` and \`hashsum\` commands \_do not\_ support extended check reporting features. > Instead, they work as if \`--combined -\` was supplied (i.e. \_all sigils\_ are reported to \*\*stdout\*\*, unless \`--progress\` is enabled). > This behavior is compatible with \`-c\` in GNU utilities. > 4. All proposed commands support filtering, which can be useful if a sum file is located beside the files to be checked. > 5. All variants support the \`--download\` flag and respect \`--checkers\`. > 6. The \`--checkfile\` option might be named as \`--check-from\` or \`--check-by\` (with short name \`-C\`), > or just \`--check\` if not taken. I'm leaving the decision up to reviewer. > > \### Was the change discussed in an issue or in the forum before? > > Approved by https://github.com/rclone/rclone/issues/157#issuecomment-532762729 > Fixes #1005 > > \### Checklist > > \- \[x\] I have read the \[contribution guidelines\](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request). > \- \[x\] I have added tests for all changes in this PR if appropriate. > \- \[x\] I have added documentation for the changes if appropriate. > \- \[x\] All commit messages are in \[house style\](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). > \- \[x\] I'm done, this Pull Request is ready for review :-Q ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [July 26, 2021, 8:10pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/5 "2021-07-26T20:10:13Z")
you can create a sum file using rclone for example, `rclone hashsum md5 . --output-file=sum.txt -vv` `rclone check sum.txt . --checkfile=md5 --one-way -vv` ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [August 20, 2021, 8:06pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/6 "2021-08-20T20:06:00Z")
I'm working on verifying this but does anyone know if the filePath in the SUM file can be absolute? or just it just need to be relative? like it is in the output of `rclone hashsum md5 . --output-file=sum.txt -vv` ie the command above outputs: `e4cb2eafeaade82c723af3a058cec2b2 Pictures/2021-02-07 17-45-40.jpeg` where as I have in my SUM file: `E4CB2EAFEAADE82C723AF3A058CEC2B2 C:\\Users\\Tony\\Pictures\\2021-02-07\ 17-45-40.jpeg` ---
**Author:** ![ivandeex](https://forum.rclone.org/user_avatar/forum.rclone.org/ivandeex/32/11183_2.png) [@ivandeex](https://forum.rclone.org/u/ivandeex)\ **Post date:** [August 20, 2021, 8:56pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/7 "2021-08-20T20:56:27Z")
> [@tb582](#): > > I'm working on verifying this but does anyone know if the filePath in the SUM file can be absolute? or just it just need to be relative? like it is in the output of rclone does not support absolute paths in SUM files. SUM paths are always treated as relative to the remote argument of the `hashsum`, `checksum`, or `check` command. absolute paths will make rclone print a error. please keep in mind that rclone is a cloud-first tool and does not have a notion of absolute paths. all paths are relative to some "remote". even local paths iterated during operations are treated as relative to the local path supplied on command line which constitutes a per-command "local remote". ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [August 20, 2021, 9:43pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/8 "2021-08-20T21:43:21Z")
did some quick testing and this worked for me. dir of the folder containing the files. ```auto b:\test>dir /b file.txt ``` sum.txt `d41d8cd98f00b204e9800998ecf8427e test/file.txt` rclone command `rclone check sum.txt b:\ --checkfile=md5 --one-way -vv` so i would test the following 1. change `C:\\Users\\Tony\\Pictures\\2021-02-07\ 17-45-40.jpeg` to `Users/Tony/Pictures/2021-02-07/17-45-40.jpeg` 2. use a command such as `rclone check sum.txt c:\ --checkfile=md5 --one-way -vv` ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [August 22, 2021, 5:03pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/9 "2021-08-22T17:03:10Z")
thanks - curious though if I'm running on windows why do I need to change my path slashes? I guess its part of the cloud first ? ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [August 22, 2021, 5:47pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/10 "2021-08-22T17:47:05Z")
in my testing, need to use `/`, not `\` tho it seems that rclone silently ignores a path that uses `\`, no warning, no error... the `error.txt` created is empty, a zero byte file sum.txt ```auto d41d8cd98f00b204e9800998ecf8427e test\file01.txt d41d8cd98f00b204e9800998ecf8427e test/file02.txt ``` ```auto rclone.exe ls b:\test 0 file01.txt 0 file02.txt rclone.exe check sum.txt b:\ --checkfile=md5 --one-way -vv --include="/test/**" --error=error.txt DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "check" "sum.txt" "b:\\" "--checkfile=md5" "--one-way" "-vv" "--error=error.txt"] DEBUG : test/file02.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK NOTICE: Local file system at //?/b:/: 0 differences found NOTICE: Local file system at //?/b:/: 1 matching files ``` ---
**Author:** ![ivandeex](https://forum.rclone.org/user_avatar/forum.rclone.org/ivandeex/32/11183_2.png) [@ivandeex](https://forum.rclone.org/u/ivandeex)\ **Post date:** [August 22, 2021, 6:35pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/11 "2021-08-22T18:35:05Z")
> [@tb582](#): > > thanks - curious though if I'm running on windows why do I need to change my path slashes? I guess its part of the cloud first ? In many clouds, say S3, the file names can contain `*?:\` and so on without limits. The only ~~greatest common divisor~~ common path separator is `/`, so rclone follows the majority. Probably in future we can add special options for `check` and `checksum` like `--skip-prefix-dirs N` that will bite off the given number of leading path directories in the SUM file like `patch -p` (and probably something to convert windows slashes). Fancy submitting a patch or at least a feature request? ---
**Author:** ![ivandeex](https://forum.rclone.org/user_avatar/forum.rclone.org/ivandeex/32/11183_2.png) [@ivandeex](https://forum.rclone.org/u/ivandeex)\ **Post date:** [August 22, 2021, 6:40pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/12 "2021-08-22T18:40:39Z")
I oracle that if you omit `--one-way` then rclone will cry aloud that it can't find the `test\file01.txt` file with backslash in the name (if my memory doesn't fail me, `--one-way` means ignore files missing on one side of the check). ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [August 24, 2021, 5:13pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/13 "2021-08-24T17:13:29Z")
> [@asdffdsa](#): > > one check sum.txt c:\ --checkfile=md5 --one-way -vv ok another question here ... I want to have rclone check my local sum file against my rclone remote - is that possible? ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [August 24, 2021, 6:04pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/14 "2021-08-24T18:04:07Z")
yes, that is possible `rclone hashsum md5 .\source --output-file=sum.txt ` sum.txt `560f9e8172f5d359e6cb2205adbcc5df file.txt` ```auto rclone checksum md5 sum.txt remote:testfolder01 -vv DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "checksum" "md5" "sum.txt" "remote:testfolder01" "-vv"] DEBUG : Creating backend with remote "sum.txt" DEBUG : file.txt: md5 = 560f9e8172f5d359e6cb2205adbcc5df OK NOTICE: S3 bucket testfolder01: 0 differences found NOTICE: S3 bucket testfolder01: 1 matching files ``` ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [August 24, 2021, 6:07pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/15 "2021-08-24T18:07:01Z")
well, i was taught in greek mythology high school class. never trust an oracle that gives a correct answer the first time... ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [August 24, 2021, 9:09pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/16 "2021-08-24T21:09:16Z")
Alright so I had rclone generate a sample sum file: ```auto 94918a22d08f0af48da351e517ad9ae6 NTUSER.DAT{53b39e88-18c4-11ea-a811-000d3aa4692b}.TM.blf ERROR NTUSER.DAT 6f7d4c882b008493c20d4b590e548de3 _netrc ERROR ntuser.dat.LOG1 ERROR ntuser.dat.LOG2 6fc234ad3752e1267b34fb12bcd6718b ntuser.ini b4f52fc75ea9380f652e0bb17055e1f4 NTUSER.DAT{53b39e88-18c4-11ea-a811-000d3aa4692b}.TMContainer00000000000000000001.regtrans-ms c5373647232233e265896e88703b1ce2 .gitconfig 59071590099d21dd439896592338bf95 NTUSER.DAT{53b39e88-18c4-11ea-a811-000d3aa4692b}.TMContainer00000000000000000002.regtrans-ms 293e53c7631fce3a3ab033458aa85a2c sum2.txt c0aec59d3c84d2d7928e3bffc9999f7b sum.txt 793bc044a231708c798287dc6a01924b summmm.txt 9e36cc3537ee9ee1e3b10fa4e761045b Desktop/desktop.ini ``` If I now use that file to try and run rclone check it complains that there's invalid lines?? ```auto rclone check C:\Users\Tony\sum2.txt remote:/ --checkfile=md5 --filter-from "C:\Program Files (x86)\rclone\filter-file.txt" 2021/08/24 17:06:15 NOTICE: sum2.txt: improperly formatted checksum line 1 2021/08/24 17:06:15 NOTICE: sum2.txt: improperly formatted checksum line 3 2021/08/24 17:06:15 NOTICE: sum2.txt: improperly formatted checksum line 4 2021/08/24 17:06:16 ERROR : My Videos/.DS_Store: sum not found 2021/08/24 17:06:16 ERROR : My Videos/100_1324.MOV: sum not found 2021/08/24 17:06:16 ERROR : My Videos/100_1325.MOV: sum not found ``` ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [August 24, 2021, 9:14pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/17 "2021-08-24T21:14:21Z")
post the command used to generate the sum file. post `filter-file.txt` ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [August 24, 2021, 9:18pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/18 "2021-08-24T21:18:10Z")
`rclone hashsum md5 . --output-file=sum2.txt -vv` filterfile ```auto - /My Pictures/Scans/** - /My Pictures/Mom/** - /My Videos/google_unsupported/** + /My Pictures/** + /My Videos/** - /** - **.xmp - **.DS_Store - **.ini ``` ---
**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\ **Post date:** [August 24, 2021, 9:24pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/21 "2021-08-24T21:24:27Z")
what is the purpose for creating the hashsum, what you do plan to with it? ---
**Author:** ![tb582](https://forum.rclone.org/letter_avatar_proxy/v4/letter/t/8edcca/32.png) [@tb582](https://forum.rclone.org/u/tb582)\ **Post date:** [August 24, 2021, 9:26pm UTC](https://forum.rclone.org/t/rclone-check-sum/25566/22 "2021-08-24T21:26:26Z")
I want to check the files on my remote against the SUM file. Any that exist on the remote but not the SUM file I'll copy locally for further processing. [Next page](https://forum.rclone.org/t/rclone-check-sum/25566.md?page=2)