Hasher Bugs: (1) copyto/moveto do not always works and (2) hashsum --download does not always save hash

What is the problem you are having with rclone?

See the narrative below but

  1. Using copyto and moveto does not always capture the hash. Especially on a second run (which is odd)
  2. Specifying a path with hashsum and --download only caches that path for explicit recall.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: darwin 10.15.7 (64 bit)
- os/kernel: 19.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.9
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

hasher on crypt on local. Testing

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Not ignored but see the narrative below

The rclone config contents with secrets removed.

Not ignored but see the narrative below

A log from the command with the -vv flag

Not ignored but see the narrative below (inline)


Clean everything up. Note that this password is just playing so it is not sensitive. We use crypt because (a) I will eventually be using it and (b) it mimicks a remote without hashes

export RCLONE_CACHE_DIR='cache'
export RCLONE_TEMP_DIR='tmp'

rm -r $RCLONE_CACHE_DIR 2> /dev/null
mkdir -p $RCLONE_CACHE_DIR
mkdir -p $RCLONE_TEMP_DIR

rm -rf remote/ 2>/dev/null
mkdir -p remote/


export RCLONE_CONFIG=cfg1

cat << EOF > cfg1
[base]
type = alias
remote = remote/

[crypt]
type = crypt
remote = base:
filename_encryption = off
directory_name_encryption = false
password = ZWeBDBLmxXZNusb2tQ73V0Ha9AM
password2 = b8qxr_wqSTyvDS7-_EdR3_PM

[hasher]
type = hasher
remote = crypt:
EOF

Bug 1: copyto and/or moveto do not set hasher

Copying and moving does not always set the hash. It works sometimes which is odd. BUt never on the second run

echo "testmove" > move.txt && rclone moveto move.txt hasher:moved.txt
echo "testcopy" > copy.txt && rclone copyto copy.txt hasher:copied.txt
rclone hashsum SHA1 hasher:

echo "testmove" > move.txt && rclone moveto move.txt hasher:moved.txt
echo "testcopy" > copy.txt && rclone copyto copy.txt hasher:copied.txt
rclone hashsum SHA1 hasher: -vv
9c33f710a061db7210f2d77cd98780f5d1e797a3  moved.txt
d44b893ca7b34450022a104ea4dc5304dfb01e2a  copied.txt
2022/05/14 16:51:06 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:06 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:06 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:06 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:" "-vv"]
2022/05/14 16:51:06 DEBUG : Creating backend with remote "hasher:"
2022/05/14 16:51:06 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:06 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:06 DEBUG : Creating backend with remote "crypt:"
2022/05/14 16:51:06 DEBUG : Creating backend with remote "base:"
2022/05/14 16:51:06 DEBUG : Creating backend with remote "remote/"
2022/05/14 16:51:06 DEBUG : fs cache: renaming cache item "remote/" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:06 DEBUG : fs cache: renaming cache item "base:" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:06 DEBUG : hasher::hasher:: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:06 DEBUG : crypt~hasher.bolt: Opened for reading in 73.694µs
2022/05/14 16:51:06 DEBUG : moved.txt: getHash: no record
                                          moved.txt
2022/05/14 16:51:06 DEBUG : copied.txt: getHash: no record
                                          copied.txt
2022/05/14 16:51:06 DEBUG : 5 go routines active

Nothing but it should be there from the copy or move

Clean up for test2

rclone delete hasher:moved.txt
rclone delete hasher:copied.txt

Bug 2: Explicity specifying a file does not get saved for use more generally

If you use hashsum with --download on a specified path, it only gets saved for use when that path is specified

Write a file through the hasher remote and verify it works

echo "testfile" | rclone rcat hasher:testfile.txt
rclone hashsum SHA1 hasher:
fcff57d112692c8f1c3d330714358c59f1c268cc  testfile.txt

Invalidate the hash by changing the file ModTime on the underlying file

touch -a -m -t 2022051400.00 remote/testfile.txt.bin
rclone hashsum SHA1 hasher:
                                          testfile.txt

Parse it explicitly

rclone hashsum SHA1 hasher:testfile.txt --download -vv
2022/05/14 16:51:23 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:23 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:23 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:23 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:testfile.txt" "--download" "-vv"]
2022/05/14 16:51:23 DEBUG : Creating backend with remote "hasher:testfile.txt"
2022/05/14 16:51:23 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:23 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:23 DEBUG : Creating backend with remote "crypt:testfile.txt"
2022/05/14 16:51:23 DEBUG : Creating backend with remote "base:testfile.txt.bin"
2022/05/14 16:51:23 DEBUG : Creating backend with remote "remote/testfile.txt.bin"
2022/05/14 16:51:23 DEBUG : fs cache: adding new entry for parent of "remote/testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:23 DEBUG : fs cache: adding new entry for parent of "base:testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:23 DEBUG : hasher::hasher:testfile.txt: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:23 DEBUG : crypt~hasher.bolt: Opened for reading in 80.596µs
2022/05/14 16:51:23 DEBUG : crypt~hasher.bolt: released
2022/05/14 16:51:23 DEBUG : crypt~hasher.bolt: Opened for writing in 74.377µs
fcff57d112692c8f1c3d330714358c59f1c268cc  testfile.txt
2022/05/14 16:51:24 DEBUG : 6 go routines active
rclone hashsum SHA1 hasher: -vv
2022/05/14 16:51:32 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:32 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:32 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:32 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:" "-vv"]
2022/05/14 16:51:32 DEBUG : Creating backend with remote "hasher:"
2022/05/14 16:51:32 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:32 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:32 DEBUG : Creating backend with remote "crypt:"
2022/05/14 16:51:32 DEBUG : Creating backend with remote "base:"
2022/05/14 16:51:32 DEBUG : Creating backend with remote "remote/"
2022/05/14 16:51:32 DEBUG : fs cache: renaming cache item "remote/" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:32 DEBUG : fs cache: renaming cache item "base:" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:32 DEBUG : hasher::hasher:: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:32 DEBUG : crypt~hasher.bolt: Opened for reading in 74.423µs
2022/05/14 16:51:32 DEBUG : testfile.txt: getHash: fingerprint changed
                                          testfile.txt
2022/05/14 16:51:32 DEBUG : 5 go routines active

This should have worked

Note the following still does

rclone hashsum SHA1 hasher:testfile.txt -vv
2022/05/14 16:52:02 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:52:02 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:52:02 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:52:02 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:testfile.txt" "-vv"]
2022/05/14 16:52:02 DEBUG : Creating backend with remote "hasher:testfile.txt"
2022/05/14 16:52:02 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:52:02 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:52:02 DEBUG : Creating backend with remote "crypt:testfile.txt"
2022/05/14 16:52:02 DEBUG : Creating backend with remote "base:testfile.txt.bin"
2022/05/14 16:52:02 DEBUG : Creating backend with remote "remote/testfile.txt.bin"
2022/05/14 16:52:02 DEBUG : fs cache: adding new entry for parent of "remote/testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:52:02 DEBUG : fs cache: adding new entry for parent of "base:testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:52:02 DEBUG : hasher::hasher:testfile.txt: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:52:02 DEBUG : crypt~hasher.bolt: Opened for reading in 74.973µs
2022/05/14 16:52:02 DEBUG : testfile.txt: cached sha1 = "fcff57d112692c8f1c3d330714358c59f1c268cc"
fcff57d112692c8f1c3d330714358c59f1c268cc  testfile.txt
2022/05/14 16:52:02 DEBUG : 5 go routines active

Note that this does still work


pinging @ivandeex to let me know. I can open these on github

I think you should put these on github - you've made nice repros and they seem like geniuine bugs - thank you