Hasher Bugs: (1) copyto/moveto do not always works and (2) hashsum --download does not always save hash

jwink3101 · May 14, 2022, 10:57pm

What is the problem you are having with rclone?

See the narrative below but

Using copyto and moveto does not always capture the hash. Especially on a second run (which is odd)
Specifying a path with hashsum and --download only caches that path for explicit recall.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: darwin 10.15.7 (64 bit)
- os/kernel: 19.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.9
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

hasher on crypt on local. Testing

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Not ignored but see the narrative below

The rclone config contents with secrets removed.

Not ignored but see the narrative below

A log from the command with the `-vv` flag

Not ignored but see the narrative below (inline)

Clean everything up. Note that this password is just playing so it is not sensitive. We use crypt because (a) I will eventually be using it and (b) it mimicks a remote without hashes

export RCLONE_CACHE_DIR='cache'
export RCLONE_TEMP_DIR='tmp'

rm -r $RCLONE_CACHE_DIR 2> /dev/null
mkdir -p $RCLONE_CACHE_DIR
mkdir -p $RCLONE_TEMP_DIR

rm -rf remote/ 2>/dev/null
mkdir -p remote/


export RCLONE_CONFIG=cfg1

cat << EOF > cfg1
[base]
type = alias
remote = remote/

[crypt]
type = crypt
remote = base:
filename_encryption = off
directory_name_encryption = false
password = ZWeBDBLmxXZNusb2tQ73V0Ha9AM
password2 = b8qxr_wqSTyvDS7-_EdR3_PM

[hasher]
type = hasher
remote = crypt:
EOF

Bug 1: copyto and/or moveto do not set hasher

Copying and moving does not always set the hash. It works sometimes which is odd. BUt never on the second run

echo "testmove" > move.txt && rclone moveto move.txt hasher:moved.txt
echo "testcopy" > copy.txt && rclone copyto copy.txt hasher:copied.txt
rclone hashsum SHA1 hasher:

echo "testmove" > move.txt && rclone moveto move.txt hasher:moved.txt
echo "testcopy" > copy.txt && rclone copyto copy.txt hasher:copied.txt
rclone hashsum SHA1 hasher: -vv

9c33f710a061db7210f2d77cd98780f5d1e797a3  moved.txt
d44b893ca7b34450022a104ea4dc5304dfb01e2a  copied.txt
2022/05/14 16:51:06 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:06 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:06 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:06 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:" "-vv"]
2022/05/14 16:51:06 DEBUG : Creating backend with remote "hasher:"
2022/05/14 16:51:06 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:06 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:06 DEBUG : Creating backend with remote "crypt:"
2022/05/14 16:51:06 DEBUG : Creating backend with remote "base:"
2022/05/14 16:51:06 DEBUG : Creating backend with remote "remote/"
2022/05/14 16:51:06 DEBUG : fs cache: renaming cache item "remote/" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:06 DEBUG : fs cache: renaming cache item "base:" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:06 DEBUG : hasher::hasher:: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:06 DEBUG : crypt~hasher.bolt: Opened for reading in 73.694µs
2022/05/14 16:51:06 DEBUG : moved.txt: getHash: no record
                                          moved.txt
2022/05/14 16:51:06 DEBUG : copied.txt: getHash: no record
                                          copied.txt
2022/05/14 16:51:06 DEBUG : 5 go routines active

Nothing but it should be there from the copy or move

Clean up for test2

rclone delete hasher:moved.txt
rclone delete hasher:copied.txt

Bug 2: Explicity specifying a file does not get saved for use more generally

If you use hashsum with --download on a specified path, it only gets saved for use when that path is specified

Write a file through the hasher remote and verify it works

echo "testfile" | rclone rcat hasher:testfile.txt
rclone hashsum SHA1 hasher:

fcff57d112692c8f1c3d330714358c59f1c268cc  testfile.txt

Invalidate the hash by changing the file ModTime on the underlying file

touch -a -m -t 2022051400.00 remote/testfile.txt.bin
rclone hashsum SHA1 hasher:

                                          testfile.txt

Parse it explicitly

rclone hashsum SHA1 hasher:testfile.txt --download -vv

2022/05/14 16:51:23 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:23 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:23 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:23 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:testfile.txt" "--download" "-vv"]
2022/05/14 16:51:23 DEBUG : Creating backend with remote "hasher:testfile.txt"
2022/05/14 16:51:23 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:23 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:23 DEBUG : Creating backend with remote "crypt:testfile.txt"
2022/05/14 16:51:23 DEBUG : Creating backend with remote "base:testfile.txt.bin"
2022/05/14 16:51:23 DEBUG : Creating backend with remote "remote/testfile.txt.bin"
2022/05/14 16:51:23 DEBUG : fs cache: adding new entry for parent of "remote/testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:23 DEBUG : fs cache: adding new entry for parent of "base:testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:23 DEBUG : hasher::hasher:testfile.txt: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:23 DEBUG : crypt~hasher.bolt: Opened for reading in 80.596µs
2022/05/14 16:51:23 DEBUG : crypt~hasher.bolt: released
2022/05/14 16:51:23 DEBUG : crypt~hasher.bolt: Opened for writing in 74.377µs
fcff57d112692c8f1c3d330714358c59f1c268cc  testfile.txt
2022/05/14 16:51:24 DEBUG : 6 go routines active

rclone hashsum SHA1 hasher: -vv

2022/05/14 16:51:32 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:32 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:32 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:32 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:" "-vv"]
2022/05/14 16:51:32 DEBUG : Creating backend with remote "hasher:"
2022/05/14 16:51:32 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:32 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:32 DEBUG : Creating backend with remote "crypt:"
2022/05/14 16:51:32 DEBUG : Creating backend with remote "base:"
2022/05/14 16:51:32 DEBUG : Creating backend with remote "remote/"
2022/05/14 16:51:32 DEBUG : fs cache: renaming cache item "remote/" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:32 DEBUG : fs cache: renaming cache item "base:" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:32 DEBUG : hasher::hasher:: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:32 DEBUG : crypt~hasher.bolt: Opened for reading in 74.423µs
2022/05/14 16:51:32 DEBUG : testfile.txt: getHash: fingerprint changed
                                          testfile.txt
2022/05/14 16:51:32 DEBUG : 5 go routines active

This should have worked

Note the following still does

rclone hashsum SHA1 hasher:testfile.txt -vv

2022/05/14 16:52:02 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:52:02 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:52:02 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:52:02 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:testfile.txt" "-vv"]
2022/05/14 16:52:02 DEBUG : Creating backend with remote "hasher:testfile.txt"
2022/05/14 16:52:02 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:52:02 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:52:02 DEBUG : Creating backend with remote "crypt:testfile.txt"
2022/05/14 16:52:02 DEBUG : Creating backend with remote "base:testfile.txt.bin"
2022/05/14 16:52:02 DEBUG : Creating backend with remote "remote/testfile.txt.bin"
2022/05/14 16:52:02 DEBUG : fs cache: adding new entry for parent of "remote/testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:52:02 DEBUG : fs cache: adding new entry for parent of "base:testfile.txt.bin", "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:52:02 DEBUG : hasher::hasher:testfile.txt: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:52:02 DEBUG : crypt~hasher.bolt: Opened for reading in 74.973µs
2022/05/14 16:52:02 DEBUG : testfile.txt: cached sha1 = "fcff57d112692c8f1c3d330714358c59f1c268cc"
fcff57d112692c8f1c3d330714358c59f1c268cc  testfile.txt
2022/05/14 16:52:02 DEBUG : 5 go routines active

Note that this does still work

jwink3101 · May 14, 2022, 10:59pm

pinging @ivandeex to let me know. I can open these on github

ncw · May 16, 2022, 9:07am

I think you should put these on github - you've made nice repros and they seem like geniuine bugs - thank you

system · June 15, 2022, 9:08am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.