Stickyimport of hasher does not always work

Suspected Bug
1

What is the problem you are having with rclone?

Sticky import with hasher is not working. See narrative below.

This is in reply to @ivandeex on this past post

Run the command 'rclone version' and share the full output of the command.

latest (just kidding)

rclone v1.57.0
- os/version: darwin 10.15.7 (64 bit)
- os/kernel: 19.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

crypt,hasher,local (to test. Also seen with OneDrive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone backend stickyimport hasher: SHA1 hashes.sha1 -vv

The rclone config contents with secrets removed.

See narrative. It's there

A log from the command with the -vv flag

See narrative

Narrative

Rclone setup. (these are madeup. Not sensitive passwords)

export RCLONE_CONFIG='temp.cfg'
rclone config show

[crypt]
type = crypt
remote = testdir
password = uET7jspvwfls12RLLrsYDS2mSw
password2 = eSWd42rJIPCs5P0wQaGNcJosiA

[hasher]
type = hasher
remote = crypt:
hashes = sha1

Build the directories

rm -rf testdir # used by crypt

echo "file1" | rclone rcat crypt:file1.txt
echo "file2" | rclone rcat crypt:subdir/file2.txt
echo "file3" | rclone rcat crypt:subdir/subsub/file3.txt

Get the hashes from crypt (not hasher)

rclone hashsum SHA1 --download crypt: --output-file hashes.sha1

cat hashes.sha1

38be7d1b981f2fb6a4a0a052453f887373dc1fe8  file1.txt
639daad06642a8eb86821ff7649e86f5f59c6139  subdir/file2.txt
1eb2267bae4e47cab81f8866bbc7e06764ea9be0  subdir/subsub/file3.txt

Now work with the hasher

Clear the hasher just to be sure

rclone backend drop hasher:

Confirm no hashes

rclone hashsum SHA1 hasher:

                                          file1.txt
                                          subdir/file2.txt
                                          subdir/subsub/file3.txt

Regular import

rclone backend import hasher: SHA1 hashes.sha1

verify

rclone hashsum SHA1 hasher:

38be7d1b981f2fb6a4a0a052453f887373dc1fe8  file1.txt
639daad06642a8eb86821ff7649e86f5f59c6139  subdir/file2.txt
1eb2267bae4e47cab81f8866bbc7e06764ea9be0  subdir/subsub/file3.txt

Drop and then verify it is dropped

rclone backend drop hasher:
rclone hashsum SHA1 hasher:

                                          file1.txt
                                          subdir/file2.txt
                                          subdir/subsub/file3.txt

Now use a sticky import

rclone backend stickyimport hasher: SHA1 hashes.sha1 -vv

2022/01/11 14:52:16 DEBUG : Setting --config "temp.cfg" from environment variable RCLONE_CONFIG="temp.cfg"
2022/01/11 14:52:16 DEBUG : Setting --password-command "rclone-pass-store echo" from environment variable RCLONE_PASSWORD_COMMAND="rclone-pass-store echo"
2022/01/11 14:52:16 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "backend" "stickyimport" "hasher:" "SHA1" "hashes.sha1" "-vv"]
2022/01/11 14:52:16 DEBUG : Using config file from "/Users/<USER>/temp.cfg"
2022/01/11 14:52:16 INFO  : Hasher is EXPERIMENTAL!
2022/01/11 14:52:16 DEBUG : Creating backend with remote "crypt:"
2022/01/11 14:52:16 DEBUG : Creating backend with remote "testdir"
2022/01/11 14:52:16 DEBUG : fs cache: renaming cache item "testdir" to be canonical "/Users/<USER>/testdir"
2022/01/11 14:52:16 DEBUG : hasher::hasher:: Groups by usage: cached [sha1], passed [], auto [sha1], slow [], supported [sha1]
2022/01/11 14:52:16 DEBUG : Creating backend with remote "hashes.sha1"
2022/01/11 14:52:16 DEBUG : fs cache: adding new entry for parent of "hashes.sha1", "/Users/<USER>"
2022/01/11 14:52:16 DEBUG : crypt~hasher.bolt: Opened for writing in 24.78771ms
2022/01/11 14:52:17 INFO  : Summary: 3 checksum(s) imported
2022/01/11 14:52:17 DEBUG : 5 go routines active

Test it

rclone hashsum SHA1 hasher: -vv

2022/01/11 14:52:17 DEBUG : Setting --config "temp.cfg" from environment variable RCLONE_CONFIG="temp.cfg"
2022/01/11 14:52:17 DEBUG : Setting --password-command "rclone-pass-store echo" from environment variable RCLONE_PASSWORD_COMMAND="rclone-pass-store echo"
2022/01/11 14:52:17 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:" "-vv"]
2022/01/11 14:52:17 DEBUG : Creating backend with remote "hasher:"
2022/01/11 14:52:17 DEBUG : Using config file from "/Users/<USER>/temp.cfg"
2022/01/11 14:52:17 INFO  : Hasher is EXPERIMENTAL!
2022/01/11 14:52:17 DEBUG : Creating backend with remote "crypt:"
2022/01/11 14:52:17 DEBUG : Creating backend with remote "testdir"
2022/01/11 14:52:17 DEBUG : fs cache: renaming cache item "testdir" to be canonical "/Users/<USER>/testdir"
2022/01/11 14:52:17 DEBUG : hasher::hasher:: Groups by usage: cached [sha1], passed [], auto [sha1], slow [], supported [sha1]
2022/01/11 14:52:17 DEBUG : crypt~hasher.bolt: Opened for reading in 81.385µs
                                          file1.txt
                                          subdir/subsub/file3.txt
                                          subdir/file2.txt
2022/01/11 14:52:17 DEBUG : 5 go routines active
2

Any thoughts @ivandeex ?

3

Looks like a bug. Need to reproduce but can't switch for rclone immediately. Can one "mark" a forum topic "assigned" on themselves? If this was a tracker ticket I'd assign on myself so I do not forget later.

4

No worries as if you think that is the case.

@jwink3101 - can you tag a bug on github and I can assign that to our man and we can link it here.

5

I tried to tag you in the post. I wonder if it has to be a reply to work.

I’ll make a GitHub issue and reference back.

This example consistently doesn’t work but I am not sure why. As I mentioned in the original thread, it failed in testing but then worked. Let me know if there’s more you need from me.

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.