Strange output for cryptcheck

asdffdsa · April 7, 2020, 3:00pm

What is the problem you are having with rclone?

when running cryptcheck, i am getting the following in log.

2020/04/07 07:31:23 DEBUG : Aldous.Huxley's.Brave.New.World-ek5vse2_Aq0.mp4: OK - could not check hash
2020/04/07 07:31:24 DEBUG : FutureShock.OrsonWelles-vVJrJk3q3MA.mp4: OK
2020/04/07 07:31:24 DEBUG : FutureShock.OrsonWelles-vVJrJk3q3MA.mp4: OK

notice that

what does this mean and why cannot rclone check the hash but the file is OK?
OK - could not check hash
for each OK in the log, there are two entries.
rclone is not checking hashes. the command would take a long time with hundreds of movie files to hash.
thanks

What is your rclone version (output from `rclone version`)

v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

windows10.64

Which cloud storage system are you using? (eg Google Drive)

wasabi, using crypted remote

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

2020/04/07 07:34:59 DEBUG : rclone: Version "v1.51.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "cryptcheck" "s:\\data\\m\\media\\movies\\" "wasabimediacrypt:/m/media/movies/" "--progress" "--log-level=DEBUG" "--log-file=C:\\data\\rclone\\scripts\\rr\\other\\test\\log.include.txt"]

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

2020/04/07 07:34:59 DEBUG : rclone: Version "v1.51.0" starting with parameters ["c:\data\rclone\scripts\rclone.exe" "cryptcheck" "s:\data\m\media\movies\" "wasabimediacrypt:/m/media/movies/" "--progress" "--log-level=DEBUG" "--log-file=C:\data\rclone\scripts\rr\other\test\log.include.txt"]
2020/04/07 07:34:59 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/04/07 07:34:59 DEBUG : Using config file from "c:\data\rclone\scripts\rclone.conf"
2020/04/07 07:34:59 INFO : Using MD5 for hash comparisons
2020/04/07 07:34:59 INFO : Encrypted drive 'wasabimediacrypt:/m/media/movies/': Waiting for checks to finish
2020/04/07 07:34:59 DEBUG : Aldous.Huxley's.Brave.New.World-ek5vse2_Aq0.mp4: OK - could not check hash
2020/04/07 07:35:00 DEBUG : FutureShock.OrsonWelles-vVJrJk3q3MA.mp4: OK
2020/04/07 07:35:00 DEBUG : FutureShock.OrsonWelles-vVJrJk3q3MA.mp4: OK
2020/04/07 07:35:07 NOTICE: Encrypted drive 'wasabimediacrypt:/m/media/movies/': 0 differences found
2020/04/07 07:35:07 NOTICE: Encrypted drive 'wasabimediacrypt:/m/media/movies/': 148 hashes could not be checked
2020/04/07 07:35:07 NOTICE: Encrypted drive 'wasabimediacrypt:/m/media/movies/': 384 matching files
2020/04/07 07:35:07 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 384 / 384, 100%
Elapsed time: 7.1s

ncw · April 8, 2020, 5:07pm

What it looks like here is all the large files have no hash.

Large files don't have a hash natively on S3, however in v1.40 rclone started adding a bit of extra metadata with the hash.

I'm guessing these files were uploaded with an old version of rclone or with a different tool. Is that correct?

asdffdsa · April 8, 2020, 5:10pm

hi,
for all operations, v1.51.0
a few days ago i uploaded 500+GB to wasabi.
and when i tried to run cryptcheck, i got those messages.

another tool?
besides rclone, that can uploaded files to a rclone crypt?

ncw · April 8, 2020, 7:39pm

Sorry missed the fact you were using crypt - sorry.

Can you check the underlying files have md5sums? Do this with

rclone md5sum wasabi:bucket

where wasabi is your unencrypted remote (the remote line in your crypt setup).

Can you paste your config (with secrets removed) and the command you used for uploading?

asdffdsa · April 8, 2020, 7:46pm

"wasabimediacrypt": {
    "directory_name_encryption": "true",
    "filename_encryption": "standard",
    "remote": "wasabieast2:abcdefg",
    "type": "crypt"

rclone.exe md5sum wasabieast2:abcdefg

3dcf09328ad729e9865bdf47800168b9  t67sduuj5vhckknctoo3gdjoj8/atlvb0u8l50b3hv7e82doui1mk/nbg0p20au18m7pc997d15jrh9o/aimcranaad4stoaisml5qt60ss
56ed161fc5f76df384fdb2f17fd13b8c  t67sduuj5vhckknctoo3gdjoj8/atlvb0u8l50b3hv7e82doui1mk/nbg0p20au18m7pc997d15jrh9o/u8eu7fd4imb6pa4dm827k3gct8
4ad0071b37451e2f1ca77faa904604eb  t67sduuj5vhckknctoo3gdjoj8/atlvb0u8l50b3hv7e82doui1mk/ts8j3trmbfkt5lq9k1gsenf4bs/0ia1e07i8ab4f62m4p4s59ug1o
0fec278ea8094a072b484f25c2bd4cf5  t67sduuj5vhckknctoo3gdjoj8/atlvb0u8l50b3hv7e82doui1mk/ts8j3trmbfkt5lq9k1gsenf4bs/1uku4a0ubck3qem7uh6lkc230g

ncw · April 8, 2020, 7:52pm

Did all the files have md5sums?

asdffdsa · April 8, 2020, 8:10pm

for this file, there is no hash

t67sduuj5vhckknctoo3gdjoj8/b9269d796dt1j2er0eq7gjfbrs/a8b66hvf9pl7d1jfmo9sj0b4ns/6ot3tb5v4j8edpc2d6i1tkgl670toqkkf5p2o8cu97nl2kc95cbbeimq4vh4152parj3uq6clmjljivpha0vpn219aup2ljd6bbjmddprmasqe8sp3tpl8plh8ij5387

command:
rclone cryptdecode wasabimediacrypt: t67sduuj5vhckknctoo3gdjoj8/b9269d796dt1j2er0eq7gjfbrs/a8b66hvf9pl7d1jfmo9sj0b4ns/6ot3tb5v4j8edpc2d6i1tkgl670toqkkf5p2o8cu97nl2kc95cbbeimq4vh4152parj3uq6clmjljivpha0vpn219aup2ljd6bbjmddprmasqe8sp3tpl8plh8ij5387

output:

t67sduuj5vhckknctoo3gdjoj8/b9269d796dt1j2er0eq7gjfbrs/a8b66hvf9pl7d1jfmo9sj0b4ns/6ot3tb5v4j8edpc2d6i1tkgl670toqkkf5p2o8cu97nl2kc95cbbeimq4vh4152parj3uq6clmjljivpha0vpn219aup2ljd6bbjmddprmasqe8sp3tpl8plh8ij5387 m/media/movies/Who's.Afraid.Of.Virginia.Woolf.1966.720p.BluRay.x264-[YTS.AG].mp4

and the file size is 937.3MB

ncw · April 9, 2020, 10:24am

OK... Files without hashes will cause the problem.

Did you transfer them from somewhere other than local disk? If you did a cloud -> cloud copy from a source which didn't support md5s that could explain it.

asdffdsa · April 9, 2020, 1:38pm

i used rclone v1.51.0 to copy local to crypt at wasabi.

you wrote that
"adding a bit of extra metadata with the hash."
and
"Files without hashes will cause the problem"

if rclone adds the hash and the some hashes are missing, that sounds like a rclone bug?

ncw · April 9, 2020, 3:52pm

OK I see what is going on.

For the small files (below --s3-upload-cutoff) everything is fine.

For the large files above --s3-upload-cutoff what is happening is this

When rclone uploads a chunked file to s3

it asks the source backend for the MD5SUM of the file before the transfer
if the source backend does not have that then the MD5SUM will be blank
In the case of crypt, it does not have the MD5SUM - it would need to read the file then MD5 it first

This could be fixed by

Making a special case in crypt so that if the source file is a local file, we do read it, encrypt the data and md5sum it to produce the hash. This wouldn't be very much more intensive than hashing a local file
If the md5 can't be read from the source backend, then calculate it on the fly and add it to the metadata afterwards. Note that you can only supply the metadata at the start of the process so supplying it at the end would mean doing an S3 copy operation which can be expensive for large objects. We did think of doing this before but decided that people wouldn't appreciate the extra costs.

I have had a go with option 1 which I think is quite a good compromise - let me know what you think.

https://beta.rclone.org/branch/v1.51.0-153-gd3d174b5-fix-crypt-src-hash-beta/ (uploaded in 15-30 mins)

asdffdsa · April 9, 2020, 4:06pm

thanks,
yes, i was doing some testing and i was thinking it had something to do with multi-part uploads.
has something to do with --s3-chunk-size.

i am little confused by all this.
i just want to have a hash for every file in every scenario.

option 1 is good because there be a hash for all crypted files.
but i think i will have to re-upload the 1TB of data, as i need the hash. no big deal.

option 2, when
--- uploading local file
and
uploading to non-crypted remotes like wasabi, s3 clone.
if there any scenario where there will not be a hash?

also, when testing the beta, with crypted remote.
how can i, for sure, trigger the problem.
is there a rclone command flags that will be a good test of the new beta.

i see the beta is up and i am downloading now...
again, thanks

ncw · April 9, 2020, 4:10pm

Unavoidable alas!

Option 2 is the "perfect" option - there will never not be a hash.

Option 1 won't have a hash if you transfer from a non-local remote which doesn't support MD5 hashes (no crypt) or any non-local remote (with crypt).

Let me know how you get on with the beta. Try some transfers then try some cryptcheck.

asdffdsa · April 9, 2020, 4:11pm

i am lucky to have wasabi and 1Gbps uploads, and of course rclone

when testing the beta, with crypted remote.
how can i, for sure, trigger the problem.
is there a rclone command flags that will be a good test of the new beta.

ncw · April 9, 2020, 4:13pm

Uploading large files to crypted wasabi will mean that cryptcheck complains about missing hashes with 1.51.0 but that should be fixed with the beta

Whatever you were doing last time worked quite well!

asdffdsa · April 9, 2020, 4:20pm

the beta has fixed that probem.

but for each crypted file, there are two entries in the log.
is rclone?
checking each file twice
or
just double-logging

2020/04/09 12:18:29 DEBUG : rclone: Version "v1.51.0-153-gd3d174b5-fix-crypt-src-hash-beta" starting with parameters ["C:\data\rclone\scripts\rr\other\test\rclone-v1.51.0-153-gd3d174b5.exe" "cryptcheck" "Z:\m\test\" "secret1:" "--log-file=C:\data\rclone\scripts\rr\other\test\testcrypttest01.txt" "--log-level=DEBUG"]
2020/04/09 12:18:29 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/04/09 12:18:29 DEBUG : Using config file from "c:\data\rclone\scripts\rclone.conf"
2020/04/09 12:18:29 INFO : Using MD5 for hash comparisons
2020/04/09 12:18:29 DEBUG : Encrypted drive 'secret1:': Waiting for checks to finish
2020/04/09 12:18:29 DEBUG : 001.txt: OK
2020/04/09 12:18:29 DEBUG : 001.txt: OK
2020/04/09 12:18:29 DEBUG : 002.txt: OK
2020/04/09 12:18:29 DEBUG : 002.txt: OK
2020/04/09 12:18:29 DEBUG : 020MB: OK
2020/04/09 12:18:29 DEBUG : 020MB: OK
2020/04/09 12:18:29 DEBUG : 100.txt: OK
2020/04/09 12:18:29 DEBUG : 100.txt: OK
2020/04/09 12:18:30 DEBUG : 100MB: OK
2020/04/09 12:18:30 DEBUG : 100MB: OK
2020/04/09 12:18:30 DEBUG : 200MB: OK
2020/04/09 12:18:30 DEBUG : 200MB: OK
2020/04/09 12:18:31 DEBUG : 300MB: OK
2020/04/09 12:18:31 DEBUG : 300MB: OK
2020/04/09 12:18:31 NOTICE: Encrypted drive 'secret1:': 0 differences found
2020/04/09 12:18:31 NOTICE: Encrypted drive 'secret1:': 7 matching files

asdffdsa · April 9, 2020, 4:25pm

also, good is that i see new entries when uploading
2020/04/09 12:18:14 DEBUG : 100MB: Found object directly
2020/04/09 12:18:14 DEBUG : 100MB: Computing MD5 hash on 100MB

ncw · April 9, 2020, 5:20pm

Yes it is just a big of double logging... Try this (which has the previous stuff in too)

https://beta.rclone.org/branch/v1.51.0-154-ga283c655-fix-crypt-src-hash-beta/ (uploaded in 15-30 mins)

system · April 12, 2020, 5:20pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.