How to specify hash method to --checksum?

Help and Support
1

Hello,

I'm using rclone v1.50.2 and Nextcloud 17.0.1. When I specify -c (--checksum) flag to rclone copy command to my Nextcloud server, it fails.

Command: rclone copy -n -c $DIR/test2 NC:/tmp/test2 --log-file /tmp/rc.log

Log:
2019/11/22 14:37:27 NOTICE: webdav root 'tmp/test2': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

Nextcloud seems not be able to calc. MD5, however, SHA-1 is OK ("rclone hashsum SHA-1 NC:/tmp/test2" shows hash vals).

So, would you show me how to specify hash method (ie. MD5 or SHA-1) to --checksum flag?

Thanks in advance.

buttie

2

Hmm I think that must be a bug - the source and dest do have hashes in common, it is just that it all the MD5 hashes are blank...

So rclone thinks nextcloud support md5 and sha1, however all the md5 hashes will read as "".

Does nextcloud actually support md5 at all?

There isn't a way currently to specify which hash you use. Rclone chooses one for you. I could put a flag in for that to say which checksum you want if there is a choice (or maybe a preferred list).

If a remote supports more than one and rclone gets a blank for one of them it could try the next hash along. This would work but be quite inefficient.

I could implement a flag to say which checksums the webdav remote does support.

But maybe nextcloud really only does support sha1? Certainly that is what I see on my nexcloud instance.

Thoughts?

3

Hi, thanks for the reply.

Although I am not sure all Nextcloud instances has the same feature, my server returns only ""s to "rclone hashsum MD5".

Does nextcloud actually support md5 at all?

So, I can answer only for my server, it's "No".

I could put a flag in for that to say which checksum you want if there is a choice (or maybe a preferred list).

Yea, it would be very useful for me.

But maybe nextcloud really only does support sha1? Certainly that is what I see on my nexcloud instance.

Sorry, I am completely not sure about this. But, I suspect that the "server side encryption" feature, which is enabled on my server, may have something to do with it.

4

I just had a look through the webdav backend code...

The reason the md5 doesn't get set is because rclone uploads the sha1 in preference.

I wonder if that would be the best thing to fix - to make rclone upload both the md5 and the sha1.

What do you think?

5

Thanks for finding the reason.
Yes, I agree to your solution.

6

OK give this a go

When you upload things to nextcloud with it it will set both MD5 and SHA1 which should mean that syncing with --checksum works properly.

https://beta.rclone.org/branch/v1.50.2-060-g130ce531-fix-nextcloud-sums-beta/ (uploaded in 15-30 mins)

7

Thanks for the beta, however, it shows something strange behaviour.

I tried after removed all the files in the dest. dir on Nextcloud. rclone doesn't work as expected:

  1. Some file(eg. JPEG file (about 140KB))'s MD5 will not be obtained with "rclone hashsum MD5".

    $ ./tmp/rclone hashsum MD5 NC:/tmp/test2
    21004.jpg
    1d7fa5e8cd4c6178600b5debe09703c4 21006.png
    9928a8a56057c77d0328c2ccc7d71d07 AR6005.pdf
    d41d8cd98f00b204e9800998ecf8427e empty.txt
    331be4f8eb005dc20b448d93f1bbda97 sync-log.txt
    3fc7e8deb6d7194876e32af47614df55 test.txt

    $ ./tmp/rclone hashsum SHA-1 NC:/tmp/test2
    65b428c75e84b05439306da3f889d68b4fcd876b 21004.jpg
    21006.png
    AR6005.pdf
    empty.txt
    sync-log.txt
    test.txt

  2. After modified a file locally and "rclone copy -c", "no hashes" warning is shown and some files'(not the modified one) MD5 and SHA-1 will not be obtained.

    $ ./tmp/rclone copy -v -c test2 NC:/tmp/test2
    2019/11/25 05:20:21 INFO : webdav root 'tmp/test2': Waiting for checks to finish
    2019/11/25 05:20:21 NOTICE: webdav root 'tmp/test2': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
    2019/11/25 05:20:21 INFO : webdav root 'tmp/test2': Waiting for transfers to finish
    2019/11/25 05:20:25 INFO : test.txt: Copied (replaced existing)
    2019/11/25 05:20:25 INFO :
    Transferred: 47 / 47 Bytes, 100%, 10 Bytes/s, ETA 0s
    Checks: 0 / 5, 100%
    Transferred: 1 / 1, 100%
    Elapsed time: 4.5s

    $ ./tmp/rclone hashsum MD5 NC:/tmp/test2
    21006.png
    9928a8a56057c77d0328c2ccc7d71d07 AR6005.pdf
    empty.txt
    331be4f8eb005dc20b448d93f1bbda97 sync-log.txt
    3e854346a5b2e5bd52bd4a4988885431 test.txt

    $ ./tmp/rclone hashsum SHA-1 NC:/tmp/test2
    1e9996808f12a78d457b6171805b5612cf31f4d0 21006.png
    AR6005.pdf
    da39a3ee5e6b4b0d3255bfef95601890afd80709 empty.txt
    sync-log.txt
    test.txt

And, as I have misunderstood the hash feature, I would like to know:
For WebDAV server, hash values are calculated by rclone (not by the server), and sent to the server. Then, I wonder if a file is modified externally, update chaeck with hash(checksum) will fail: is it right?

8

That looks like your version of next cloud has ignored the two hashes rclone sent and just used one of them. That isn't what my nextcloud did.

What version of nextcloud are you using? I'm using 16.0.5

Here is how to find out: https://help.nextcloud.com/t/what-version-of-nextcloud-am-i-using/45220

If any hashes are blank you will get that warning.

I think for next cloud hash values are calculated by rclone but checked on upload.

This will detect if the file is modified on nextcloud somehow.

Maybe the thing to do is for rclone to declare that nextcloud only supports SHA1 hashes and just upload the one hash.

9

Hi,

My Nextcloud version is 17.0.1. I wonder its spec on hash vals. changed in this version...

And, thanks for the answer.

10

OK that explains it.

I think the best thing to to might be to make rclone only support SHA1 on nextcloud. This will be backwards compatiible with old versions of rclone and will fix your problem.

What do you think?

11

I agree to your proposal. It might be the best at this time.

12

OK give this a go!

https://beta.rclone.org/branch/v1.50.2-062-g9e8b73b3-fix-nextcloud-sums-beta/ (uploaded in 15-30 mins)

13

Hi, the beta is OK (worked as expected).

After copied files to Nextcloud, all files' SHA-1 hashes are displayed with "rclone hashsum SHA-1 NC:remote-dir",
and, changed a file locally then "rclone copy -v -c local-dir NC:remote-dir" succeeded without any warnings on --checksum.

Thank you.

14

Great - thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.51

15

I see. I am looking forward to v1.51.

1 Like
16

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.