Additional flag on rclone check to output hashsum

Feature
1

I've been looking through the docs, and I wanted to see if there's a way to print hashes from the rclone check command. Seems like check is already getting hashes from the remote, or downloading and hashing when using the --download flag, but there isn't a way to print those hashes from the check command.

The use case is that my org needs to provide content hashes for each file we manage. Currently I'm using a Jenkins pipeline to copy files from one remote to another, then run a check to verify integrity, and I could add a step to that to run hashsum on each file, but that would add a lot more time and if there's a way to cut that down some, I'd like to investigate it.

2

hello and welcome to the forum,

not sure your exact use case but this does print out the hash.

rclone check ~/test wasabi01:zork -vv
DEBUG : file.txt: md5 = c4ca4238a0b923820dcc509a6f75849b OK
3

Oh, I did not see this in the docs. Thank you, I'll give this a try!

4

yeah, rclone has different level of logging and formats such as json.

1 Like
5

and i assume that you know about

rclone hashsum md5 wasabi01:zork
c4ca4238a0b923820dcc509a6f75849b  file.txt
6

Yes, just didn't want to go through each file on the remote and re-hash it if I didn't have to, especially when I'm already running check --download

I'm usually copying genomic data, so these can be up to 50 GB per file, couple hundred files at a time. Anything I can do to cut down on time is preferred. :sweat_smile:

7

if you copy a file, rclone will print out the hash

rclone copy ~/test wasabi01:zork -vv
DEBUG : file.txt: md5 = c4ca4238a0b923820dcc509a6f75849b OK
INFO  : file.txt: Copied (new)
8

That is good to know, it'd be a lot quicker to diff that output. Thanks for the tip! I guess I need to be a little more diligent about looking at log levels before assuming something isn't happening.

9

Is there for the log to also output sha256 hashes, or just md5? I don't think I need both, just curious.

10

that is documented at
https://rclone.org/commands/rclone_hashsum/

11

Right, but not as a part of check? Only with hashsum?

12

Rclone will log any hashes it uses for the rclone check but it will only use one (if multiple are available).

Generally speaking which hashes are available depends on the backend in use - you can see which backends support which hashes in the overview docs.

You can of course calculate any hash with rclone hashsum --download but rclone check ---download will default to MD5 as that is fast and very good at detecting bit rot. There isn't away of forcing rclone check to use any particular checksum, but there probably could be!

1 Like
13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.