Hasher overlay remote, generate missing hashes

I'm using hasher overlay for crypt remote, and i know there is a bit brute force method to generate hashes, like this: rclone hashsum md5 --download hasherremote:/folder > /dev/null, but what if i want only the missing hashes to be generated? So those hashes that are not in checksum cache .bolt database files.
Is it possible to do this?

I think if you remove --download and add --hasher-auto-size 1P it should do what you want.

as i see it also re checks all files, not just the ones without hash

It does walk the whole tree, however if a file has a hash cached in the db I think it should return the cached hash instead of recalculating the hash (which would require downloading and reading the entire file). So I think this should achieve your goal, unless I'm misunderstanding the goal.

If it doesn't seem to behave as I describe, it would be helpful to see a debug log to investigate further.

I think you're right.
For improvement: i see that if i use the --hasher-auto-size and copying files in same remote, so rclone uses server-side copy, then the hashes aren't just copied to the new files, but re-hashing all.
I think copying the hashes is enough for server-side copy. I don't know how rclone verifies the copies it not server-side, but maybe for those its enough too.

1 Like

That's a good idea. I would suggest opening a Feature Request issue on GitHub.

i did: Hasher, copy hashes on server-side copy (maybe simple copy too) · Issue #7815 · rclone/rclone · GitHub

1 Like