Read that 'MD5/SHA1 hashes checked at all times' but logs state otherwise

Help and Support
1

hello and thanks for taking the time to read this.

I am a newbiew, have been using rclone and testing it.
I read that ‘MD5/SHA1 hashes checked at all times’ but whenever I look at the logfile, I always see that no files are checked. I read on the forum about in some cases, checksum is not fone for large files.
So the file I uploaded is just three bytes.

can someone help me understand if all files are checked at all times or do I need to run a rclone check after a rclone copy?

there seems to be a lot of confusion in the forum about when checksums are used and when they are not.
perhaps someone can create a dedicated faq about it.

thanks much,
jojo.the.humanmonkey

‘Transferred: 3 / 3 Bytes, 100%, 3 Bytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 700ms’

2

When files are transferred rclone checks the checksum of the uploaded file vs the local file to ensure data integrity.

When syncing, rclone compares a local file with a remote file in various different ways.

  • by default it will use modification time and size
  • with --checksum it will use the checksum and size
  • with --size-only it will just use the size

If you want to verify your local files are the same as the remote files checking their sizes and checksums, then use rclone check.

3

thanks but the main webpage states that all transfer are checked but that does not seem to be true.
and how do I disable that picure of yourself in the forum?

4

How are you seeing that?

You mean disable the images of people in the forum? I don’t know if there is a setting for that, but check the forum settings!

5

hi and thanks for the responding.

from the homepage https://rclone.org/
“MD5/SHA1 hashes checked at all times for file integrity”

but when I look at the logs, the file that are checked are 0

"Transferred: 1.871G / 1.871 GBytes, 100%, 4.206 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 5 / 5, 100%
Elapsed time: 7m35.5s’

thanks

6

A “check” is when there is a file there and it compares. You have 5 files transferred so no checks since they are new.

7

thanks the the quick reply.
i am stil confused as the hompage is clear
"MD5/SHA1 hashes checked at all times for file integrity’
at ALL times, but clearly this is not the case, correct?

and also the claim ‘rsync for the cloud’ does not seem to be true as rclone does not do delta updates and has nothing at all in common with rsync, correct?

i am liking rlcone and it is very easy to use but the homepage should be accurate and truthful about what the software is and does.

thanks much,
david

8

Yes, rclone has a sync function almost identical to rsync. The command is called ‘sync’.

https://rclone.org/commands/rclone_sync/

I’m not sure what you mean by not checked as you have options to use other methods to check.

It depends on how much you want to walk down the semantics journey.

It doesn’t check at “all” times because you can use an option to check other ways.

If you find something incorrect, you can always submit a pull request to get it adjusted.

9

thanks and this is my last attempt to clarify. rclone is nothing at all like rsync.
most any software can copy and sync ENTIRE files, that is trivial and easy to implement.
.
rclone:if only one byte of a one gigabye file changes, rclone will copy the entire one gigabyte file, not just the one byte that changed.
rsync: can only update the one byte, not the entire one gigabyte.

I like rclone and use it everyday but please, do not compare rclone to rsync… that is an insult to rsync.


"The recipient splits its copy of the file into chunks and computes two checksums for each chunk: the MD5 hash, and a weaker but easier to compute ‘rolling checksum’.[23] It sends these checksums to the sender.

The sender quickly computes the rolling checksum for each chunk in its version of the file; if they differ, it must be sent. If they’re the same, the sender uses the more computationally expensive MD5 hash to verify the chunks are the same’

10

I’m familiar with rsync as I use it as well.

rclone sync and rsync are similar so not sure what your point is.

If you’d like to adjust some wording, feel free to submit a pull request.

Thanks and if you have any rclone questions, let me know and we’re more than happy to help :slight_smile:

1 Like
11

thanks for taking the time to comment