Is rclone verifying file retrieval when doing hash <TypeOfHash> --download

What is the problem you are having with rclone?

Not really a problem, but a question.

As I read in the documentation and was able to find out myself, rclone verifies the file transfer when sending and retrieving files if the remote peer supports hashing.
For example, when I upload a file to my Minio instance, rclone uses md5 hashing (if not disabled).

However, when I want to retrieve the sha256 of a file stored in my Minio instance, the data retrieval does not seem to be checked. At least I could not see any logs when I run something like:

 rclone hashsum sha256 name-from-my-minio-instance:test-file.bin --download -vvv

Am I wrong with my thesis?

Run the command 'rclone version' and share the full output of the command.

rclone 1.64.2

  • os/version: fedora 40 (64 bit)
  • os/kernel: 6.9.9-200.fc40.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

minio (s3)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 rclone hashsum sha256 name-from-my-minio-instance:test-file.bin --download -vvv

hi,
can you please do the following:

  1. rclone selfupdate
  2. post rclone config redacted
  3. post a full debug log

Thanks for the answer.

Did what you asked me todo.
So rclone version now prints out

rclone v1.67.0

  • os/version: fedora 40 (64 bit)
  • os/kernel: 6.9.9-200.fc40.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: none

rclone config redacted

[dietpi_minio-dev]
type = s3
provider = Minio
access_key_id = XXX
secret_access_key = XXX
endpoint = https://192.168.177.26:9012
no_check_certificate = true

debug log

rclone hashsum sha256 dietpi_minio-dev:test1/lubuntu-24.04-desktop-amd64.iso -vvv --no-check-certificate --download

2024/07/27 21:26:36 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "hashsum" "sha256" "dietpi_minio-dev:test1/lubuntu-24.04-desktop-amd64.iso" "-vvv" "--no-check-certificate" "--download"]
2024/07/27 21:26:36 DEBUG : Creating backend with remote "dietpi_minio-dev:test1/lubuntu-24.04-desktop-amd64.iso"
2024/07/27 21:26:36 DEBUG : Using config file from "/home/andreask/.config/rclone/rclone.conf"
2024/07/27 21:26:36 DEBUG : Resolving service "s3" region "us-east-1"
2024/07/27 21:26:36 DEBUG : fs cache: adding new entry for parent of "dietpi_minio-dev:test1/lubuntu-24.04-desktop-amd64.iso", "dietpi_minio-dev:test1"
f7ed99b368e00ab7f3fa2c05cefd912f43fc19a8ab9d4d241432d7d9e97e3491 lubuntu-24.04-desktop-amd64.iso
2024/07/27 21:27:42 DEBUG : 8 go routines active

minio support MD5 based hashes natively. So that is what rclone uses for file integrity checks. Rclone does not support SHA256 hashes when used with S3 (I'm not 100% sure if you can use sha256 hashes with s3). So your rclone hashsum sha256 --download command will stream the file from the remote and report the sha256 hash. That is a great way of double checking file integrity as you actually fetch it.

(I'm not 100% sure if you can use sha256 hashes with s3)

Yes, it does. Checking object integrity - Amazon Simple Storage Service

So your rclone hashsum sha256 --download command will stream the file from the remote and report the sha256 hash.

Yes, and that is my question / ("concern"). If I'm not mistaken this streaming process isn't checked / protected by using a already known hashsum which the remote supports. Yes, I know you could call me kind of persimistic or paranoid but the download is checked/verified atleast if a supported hashsum exists.

That is a great way of double checking file integrity as you actually fetch it.

But that just works if I know the sha256 before retrieving the data.

All transfers to and from S3 are checked with md5 sums.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.