File reads different hash after each download, causing retries

What is the problem you are having with rclone?

When copying files from a google drive to my local file system (external SSD to be precise) one file seems to get messed up over and over again.
After downloading lots of files, two of them were faulty. Retrying fixed one of the errors, but the last file seems to be broken no matter what and how often I retry:

$ rclone copy drive:'/stuff/' local:'/Volumes/stuff/stuff/' --progress --checksum
2024/05/06 18:08:19 ERROR : problem_file.partial: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "5350313b2b9606d15ad8d1f9c0b24aca"
2024/05/06 18:08:19 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "5350313b2b9606d15ad8d1f9c0b24aca"
2024/05/06 18:15:59 ERROR : problem_file.partial: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "2f1464e6e6ee22a906cc6b1109cfc07b"
2024/05/06 18:15:59 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "2f1464e6e6ee22a906cc6b1109cfc07b"
2024/05/06 18:22:23 ERROR : problem_file.partial: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "ade3c797e21e736619daed39c1a33357"
2024/05/06 18:22:24 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "ade3c797e21e736619daed39c1a33357"
Transferred:   	   15.941 GiB / 15.941 GiB, 100%, 530.767 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                69 / 69, 100%
Elapsed time:     20m46.5s
2024/05/06 18:22:24 Failed to copy: corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "ade3c797e21e736619daed39c1a33357"

I've been through this over and over again, each time the hash for my freshly downloaded file is different while other files I downloaded as well are not affected and perfectly fine.

Downloading the file in question via my web browser and pasting it into the folder seems to have worked, the hashes now match and the downloads are skipped. But after deleting the file, it once again fails to download that particular file (see full log below)

How can I prevent such problems in the future?

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: darwin 14.1.2 (64 bit)
- os/kernel: 23.1.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy drive:'/stuff/' local:'/Volumes/stuff/stuff/' --progress --checksum

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

The drives in question:

[drive]
type = drive
token = XXX
team_drive = 
root_folder_id = XXX

[local]
type = local

A log from the command that you were trying to run with the -vv flag

FULL LOG: rclone error log - Pastebin.com (exceeded char limit for post, sry)

Note the different hashes each time.
From an earlier attempt:

[...] corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "d36513303893610144fc8a8685638f53"
[...]
[...] corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "44c137176f8e50779e75bd89b7cb630e"
[...]
[...] corrupted on transfer: md5 hash differ "d3391c54d761c7e8a9babca6b9a0cbf1" vs "b037a47a90c547c722f934eaf229af3b"
[...]

welcome to the forum,

RATE_LIMIT_EXCEEDED
first, should fix that.
as per rclone docs, create your own client id+secret and then re-create the remote from scratch.

to keep the log small, copy a single file, use --retries=1 --inplace

thx :slight_smile:

I am still not quite sure what has caused the issues (which shortly after spread to any and all downloads I attempted, each of them failing with a hash mismatch) of if the mistake is even to be found within rclone and its setup itself.

Creating a separate API key did not solve the problem, however applying --multi-thread-streams 1 seems to have fixed it for now...

Is this a common problem / workaround?

i do not know.

in my case, due to recent changes to multi-thread behavior and some bug/issues, i use --multi-thread-streams=0

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.