Could Chunker's Hash Types Include QuickXOR for OneDrive?

jared · July 23, 2024, 2:06pm

What is the problem you are having with rclone?

the supported hash types by chunker, according to the docs, are MD5 an SHA1. the business version of Onedrive uses Quickxor hashes, and the consumer version of Onedrive recently switched from SHA1 to Quickxor as well. i believe that means Chunker cannot hash files to verify transfer now.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
os/kernel: 10.0.17763.4645 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.22.4
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Microsoft Onedrive and Microsoft Sharepoint

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone check --verbose --progress "E:\Machine Backups\Veeam\Poultry PC" "SLSharePointCryptChunk:Poultry PC"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[SLSharePointCryptChunk]
type = chunker
remote = SLSharePointCrypt:
chunk_size = 200Mi
hash_type = sha1quick

A log from the command that you were trying to run with the `-vv` flag

C:\rclone>rclone check --verbose --progress "E:\Machine Backups\Veeam\Poultry PC" "SLSharePointCryptChunk:Poultry PC" -vv
2024/07/23 09:02:03 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "check" "--verbose" "--progress" "E:\\Machine Backups\\Veeam\\Poultry PC" "SLSharePointCryptChunk:Poultry PC" "-vv"]
2024/07/23 09:02:03 DEBUG : Creating backend with remote "E:\\Machine Backups\\Veeam\\Poultry PC"
2024/07/23 09:02:03 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/07/23 09:02:03 DEBUG : fs cache: renaming cache item "E:\\Machine Backups\\Veeam\\Poultry PC" to be canonical "//?/E:/Machine Backups/Veeam/Poultry PC"
2024/07/23 09:02:03 DEBUG : Creating backend with remote "SLSharePointCryptChunk:Poultry PC"
2024/07/23 09:02:03 DEBUG : Creating backend with remote "SLSharePointCrypt:Poultry PC"
2024/07/23 09:02:03 DEBUG : Creating backend with remote "SLSharePoint:Veeam/笠歨䯔⇺㔴悁揙琽鉟"
2024/07/23 09:02:04 DEBUG : Reset feature "ListR"
2024/07/23 09:02:04 INFO  : Using sha1 for hash comparisons
2024/07/23 09:02:04 DEBUG : Chunked 'SLSharePointCryptChunk:Poultry PC': Waiting for checks to finish
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-14T091404_ADB3.vib: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-05-04T220201_38A9.vib: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-21T114455_12B6.vib: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-03-21T101607_0A85.vib: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-01T201359_5A40.vib: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-04-20T201244_9CFB.vib: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-01T162412_D872.vbk: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-05-04T220201_38A9.vib: OK - could not check hash
2024/07/23 09:02:05 DEBUG : Poultry PC.7D2024-05-27T091412_73FC.vbk: Dst hash empty - aborting Src hash check
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-14T091404_ADB3.vib: OK - could not check hash
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-03-21T101607_0A85.vib: OK - could not check hash
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-21T114455_12B6.vib: OK - could not check hash
2024/07/23 09:02:05 DEBUG : Poultry PC.4D2024-02-01T201359_5A40.vib: OK - could not check hash

kapitainsky · July 23, 2024, 2:44pm

Use:

hash_type = sha1all

or

hash_type = md5all

This way ALL files will have hash calculated and stored. Especially when you use crypt remote as well (no hash support). In such case even if quickXOR was supported by chunker it would not help.

You chose sha1quick which falls back to MD5 (which is not available for OneDrive nor crypt).

jared · July 24, 2024, 8:50pm

ok. so long story short, i can not get hashes when i wrap a crypt remote with a chunker remote?

kapitainsky · July 25, 2024, 3:39am

You can. Use chunkier hash options I suggested.

jared · July 25, 2024, 7:50pm

ok. here is as i updated the config file:

[SLSharePointCryptChunk]
type = chunker
remote = SLSharePointCrypt:
chunk_size = 200Mi
hash_type = sha1all

yet, here is the log output when running a check:

C:\rclone>rclone check --verbose --progress "E:\Machine Backups\Veeam\Poultry PC" "SLSharePointCryptChunk:Poultry PC" -vv
2024/07/25 14:45:22 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "check" "--verbose" "--progress" "E:\\Machine Backups\\Veeam\\Poultry PC" "SLSharePointCryptChunk:Poultry PC" "-vv"]
2024/07/25 14:45:22 DEBUG : Creating backend with remote "E:\\Machine Backups\\Veeam\\Poultry PC"
2024/07/25 14:45:22 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/07/25 14:45:22 DEBUG : fs cache: renaming cache item "E:\\Machine Backups\\Veeam\\Poultry PC" to be canonical "//?/E:/Machine Backups/Veeam/Poultry PC"
2024/07/25 14:45:22 DEBUG : Creating backend with remote "SLSharePointCryptChunk:Poultry PC"
2024/07/25 14:45:23 DEBUG : Creating backend with remote "SLSharePointCrypt:Poultry PC"
2024/07/25 14:45:23 DEBUG : Creating backend with remote "SLSharePoint:Veeam/笠歨䯔⇺㔴悁揙琽鉟"
2024/07/25 14:45:23 DEBUG : OneDrive root 'Veeam/笠歨䯔⇺㔴悁揙琽鉟': Token expired but no uploads in progress - doing nothing
2024/07/25 14:45:23 DEBUG : SLSharePoint: Loaded invalid token from config file - ignoring
2024/07/25 14:45:24 DEBUG : Saving config "token" in section "SLSharePoint" of the config file
2024/07/25 14:45:24 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/07/25 14:45:24 DEBUG : SLSharePoint: Saved new token in config file
2024/07/25 14:45:25 DEBUG : Reset feature "ListR"
2024/07/25 14:45:25 INFO  : Using sha1 for hash comparisons
2024/07/25 14:45:25 DEBUG : Chunked 'SLSharePointCryptChunk:Poultry PC': Waiting for checks to finish
2024/07/25 14:45:26 DEBUG : Poultry PC.102D2024-07-22T135210_57CB.vbk: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-03-21T101607_0A85.vib: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-02-14T091404_ADB3.vib: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-02-01T162412_D872.vbk: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-05-04T220201_38A9.vib: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-02-21T114455_12B6.vib: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-02-01T201359_5A40.vib: Dst hash empty - aborting Src hash check
2024/07/25 14:45:26 DEBUG : Poultry PC.4D2024-04-20T201244_9CFB.vib: Dst hash empty - aborting Src hash check

kapitainsky · July 25, 2024, 10:51pm

It will only work for files uploaded via this chunker remote with new hash_type settings. No composite files uploaded before have any hash stored.

So you have to not only update your config file but re-upload all files.

jared · July 27, 2024, 1:34am

ah ok. well, i use hashing to make sure files don't go corrupt over time. so a stored hash is not ideal. it's happened to me in the past, so i'm kind of paranoid. so some day (i say some day cause i don't know how feasible something like this is) ideally chunker could transparently assemble files and hash them. without downloading them from the remote (a)

thanks for taking the time to ruffle through this for me

kapitainsky · July 27, 2024, 5:40am

Ok. In such case you could force content download during check.

As per rclone check docs:

If you supply the --download flag, it will download the data from both remotes and check them against each other on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.

system · August 26, 2024, 5:40am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.