Is the checksum command synchronous instead of parallel?

What is the problem you are having with rclone?

When running the checksum command on the local filesystem, the --progress output shows files being processed one at a time instead of in parallel. I tried setting --checkers=4 but it still does them one at a time.

It's probably not a display problem with --progress because the output file from --combined lists the files in a sorted order (breadth first?), unlike the --output-file of the hashsum command where the hashes are listed out-of-order because the parallelism makes the order unpredictable.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: Microsoft Windows 11 Enterprise 22H2 (64 bit)
- os/kernel: 10.0.22621.1702 Build 22621.1702.1702 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone checksum sha1 c:\tmp\hashes.sum c:\a_folder
--combined c:\tmp\checksums.diff
-v --log-file=c:\tmp\checksumlogs.log

The rclone config contents with secrets removed.

I don't have a config file. Rclone's using it's default settings.

2023/05/30 21:51:04 NOTICE: Config file "C:\\Users\\ORYLY\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults

A log from the command with the -vv flag

(I manually redacted the actual folder names. It should be consistent with the actual output, but if some path looks weird then I might've flubbed the redaction.)

2023/05/30 22:03:14 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "checksum" "sha1" "c:/tmp/hashes.sum" "c:/a_folder/" "--combined" "c:/tmp/checksums.diff" "--progress" "-vv" "--log-file=c:/tmp/checksumlogs.log"]
2023/05/30 22:03:14 DEBUG : Creating backend with remote "c:/tmp/hashes.sum"
2023/05/30 22:03:14 NOTICE: Config file "C:\\Users\\ORYLY\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults
2023/05/30 22:03:14 DEBUG : fs cache: adding new entry for parent of "c:/tmp/hashes.sum", "//?/C:/tmp"
2023/05/30 22:03:14 DEBUG : Creating backend with remote "c:/a_folder/"
2023/05/30 22:03:14 DEBUG : fs cache: renaming cache item "c:/a_folder/" to be canonical "//?/c:/a_folder"
2023/05/30 22:03:14 DEBUG : video/example.mp4: sha1 = a109860fda68c56967f6830b7c9172f13a1aa748 OK
2023/05/30 22:03:14 DEBUG : photos/example.jpg: sha1 = f621834594b7b2575335b386377428739b4c34dd OK
<and so on for all the files>

To my understanding, it happens one by one against each file and I am not aware of a way to change that.

Yes checksum happens sequentially. No reason why it shouldn't happen in parallel though. That will randomise the output order though, but I don't think that is guaranteed anyway.

Oh, it turns out that the rlcone check command does nearly the same thing while operating in parallel, and you can make it act like rclone checksum if you supply a --checkfile to it.

Bikeshed warning: maybe to the documentation of both commands to distinguish their differences and when each should be used?

Yes I'd recommend rclone check here

Yes there is some overlap between those commands and we should probably make rclone checksum work the same as rclone check as far as parallel working is concerned.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.