Hashsum doesn't follow the expected order when used with --files-from

aliron19 · July 18, 2025, 6:11pm

What is the problem you are having with rclone?

When I use the hashsum command with the --files-from tag, it doens't process the files in the expected order, despite adding --no-traverse --transfers 1 --checkers 1

I first run this command to get a list of files of the current test directory (in my local file system):

rclone lsf -R --files-only ./ | sort --ignore-case > list_of_files

Which generates the file list_of_files with the following content:

dir1/file10
dir1/file11
dir1/file12
dir1/file13
dir1/file14
dir1/file15
dir1/file16
dir1/file17
dir1/file18
dir1/file19
file00
file01
file02
file03
file04
file05
file06
file07
file08
file09
list_of_files

Then, I run the this command to obtain the SHA1 hash of those files:

rclone hashsum sha1 --files-from list_of_files --no-traverse --transfers 1 --checkers 1 ./

I expected the files to be processed as they appear in list_of_files, in the same order, but instead they are hashed in a different way. This is the output I get:

da39a3ee5e6b4b0d3255bfef95601890afd80709  file00
da39a3ee5e6b4b0d3255bfef95601890afd80709  file01
da39a3ee5e6b4b0d3255bfef95601890afd80709  file02
da39a3ee5e6b4b0d3255bfef95601890afd80709  file03
da39a3ee5e6b4b0d3255bfef95601890afd80709  file04
da39a3ee5e6b4b0d3255bfef95601890afd80709  file05
da39a3ee5e6b4b0d3255bfef95601890afd80709  file06
da39a3ee5e6b4b0d3255bfef95601890afd80709  file07
da39a3ee5e6b4b0d3255bfef95601890afd80709  file08
da39a3ee5e6b4b0d3255bfef95601890afd80709  file09
f0c19035dab5178043ebc0d7d07b337df8d12a93  list_of_files
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file10
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file11
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file12
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file13
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file14
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file15
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file16
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file17
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file18
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file19

As you can see, files from file00 to file09 are hashed first, then list_of_files, and finally files form file10 to file19, which is not the order they appear in list_of_files.

Is it possible to force rclone hashsum sha1 to follow the order of list_of_files?

Thanks

Run the command 'rclone version' and share the full output of the command.

>rclone version
rclone v1.68.2
- os/version: Microsoft Windows 10 Pro 1909 (64 bit)
- os/kernel: 10.0.18363.900 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local files stored in a hard drive.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone hashsum sha1 --files-from list_of_files --no-traverse --transfers 1 --checkers 1 ./

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

It doesn't apply, since I'm using a local file system.

A log from the command that you were trying to run with the `-vv` flag

>rclone hashsum sha1 --files-from list_of_files --no-traverse --transfers 1 --checkers 1 ./ -vv
2025/07/18 20:03:49 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "hashsum" "sha1" "--files-from" "list_of_files" "--no-traverse" "--transfers" "1" "--checkers" "1" "./" "-vv"]
2025/07/18 20:03:49 DEBUG : Creating backend with remote "./"
2025/07/18 20:03:49 DEBUG : Using config file from "C:\\Users\\Bob\\.config\\rclone\\rclone.conf"
2025/07/18 20:03:49 DEBUG : fs cache: renaming cache item "./" to be canonical "//?/D:/test"
da39a3ee5e6b4b0d3255bfef95601890afd80709  file00
da39a3ee5e6b4b0d3255bfef95601890afd80709  file01
da39a3ee5e6b4b0d3255bfef95601890afd80709  file02
da39a3ee5e6b4b0d3255bfef95601890afd80709  file03
da39a3ee5e6b4b0d3255bfef95601890afd80709  file04
da39a3ee5e6b4b0d3255bfef95601890afd80709  file05
da39a3ee5e6b4b0d3255bfef95601890afd80709  file06
da39a3ee5e6b4b0d3255bfef95601890afd80709  file07
da39a3ee5e6b4b0d3255bfef95601890afd80709  file08
da39a3ee5e6b4b0d3255bfef95601890afd80709  file09
f0c19035dab5178043ebc0d7d07b337df8d12a93  list_of_files
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file10
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file11
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file12
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file13
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file14
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file15
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file16
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file17
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file18
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file19
2025/07/18 20:03:49 DEBUG : 2 go routines active

As I said in the first section, I wanted to know if is it possible to force rclone hashsum sha1 to follow the order of the lines in --files-from?

Thanks.

asdffdsa · July 18, 2025, 6:13pm

hi,

might want to rclone selfupdate and test again

aliron19 · July 18, 2025, 6:27pm

After updating to the latest version, I get the same results:

>rclone hashsum sha1 --files-from list_of_files --no-traverse --transfers 1 --checkers 1 ./ -vv
2025/07/18 20:15:30 DEBUG : rclone: Version "v1.70.3" starting with parameters ["rclone" "hashsum" "sha1" "--files-from" "list_of_files" "--no-traverse" "--transfers" "1" "--checkers" "1" "./" "-vv"]
2025/07/18 20:15:30 DEBUG : Creating backend with remote "./"
2025/07/18 20:15:30 DEBUG : Using config file from "C:\\Users\\Bob\\.config\\rclone\\rclone.conf"
2025/07/18 20:15:30 DEBUG : fs cache: renaming cache item "./" to be canonical "//?/D:/test"
da39a3ee5e6b4b0d3255bfef95601890afd80709  file00
da39a3ee5e6b4b0d3255bfef95601890afd80709  file01
da39a3ee5e6b4b0d3255bfef95601890afd80709  file02
da39a3ee5e6b4b0d3255bfef95601890afd80709  file03
da39a3ee5e6b4b0d3255bfef95601890afd80709  file04
da39a3ee5e6b4b0d3255bfef95601890afd80709  file05
da39a3ee5e6b4b0d3255bfef95601890afd80709  file06
da39a3ee5e6b4b0d3255bfef95601890afd80709  file07
da39a3ee5e6b4b0d3255bfef95601890afd80709  file08
da39a3ee5e6b4b0d3255bfef95601890afd80709  file09
f0c19035dab5178043ebc0d7d07b337df8d12a93  list_of_files
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file10
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file11
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file12
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file13
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file14
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file15
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file16
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file17
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file18
da39a3ee5e6b4b0d3255bfef95601890afd80709  dir1/file19
2025/07/18 20:15:30 DEBUG : 2 go routines active

aliron19 · August 5, 2025, 9:24pm

Hi there,

Can anyone help me to clarify this question, please?

I'd be glad to provide/test anything else if needed.

Thanks.

Animosity022 · August 6, 2025, 4:48pm

I think you'd want to put the files-from in something like alpha sorted and then you check-first / order by and do name.

Documentation

aliron19 · August 6, 2025, 7:09pm

Hi @Animosity022, and thanks for answering. According to the documentation you quoted, --check-first appears to be used only by sync, copy, and move commands, so in this case would have no effect for hashsum, which I can confirm after testing.

If I've understood you correctly, you proposed to use --check-first and --sort-by, but that would not follow the order of files as they appear on --files-from list_of_files which is the behavior I was expecting to.

So I wonder if there is a way to make rclone follow, exactly, the order of list_of_files as they appear. If it turns out to be the case, the idea is to make the hashing process restartable, because you could:

Count how many files have been processed, let's say X.
Trim list_of_files to remove the first X files, creating list_of_files_02
Launch the hashing process again, but using list_of_files_02 this time.

In that way, you could easily stop and restart the process at will, without having to hash a file twice, neither leaving anyone without hashing. I know you could make a program to achieve the same result, by creating the list_of_files_02 through coding, but that solution is far more complicated and I think it would be interesting to be able to do it as simple as possible.

Animosity022 · August 6, 2025, 7:45pm

You could then just loop through your list per command as it's checking each file anyway.

Otherwise, it does seem you'd want an enhancement to have hashsum respect order / check stuff.

aliron19 · August 6, 2025, 8:02pm

Yes, an enhancement seems to be the solution, otherwise calling rclone hash for each and every file doens't seem to be very efficient. I should open a feature request on github, right?

By the way, do you know why doens't rclone follow the order of files as they appear on list_of_files?

Animosity022 · August 6, 2025, 8:16pm

The short version is the concurrency in rclone as it's not always getting things in a specifc order.

system · September 5, 2025, 8:17pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.