Rclone checksumming in multi-thread

Help and Support
1

What is the problem you are having with rclone?

very long time to get checksums of copied files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.73.1

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-216-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.25.7
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

$ dpkg -l nfs-kernel-server
ii  nfs-kernel-server 1:1.3.4-2.5ubuntu3.7 amd64        support for NFS kernel server

$ mount -t nfs4

m-db-01:/mnt/disk on /mnt/prw1disk type nfs4 (rw,nosuid,nodev,noexec,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none, user)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ rclone --progress copy /mnt/mntDISKsnapshot_202603030400/mariadb prw1disk:mariadb --create-empty-src-dirs --multi-thread-chunk-size=256M --multi-thread-streams=8 --buffer-size=256M --size-only --order-by size,desc

The rclone config contents with secrets removed.

$ sudo rclone config redacted
[DEFAULT]

checkers = 16
transfers = 8
[prw1disk]
type = alias
remote = /mnt/prw1disk
hash_type = none

A log from the command with the -vv flag

too long to paste here

My problem is:

  1. When I’m copy with multi-threading enabled, process is going very fast.
  2. But when rclone ends with copy and tries to verify contents of the written files (default) with md5quick or even hash_type=none, it literally downloads back from the NFS all my uploaded files and do it in single thread, which takes much more time than upload.
  3. Now I know the --ignore-checksum commandline option, but I think that rclone should do the checksumming in parallel instead of disabling checksumming at all.
2

hi,

for remote prw1disk, not sure hash_type is valid, not sure it does anything at all?

not sure what that is or if it is a valid remote?

3

yes, it is literally do nothing (i’ve tried to switch off checksumming after copy)

with the [DEFAULT] section rclone starts without any complains and these parameters I’m additionally configure in command line.

Probably, I’ll remove it at all

4

you posted this topic category as bug but so far, not seeing that??

5

in my case, rclone uploads file in multi-threading while do the checksumming in single-thread.
Looks like a bug, as for me

6

afiak, that is how rclone is designed to work, so that is not a bug.

https://forum.rclone.org/t/is-the-checksum-command-synchronous-instead-of-parallel/38626

7

Generally those checksums can only be done sequentially as that’s how they were designed. Not a rclone limitation, but how the checksums work.

8

Ok, thanks for the clarification