Chunker does not merge chunks on server FS

Help and Support
1

I’m using rclone to maximise throughput during DB (~10000 files up to 10Tb) copy on NFS

rclone v1.71.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-216-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.25.3
- go/linking: static
- go/tags: none

I’m using NFS-kernel-server on ubuntu22 mounted with:
rt1:/mnt/disk on /mnt/rt1disk type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none,user)

/root/.config/rclone/rclone.conf

checkers=16
transfers=8

[rt1disk]
type = chunker
remote = /mnt/rt1disk
chunk_size = 250M
hash_type = md5quick

this config previously allows me to effectively use all bandwidth during copy:

rclone --metadata copy /mnt/disk/backup/mytestdb rt1disk:mytestdb-202510

, splitting large files to small chunks and transferring it in parallel.

Chunks were combined automatically back to original files and everything was fine

rclone config redacted
[DEFAULT]
# couldn't find type of fs for "DEFAULT"
checkers = 16
transfers = 8

[rt1disk]
type = chunker
remote = /mnt/rt1disk
chunk_size = 512M
hash_type = md5quick

[rt2disk]
type = chunker
remote = /mnt/rt2disk
hash_type = md5quick

[rt3disk]
type = chunker
remote = /mnt/rt3disk
hash_type = md5quick
### Double check the config for sensitive info before posting publicly

rclone -vv copy /mnt/disk/mariadb/gcache/galera.cache rt1disk:mariadb2

Above command leaves chunks in place without merge…

A log from the command that you were trying to run with the -vv flag

2025/11/07 12:17:23 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "-vv" "copy" "/mnt/disk/mariadb/gcache/galera.cache" "rt1disk:mariadb2"]
2025/11/07 12:17:23 DEBUG : Creating backend with remote "/mnt/disk/mariadb/gcache/galera.cache"
2025/11/07 12:17:23 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2025/11/07 12:17:23 DEBUG : fs cache: renaming child cache item "/mnt/disk/mariadb/gcache/galera.cache" to be canonical for parent "/mnt/disk/mariadb/gcache"
2025/11/07 12:17:23 DEBUG : Creating backend with remote "rt1disk:mariadb2"
2025/11/07 12:17:23 DEBUG : Creating backend with remote "/mnt/rt1disk/mariadb2"
2025/11/07 12:17:23 DEBUG : galera.cache: Need to transfer - File not found at Destination
2025/11/07 12:17:23 DEBUG : galera.cache: skip slow MD5 on source file, hashing in-transit
2025/11/07 12:17:23 DEBUG : preAllocate: got error on fallocate, trying combination 1/2: operation not supported
2025/11/07 12:17:25 INFO  : galera.cache.rclone_chunk.001_cssy3y: Moved (server-side) to: galera.cache.rclone_chunk.001
2025/11/07 12:17:25 INFO  : galera.cache.rclone_chunk.002_cssy3y: Moved (server-side) to: galera.cache.rclone_chunk.002
2025/11/07 12:17:25 INFO  : galera.cache: Copied (new)
2025/11/07 12:17:25 INFO  :
Transferred:      512.001 MiB / 512.001 MiB, 100%, 498.909 MiB/s, ETA 0s
Checks:                 2 / 2, 100%, Listed 0
Renamed:                2
Transferred:            1 / 1, 100%
Server Side Moves:      2 @ 512.001 MiB
Elapsed time:         1.5s

2025/11/07 12:17:25 DEBUG : 3 go routines active

while
``rclone ls rt1disk:mariadb2
536872216 galera.cache```

shows that everything is ok, real directory content looks different:
`` ls -al /mnt/rt1disk/mariadb2
total 524304
drwxr-xr-x 2 root root 4096 Nov 7 12:17 .
drwxr-xr-x 7 root root 4096 Nov 7 12:17 ..
-rw-r--r-- 1 root root 79 Nov 7 10:12 galera.cache
-rw-r--r-- 1 root root 536870912 Nov 7 10:12 galera.cache.rclone_chunk.001
-rw-r--r-- 1 root root 1304 Nov 7 10:12 galera.cache.rclone_chunk.002`

I’ve realize, that rclone chunker seems to be working right, but how do I split large files in pieces to transfer in parallel ?

Thanks in advance

2

yes, it seems to working right.

add those flags to the command
rclone copy /mnt/disk/mariadb/gcache/galera.cache rt1disk:mariadb2 -vv --checkers=16 transfers=8

why do you want to split the files instead of just copy them whole?

3

why do you want to split the files instead of just copy them whole?

Please take a look: my database consists of say, 500 files up to 1Gb size and one 1000Gb size. While all files transfers in parallel, everything is Ok - until single large file remaining: it uploads in 1 thread with regular speed, without parallel multiplication, which is annoying.

I was sure that chunker does what I want: it splits all files to small blocks, loading all transfer slots, increasing speed drastically. And it used to merge splitted files back couple of months ago.

Probably, I miss some option/switch? Changed software version?

4

transfer the large file first.
--check-first --order-by size,desc

5

Yes, starting from the largest file helps me a little, thank you

6

As far as I see, changing direction makes what I looking for:
sharing local folder via NFS, describing this on target server share under chunker and making copy to local disk on target server via SSH

$ sudo rclone config redacted
[DEFAULT]
checkers = 16
transfers = 8
[nfs24disk]
type = chunker
remote = /mnt/nfs24disk
chunk_size = 250M
hash_type = md5quick

sudo rclone --progress copy nfs24disk:backup/data-20251110.0001-full mariadb2 --check-first --order-by=size,desc -vv

2025/11/10 12:37:36 DEBUG : data/text.ibd: multi-thread copy: chunk 11607/11610 (778865475584-778932584448) size 64Mi finished
2025/11/10 12:37:36 DEBUG : data/text.ibd: multi-thread copy: chunk 11608/11610 (778932584448-778999693312) size 64Mi finished
2025/11/10 12:37:36 DEBUG : data/text.ibd: multi-thread copy: chunk 11609/11610 (778999693312-779066802176) size 64Mi finished
2025/11/10 12:37:36 DEBUG : data/text.ibd: multi-thread copy: chunk 11610/11610 (779066802176-779100356608) size 32Mi finished
2025/11/10 12:37:36 DEBUG : data/text.ibd: Finished multi-thread copy with 11610 parts of size 64Mi
2025/11/10 12:37:36 DEBUG : data/text.ibd.775c4bee.partial: renamed to: data/text.ibd
2025/11/10 12:37:36 INFO : data/text.ibd: Multi-thread Copied (new)

Can’t understand: why the chunk size is 65Mi and not the 250M

7

As I can see, I just need to use these options:

  • --multi-thread-chunk-size SizeSuffix          Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi)       

  • --multi-thread-cutoff SizeSuffix              Use multi-thread downloads for files above this size (default 256Mi)       

  • --multi-thread-streams int                    Number of streams to use for multi-thread downloads (default 4)       

  • --multi-thread-write-buffer-size SizeSuffix   In memory buffer size for writing when in multi-thread mode (default 128Ki)
8

Ok, now I can understand, how to use chunker - to save files solid, it should be cascaded:

[rt1disk]
type=alias
remote=/mnt/rt1disk

[rt1chunk]
type=chunker
remote=rt1disk
chunk_size=250M

// And I don’t know why remote type=local is not working, while remote type=alias works like a charm.

9

You are conflating 2 concepts:

From this thread, it seems like you need only MultithreadUpload (which is supported for Local remotes), not a Chunker remote. The Chunker remote never merges files on writes, it only does it on reads. The MultithreadUpload feature, however, does upload the files in chunks and also merges them (to be shown as a single file) after the write is complete.

1 Like