Concurrency(inflight IO) control to copy big files from local_fs(ext4) to local_fs(mounted nfs)

Zhitao_Li · October 27, 2025, 4:27am

My problems:

I’m copying 16 files, each 2GB from local ext4 to the mounted NFSv3. I did observe the copy throughput at the beginning is 2GB/s, but quickly it reduces to 10MB/s. After debugging the NFSv3 NFS server, it shows that the inflight IO can reaches to 200+, much higher than the threshold 128 of NFS server. So the server returns retriable NFS error code NFS3ERR_JUKEBOX. NFS client will retry after some timeout, which will degrade the performance.

So I want to limit the inflight IO issued by rclone to avoid meeting with the threshold setting by the NFS server. Also I still want to copy files in parallel to achieve higher performance.

I tried different flags, but all of them still issued 128+ inflight IO write to the NFS server. The flags are as follows:

./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 -P
./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 --transfers 1 -P
./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 --transfers 1 -P --multi-thread-streams 1

Run the command 'rclone version' and share the full output of the command.

[root@xc2 rclone-v1.71.1-linux-amd64]# ./rclone version
rclone v1.71.1

os/version: rocky 8.9 (64 bit)
os/kernel: 4.18.0-513.5.1.el8_9.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.25.1
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

local ext4 to mounted path (NFSv3)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 -P
./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 --transfers 1 -P
./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 --transfers 1 -P --multi-thread-streams 1

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[root@xc2 rclone-v1.71.1-linux-amd64]# ./rclone  config redacted
2025/10/27 00:26:11 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
; empty config
### Double check the config for sensitive info before posting publicly
[root@xc2 rclone-v1.71.1-linux-amd64]#

A log from the command that you were trying to run with the `-vv` flag

[root@xc2 rclone-v1.71.1-linux-amd64]# ./rclone copy /mnt/pacs/zhitao/bigfiles /mnt/fio/rclone/bigfiles3 -P -vv
2025/10/26 23:55:27 DEBUG : rclone: Version "v1.71.1" starting with parameters ["./rclone" "copy" "/mnt/pacs/zhitao/bigfiles" "/mnt/fio/rclone/bigfiles3" "-P" "-vvvvvvvv"]
2025/10/26 23:55:27 DEBUG : Creating backend with remote "/mnt/pacs/zhitao/bigfiles"
2025/10/26 23:55:27 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2025/10/26 23:55:27 DEBUG : Creating backend with remote "/mnt/fio/rclone/bigfiles3"
2025/10/26 23:55:27 DEBUG : file_0.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_1.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_10.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_11.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_12.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_13.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_14.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_15.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_2.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_3.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_4.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_5.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_6.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_7.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_8.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : file_9.bin: Need to transfer - File not found at Destination
2025/10/26 23:55:27 DEBUG : Local file system at /mnt/fio/rclone/bigfiles3: Waiting for checks to finish
2025/10/26 23:55:27 DEBUG : Local file system at /mnt/fio/rclone/bigfiles3: Waiting for transfers to finish
2025/10/26 23:55:27 DEBUG : preAllocate: got error on fallocate, trying combination 1/2: operation not supported
2025/10/26 23:55:27 DEBUG : preAllocate: got error on fallocate, trying combination 2/2: operation not supported
2025/10/26 23:56:34 DEBUG : file_11.bin: md5 = 967aa7b8098c5d2d3eb85132aca853f4 OK
2025/10/26 23:56:34 DEBUG : file_11.bin.79f0f3c4.partial: renamed to: file_11.bin
2025/10/26 23:56:34 INFO  : file_11.bin: Copied (new)
2025/10/26 23:56:35 DEBUG : file_1.bin: md5 = ea83d33260d7996694257b0ab50a4ccf OK
2025/10/26 23:56:35 DEBUG : file_1.bin.ea2f37dd.partial: renamed to: file_1.bin
2025/10/26 23:56:35 INFO  : file_1.bin: Copied (new)
2025/10/26 23:56:35 DEBUG : file_10.bin: md5 = e3d47f9125825fffd8a91a5ec3e1e5dc OK
2025/10/26 23:56:35 DEBUG : file_0.bin: md5 = c45147a0b964530b521efb78c6f55268 OK
2025/10/26 23:56:35 DEBUG : file_10.bin.11d9005e.partial: renamed to: file_10.bin
2025/10/26 23:56:35 INFO  : file_10.bin: Copied (new)
2025/10/26 23:56:35 DEBUG : file_0.bin.a5c72616.partial: renamed to: file_0.bin
2025/10/26 23:56:35 INFO  : file_0.bin: Copied (new)
2025/10/26 23:57:41 DEBUG : file_15.bin: md5 = b78464e947c3f055c8b95657f0ec38ec OK
2025/10/26 23:57:41 DEBUG : file_13.bin: md5 = a9fcac588e8278269d67ab573b1ea81f OK
2025/10/26 23:57:41 DEBUG : file_14.bin: md5 = 9abf362ee16f792834e5d79e852a3499 OK
2025/10/26 23:57:41 DEBUG : file_15.bin.d3c847a9.partial: renamed to: file_15.bin
2025/10/26 23:57:41 INFO  : file_15.bin: Copied (new)
2025/10/26 23:57:41 DEBUG : file_14.bin.3d81f077.partial: renamed to: file_14.bin
2025/10/26 23:57:41 INFO  : file_14.bin: Copied (new)
2025/10/26 23:57:41 DEBUG : file_13.bin.1d01d5a0.partial: renamed to: file_13.bin
2025/10/26 23:57:41 INFO  : file_13.bin: Copied (new)
2025/10/26 23:57:41 DEBUG : file_12.bin: md5 = 9cb054f195c29b5bdf0bcbb2393b96c5 OK
2025/10/26 23:57:41 DEBUG : file_12.bin.acd746c9.partial: renamed to: file_12.bin
2025/10/26 23:57:41 INFO  : file_12.bin: Copied (new)
2025/10/26 23:58:47 DEBUG : file_4.bin: md5 = 2f226306b0806f8afa210cabe738aa51 OK
2025/10/26 23:58:47 DEBUG : file_4.bin.38a325d4.partial: renamed to: file_4.bin
2025/10/26 23:58:47 INFO  : file_4.bin: Copied (new)
2025/10/26 23:58:47 DEBUG : file_3.bin: md5 = 1e562a84d00faa9d82917c826e85dd9e OK
2025/10/26 23:58:47 DEBUG : file_3.bin.4a4ecb18.partial: renamed to: file_3.bin
2025/10/26 23:58:47 INFO  : file_3.bin: Copied (new)
2025/10/26 23:58:48 DEBUG : file_5.bin: md5 = 7547770fa6db4e29183b66e84f15edd6 OK
2025/10/26 23:58:48 DEBUG : file_2.bin: md5 = e8acd923e6b5dbc9ed3604c344499e25 OK
2025/10/26 23:58:48 DEBUG : file_5.bin.16e2cc26.partial: renamed to: file_5.bin
2025/10/26 23:58:48 INFO  : file_5.bin: Copied (new)
2025/10/26 23:58:48 DEBUG : file_2.bin.979efe25.partial: renamed to: file_2.bin
2025/10/26 23:58:48 INFO  : file_2.bin: Copied (new)
2025/10/27 00:01:50 DEBUG : file_7.bin: md5 = efb6cc506aa50ef0ad9e71a305cce40e OK
2025/10/27 00:01:50 DEBUG : file_7.bin.2c2f2529.partial: renamed to: file_7.bin
2025/10/27 00:01:50 INFO  : file_7.bin: Copied (new)
2025/10/27 00:01:50 DEBUG : file_8.bin: md5 = 3047a5302ff0c49f0048d4f12c346303 OK
2025/10/27 00:01:50 DEBUG : file_8.bin.1c05da40.partial: renamed to: file_8.bin
2025/10/27 00:01:50 INFO  : file_8.bin: Copied (new)
2025/10/27 00:01:51 DEBUG : file_9.bin: md5 = 2af6052c9ed710404c8866a970880947 OK
2025/10/27 00:01:51 DEBUG : file_9.bin.ff9c2679.partial: renamed to: file_9.bin
2025/10/27 00:01:51 INFO  : file_9.bin: Copied (new)
2025/10/27 00:01:51 DEBUG : file_6.bin: md5 = 4fe43ed519a11f21fcfc6662ae24c91b OK
2025/10/27 00:01:51 DEBUG : file_6.bin.4ce63881.partial: renamed to: file_6.bin
2025/10/27 00:01:51 INFO  : file_6.bin: Copied (new)
Transferred:   	       32 GiB / 32 GiB, 100%, 4.710 KiB/s, ETA 0s
Checks:                 0 / 0, -, Listed 16
Transferred:           16 / 16, 100%
Elapsed time:      6m23.8s
2025/10/27 00:01:51 INFO  :
Transferred:   	       32 GiB / 32 GiB, 100%, 4.710 KiB/s, ETA 0s
Checks:                 0 / 0, -, Listed 16
Transferred:           16 / 16, 100%
Elapsed time:      6m23.8s

kapitainsky · October 27, 2025, 6:08am

To control (and limit) transfers pace you could try:

--tpslimit
--tpslimit-burst
--bwlimit

EDIT: unfortunately -–tpslimit does not work for local fs.

Zhitao_Li · October 27, 2025, 11:35am

Thanks for your reply. –bwlimit does limit the transfer throughput. However, it doesn’t work in my practice. After some efforts, I understand the issue as follows.

With --transfers 1 , rclone is just writing the destination file with write() system call. But the kernel can issue the maximum sunrpc.tcp_max_slot_table_entries write() of nfsclient at the same time.
The default value of sunrpc.tcp_max_slot_table_entries is 65536, much higher than the threshold of NFSv3 server.

Adjust the kernel parameter does work in my practice.

ncw · October 28, 2025, 10:33am

Is that your preferred solution to this?

Zhitao_Li · October 29, 2025, 6:18am

Yes. The issue results from NFS client and is fixed by the parameters of NFS client.

system · November 1, 2025, 6:18am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.