Small file performance vs robocopy

Help and Support
1

First let me say, I love RCLONE, it's really a game changer in so many ways. We have almost completely replaced 95% of our copy tools with this.

What is the problem you are having with rclone?

One thing it does struggle with is copying small files, let's say anything less than 100K.
We do a lot of local/wan copies and consistently rclone is extremely slow on small files compared to robocopy.

We have tried different combinations of transfers and checkers, from default values to high
500 transfers 5 checkers
50 transfers 20 checkers
10 transfers 5 checkers

We didn't see many gains in speeds only losses when we increased our transfers (for obviously reasons)
We also tried --size-only --fast-list --buffer-size but none of these really helped.

But with robocopy source destination /E /w:1 /r:1 /np /tee /max:1000000 /mt:64 /XO
We exponentially increased our output, what would have taken hours took minutes

What happens with rclone when copying small files is the file copy with start, it will copy files at a very good speed but once it done copying it will sit there at 0KB/s and wait, it's like it's trying to close the connection. The after 20-40 secs and it's finally done and moves on to the next file.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

  • os/version: Microsoft Windows Server 2022 Datacenter 21H2 (64 bit)
  • os/kernel: 10.0.20348.3932 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: static
  • go/tags: cmount

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
--> No, will need to try it.

Which cloud storage system are you using? (eg Google Drive)

Local/WAN CIFS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy Source Destination --transfers 40 --checkers 8 --size-only -P -u -M --fast-list --ignore-case --buffer-size 64M

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

This is not applicable as we are using CIFS local

A log from the command that you were trying to run with the -vv flag

Paste  log here
1 Like
2

welcome to the forum,

should rclone selfupdate and test again.

afiak, the problem with robocopy is that it does not verify file transfers.
where rclone does verify file transfers, and that might take additional time.

fwiw, non-cloud transfers on windows, i never use robocopy or rclone, except in rare cases.
on windows, i use fastcopy, which does verify file transfers.

and check out my howto guide
https://forum.rclone.org/t/how-to-access-smb-samba-with-rclone/42754

does nothing on local/smb, can remove that.

2 Likes
3

Thank you for the update!
I will run the latest version and post an update, I'll try to run clone with no verification and see what the speed is like.
I'll will definitely try this fast copy.

4

i have been using for 10+ years. also has a command line interface which i use with my backup scripts.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.