Slow on local network transfer?

Help and Support
1

Hi,

my problem is, that rclone is rather slow (~10 MB/Sec) for a single, large file (7 GB) on a local network transfer (Windows -> Samba share) while I have no problem to get the maximum speed (~110 MB/Sec) when copying the same file by hand e.g. via Windows Explorer to the share.

rclone v1.52.2
- os/arch: windows/amd64
- go version: go1.14.4

Windows Server 2019, x64

"D:\Tools\@Command Line Tools\rclone_x64\rclone.exe" --config "D:\Tools\@Command Line Tools\rclone_x64\.rclone.conf" sync -P --retries 1 --ignore-case --filter-from "D:\Tools\@Command Line Tools\rclone_x64\filters_Backup.txt" "D:\Backup" nounc:"\\UK-NUC-01\backup_ssd\Backup"

I've tried copy instead of sync but that didn't change anything.

[nounc]
type = local

Any suggestions to get the maximum speed?

1 Like
2

hello,

when you posted, you were asked for some info, such as a debug log file.
what version of winfsp are you using?

3 
D:\Tools\@Command Line Tools\rclone_x64>"D:\Tools\@Command Line Tools\rclone_x64\rclone.exe" -vv --config "D:\Tools\@Command Line Tools\rclone_x64\.rclone.conf" sync -P --retries 1 --ignore-case --filter-from "D:\Tools\@Command Line Tools\rclone_x64\filters_Backup.txt" "D:\Backup" nounc:"\\UK-NUC-01\backup_ssd\Backup"
2020/07/25 16:15:37 DEBUG : rclone: Version "v1.52.2" starting with parameters ["D:\\Tools\\@Command Line Tools\\rclone_x64\\rclone_v1.52.2 [org].exe" "-vv" "--config" "D:\\Tools\\@Command Line Tools\\rclone_x64\\.rclone.conf" "sync" "-P" "--retries" "1" "--ignore-case" "--filter-from" "D:\\Tools\\@Command Line Tools\\rclone_x64\\filters_Backup.txt" "D:\\Backup" "nounc:\\\\UK-NUC-01\\backup_ssd\\Backup"]
2020/07/25 16:15:37 DEBUG : Using config file from "D:\\Tools\\@Command Line Tools\\rclone_x64\\.rclone.conf"
2020/07/25 16:15:37 DEBUG : fs cache: renaming cache item "D:\\Backup" to be canonical "//?/D:/Backup"
2020/07/25 16:15:37 DEBUG : fs cache: renaming cache item "nounc:\\\\UK-NUC-01\\backup_ssd\\Backup" to be canonical "nounc://?/UNC/UK-NUC-01/backup_ssd/Backup"
2020-07-25 16:15:37 DEBUG : Local file system at //?/UNC/UK-NUC-01/backup_ssd/Backup: Waiting for checks to finish
2020-07-25 16:15:37 DEBUG : @Grant permissions to everyone.bat: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020-07-25 16:15:37 DEBUG : @Grant permissions to everyone.bat: Unchanged skipping
2020-07-25 16:15:37 DEBUG : Local file system at //?/UNC/UK-NUC-01/backup_ssd/Backup: Waiting for transfers to finish
2020-07-25 16:28:14 DEBUG : Windows_Server_2019.tbi: MD5 = d4f14e82576f51a0c5fd6b7e280d85d4 OK
2020-07-25 16:28:14 INFO  : Windows_Server_2019.tbi: Copied (new)
2020-07-25 16:28:14 DEBUG : Waiting for deletions to finish
Transferred:        6.901G / 6.901 GBytes, 100%, 9.331 MBytes/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:     12m37.2s
2020/07/25 16:28:14 INFO  :
Transferred:        6.901G / 6.901 GBytes, 100%, 9.331 MBytes/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:     12m37.2s

2020/07/25 16:28:14 DEBUG : 2 go routines active

I do not use any winfsp version.

4

Interesting...

Can you try increasing (or decreasing) --buffer to see if that makes a difference?

Note that rclone will be doing checksumming as it goes along, you can try turning this off with --ignore-checksum

1 Like
5

Thanks, ncw.

Modifying the --buffer-size doesn't help much but --ignore-checksum yields some performance.

Measurements:

    // --ignore-checksum = 52 MB/s
    // --ignore-checksum + --buffer-size 64M  = 50 MB/s
    // --ignore-checksum + --buffer-size 128M = 53 MB/s
    // --ignore-checksum + --buffer-size 256M = 55 MB/s
    // --ignore-checksum + --buffer-size 512M = 50 MB/s

Far better but it's still only the half of it what the network can handle. Could I try anything more to improve the performance?

6

That is really interesting that --ignore-checksum helps so much. On my linux laptop it makes almost no difference (I get about 150 MB/s for local -> local copy).

What spec is your computer?

Can you try the latest beta - to see if that improves things?

7

My PC has a Ryzen 7 3700x (8 core, 16 via hyperthreading) with 32 GB of RAM.
The destination machine is an Intel NUC 5i3 with 8 GB RAM.

Source and destination are both SSDs (Samsung). Source = 1 TB, ~3000 MB+ read speed, the destination is a SATA (1 TB) version with a write speed of ~500 MB+.

The latest beta is neither slower nor faster than the current release version.

8

OK so both source and destination are reasonably powerful machines with fast disks...

If you try copying a file from local disk to local disk with rclone, what performance do you get there? Try with and without --ignore-checksum

9

Ok, source is still a 1 TB NVME SSD, destination (local) now a 500 GB NVME SSD.

Without --ignore-checksum: 449 MB/s (15.7s elapsed time)
With --ignore-checksum: 2.751 GB/s (2.5s elapsed time)

10

Ok that is nice and fast!

So this must be something to do with the the network or samba.

It might be worth running through a samba performance tuning guide like this one

https://wiki.amahi.org/index.php?title=Make_Samba_Go_Faster&mobileaction=toggle_view_mobile

11

Thanks for the link. I've already used a tuned config but not with all of these values...

Before:

      # Performance tuning
      use sendfile = yes
      getwd cache = yes
      write cache size = 524288
      min receivefile size = 16384
      socket options = TCP_NODELAY IPTOS_LOWDELAY

Now:

      # Performance tuning
      use sendfile = yes
      getwd cache = yes
      write cache size = 524288
      min receivefile size = 16384
      aio read size = 16384
      aio write size = 16384
      # aio write behind = true
      socket options = SO_RCVBUF=131072 SO_SNDBUF=131072 IPTOS_LOWDELAY TCP_NODELAY

Don't want to activate aio write behind...

But this leads to 76 MB/s instead of 52 MB/s before and I think I can live with this now.
Thanks again!

12

That is great! I think rclone could be doing IO more efficiently on Windows, but I'm not sure exactly how to improve it. On linux/macOS the IO works really quickly.

13

does --ignore-checksum helps with rclone serve http?

14

I don't think so unless you use the etag as checksum flag (forgotten exactly what that is)

15

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.