Rclone is slow with local transfers

jared · December 13, 2022, 5:57pm

What is the problem you are having with rclone?

compared to windows explorer, rclone is very slow on local transfers. pinging @Ole. below there are four logs from four different scenarios:

high latency transfer with rclone (around 70 ms)
screenshot to same address with windows explorer
low latency transfer with rclone (below 1ms)
screenshot to same address with windows explorer

this seems to be baked into rclone and there is no tweaking i can do to make it function different, especially with a single file transfer; all can do is output the logs for you guys. the file i used as an example is a 3.8 GB video file. i did not let the long transfers run to conclusion, because even with this small file they take hours. they did remain stable for about 10 minutes, so i assume this tells you what you want to know. i can certainly do additional testing if needed.

Run the command 'rclone version' and share the full output of the command.

C:\rclone>rclone version
rclone v1.60.1
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.2006 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

local storage over SMB

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -vv --verbose "C:\Users\Jared\Desktop\You're My Best Friend.mp4" "\\10.0.250.1\test" --ignore-times

The rclone config contents with secrets removed.

no relevant remote in config, local path used

A log from the command with the `-vv` flag

Log 1. high latency transfer with rclone (around 70 ms) transfer was stable at average 3.7 mbps

rclone copy -vv --verbose "C:\Users\Jared\Desktop\You're My Best Friend.mp4" "\\10.0.250.1\test" --ignore-times
2022/12/13 11:22:53 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "-vv" "--verbose" "C:\\Users\\Jared\\Desktop\\You're My Best Friend.mp4" "\\\\10.0.250.1\\test" "--ignore-times"]
2022/12/13 11:22:53 DEBUG : Creating backend with remote "C:\\Users\\Jared\\Desktop\\You're My Best Friend.mp4"
2022/12/13 11:22:53 DEBUG : Using config file from "C:\\Users\\Jared\\AppData\\Roaming\\rclone\\rclone.conf"
2022/12/13 11:22:53 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\Jared\\Desktop\\You're My Best Friend.mp4", "//?/C:/Users/Jared/Desktop"
2022/12/13 11:22:53 DEBUG : Creating backend with remote "\\\\10.0.250.1\\test"
2022/12/13 11:22:53 DEBUG : fs cache: renaming cache item "\\\\10.0.250.1\\test" to be canonical "//?/UNC/10.0.250.1/test"
2022/12/13 11:22:53 DEBUG : You're My Best Friend.mp4: Transferring unconditionally as --ignore-times is in use
2022/12/13 11:23:53 INFO  :
Transferred:       24.371 MiB / 3.632 GiB, 1%, 422.222 KiB/s, ETA 2h29m20s
Transferred:            0 / 1, 0%
Elapsed time:       1m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  0% /3.632Gi, 422.219Ki/s, 2h29m20s

2022/12/13 11:24:53 INFO  :
Transferred:       48.559 MiB / 3.632 GiB, 1%, 403.764 KiB/s, ETA 2h35m8s
Transferred:            0 / 1, 0%
Elapsed time:       2m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  1% /3.632Gi, 403.766Ki/s, 2h35m8s

2022/12/13 11:25:53 INFO  :
Transferred:       72.840 MiB / 3.632 GiB, 2%, 407.803 KiB/s, ETA 2h32m35s
Transferred:            0 / 1, 0%
Elapsed time:       3m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  1% /3.632Gi, 407.801Ki/s, 2h32m35s

2022/12/13 11:26:53 INFO  :
Transferred:       97.184 MiB / 3.632 GiB, 3%, 414.080 KiB/s, ETA 2h29m16s
Transferred:            0 / 1, 0%
Elapsed time:       4m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  2% /3.632Gi, 414.078Ki/s, 2h29m16s

2022/12/13 11:27:53 INFO  :
Transferred:      121.621 MiB / 3.632 GiB, 3%, 417.440 KiB/s, ETA 2h27m4s
Transferred:            0 / 1, 0%
Elapsed time:       5m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  3% /3.632Gi, 417.439Ki/s, 2h27m4s

2022/12/13 11:28:53 INFO  :
Transferred:      145.559 MiB / 3.632 GiB, 4%, 406.636 KiB/s, ETA 2h29m58s
Transferred:            0 / 1, 0%
Elapsed time:       6m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  3% /3.632Gi, 407.061Ki/s, 2h29m49s

2022/12/13 11:29:53 INFO  :
Transferred:      169.684 MiB / 3.632 GiB, 5%, 408.146 KiB/s, ETA 2h28m25s
Transferred:            0 / 1, 0%
Elapsed time:       7m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  4% /3.632Gi, 408.143Ki/s, 2h28m25s

2022/12/13 11:30:53 INFO  :
Transferred:      193.559 MiB / 3.632 GiB, 5%, 407.992 KiB/s, ETA 2h27m28s
Transferred:            0 / 1, 0%
Elapsed time:       8m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  5% /3.632Gi, 407.993Ki/s, 2h27m28s

2022/12/13 11:31:53 INFO  :
Transferred:      217.277 MiB / 3.632 GiB, 6%, 404.921 KiB/s, ETA 2h27m35s
Transferred:            0 / 1, 0%
Elapsed time:       9m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  5% /3.632Gi, 404.921Ki/s, 2h27m35s

2022/12/13 11:32:53 INFO  :
Transferred:      241.402 MiB / 3.632 GiB, 6%, 413.456 KiB/s, ETA 2h23m33s
Transferred:            0 / 1, 0%
Elapsed time:      10m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  6% /3.632Gi, 413.811Ki/s, 2h23m25s

2022/12/13 11:33:53 INFO  :
Transferred:      265.309 MiB / 3.632 GiB, 7%, 407.632 KiB/s, ETA 2h24m36s
Transferred:            0 / 1, 0%
Elapsed time:      11m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  7% /3.632Gi, 407.632Ki/s, 2h24m36s

2022/12/13 11:34:53 INFO  :
Transferred:      289.277 MiB / 3.632 GiB, 8%, 411.598 KiB/s, ETA 2h22m12s
Transferred:            0 / 1, 0%
Elapsed time:      12m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  7% /3.632Gi, 411.599Ki/s, 2h22m12s

2022/12/13 11:35:53 INFO  :
Transferred:      313.434 MiB / 3.632 GiB, 8%, 413.984 KiB/s, ETA 2h20m23s
Transferred:            0 / 1, 0%
Elapsed time:      13m0.0s
Transferring:
 *                     You're My Best Friend.mp4:  8% /3.632Gi, 413.984Ki/s, 2h20m23s

Log 2. screenshot to same address with windows explorer. transfer was stable at average 28 mbps

Log 3. low latency transfer with rclone (below 1ms) transfer finished with average of 330 mbps

rclone copy -vv --verbose "C:\Users\Jared\Desktop\You're My Best Friend.mp4" J:\ --ignore-times
2022/12/13 11:49:53 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "-vv" "--verbose" "C:\\Users\\Jared\\Desktop\\You're My Best Friend.mp4" "J:\\" "--ignore-times"]
2022/12/13 11:49:53 DEBUG : Creating backend with remote "C:\\Users\\Jared\\Desktop\\You're My Best Friend.mp4"
2022/12/13 11:49:53 DEBUG : Using config file from "C:\\Users\\Jared\\AppData\\Roaming\\rclone\\rclone.conf"
2022/12/13 11:49:53 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\Jared\\Desktop\\You're My Best Friend.mp4", "//?/C:/Users/Jared/Desktop"
2022/12/13 11:49:53 DEBUG : Creating backend with remote "J:\\"
2022/12/13 11:49:53 DEBUG : fs cache: renaming cache item "J:\\" to be canonical "//?/J:/"
2022/12/13 11:49:53 DEBUG : You're My Best Friend.mp4: Need to transfer - File not found at Destination
2022/12/13 11:50:53 INFO  :
Transferred:        2.238 GiB / 3.632 GiB, 62%, 38.113 MiB/s, ETA 37s
Transferred:            0 / 1, 0%
Elapsed time:       1m0.0s
Transferring:
 *                     You're My Best Friend.mp4: 61% /3.632Gi, 38.113Mi/s, 37s

2022/12/13 11:51:31 DEBUG : You're My Best Friend.mp4: md5 = 5fa0664bcb99331e71ed220de489877d OK
2022/12/13 11:51:31 INFO  : You're My Best Friend.mp4: Copied (new)
2022/12/13 11:51:31 INFO  :
Transferred:        3.632 GiB / 3.632 GiB, 100%, 37.659 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m38.0s

2022/12/13 11:51:31 DEBUG : 3 go routines active

Log 4. screenshot to same address with windows explorer. transfer finished at a stable 1 gbps

Ole · December 13, 2022, 9:02pm

Hi jared,

Interesting, I just confirmed the observation by trying a local SSD to SSD single file copy - and yes rclone takes roughly twice the time and 3 times the CPU time.

I cannot explain why and don't think it can be easily changed/optimized.

I guess the primary reason is that the internals of rclone have been designed for optimal performance across a lot of very different cloud storages whereas the built-in copy in Windows has been specifically designed for optimal performance on NTFS and SMB only. You example is therefore a bit like comparing an offroad vehicle to a formula1 1 racer - on a fomula1 racing track.

How does rclone compare if you try using the new SMB backend?

PS: I like your choice of test file

ncw · December 14, 2022, 2:47pm

The primary reason is that rclone checks checksums after the copy!

If you use --ignore-checksum then rclone copy speeds should match those of explorer. They do here anyway on my Linux laptop with SSD.

Ole · December 14, 2022, 3:28pm

Argh, you are absolutely right, --ignore-checksum also makes the rclone speed similar on my Windows SSD.

jared · December 14, 2022, 9:02pm

ha! man i'm tired of winning with this tool now i'm wondering, how does windows confirm the integrity of a transfer if it doesn't do checksums? they just figure the TCP stack is good enough? or is there additional stuff they do with SMB?

i wonder if we talk of the same thing old country song? it's a local family i was making a video of. i didn't image danes listen to music

jared

Ole · December 14, 2022, 9:15pm

jared · December 14, 2022, 9:46pm

that's what i figured

that's actually what it was

Ole · December 15, 2022, 8:26am

Ah, OK I still like the text and see a lot of happy people having fun together - that still makes it a good choice

ncw · December 15, 2022, 1:07pm

It doesn't.

Very few OSes / tools care about file integrity, with the notable exception of ZFS.

asdffdsa · December 15, 2022, 1:49pm

fyi,

and REFS, a windows version of ZFS.

jared · December 16, 2022, 4:09am

nice to find common ground with a dane

wow. there's trillions of dollars of data out there sitting on windows server and it doesn't even have integrity measures

if you manually turn it on using cli.

system · December 19, 2022, 4:10am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.