Rclone serve http + local disk slow speeds

[random404]

I'm starting to play with rclone serve http + local disk to feed my servers instead of google drive.

However on me testings it seems the max I can get in a single transfer is 107 MBytes/s but I should be able to get at least 120 MBytes/s

Rclone serve is in a 10 Gbps server, and I'm doing a rclone copy in a 1 Gbps server on the same local network with a 15 GB file

rclone serve http disk:/backup -vvv --stats 2s --addr :80 --no-modtime  --user teste --pass teste

Result: 108 MBytes/s 2m11s

------------------------------------------

rclone serve http disk:/backup -vvv --stats 2s --addr :80 --no-modtime  --user teste --pass teste --max-header-bytes 8192

Result: 105 MBytes/s 2m15s

--------------------------------------------------

rclone serve http disk:/backup -vvv --stats 2s --addr :80 --no-modtime  --user teste --pass teste --max-header-bytes 81920

Results: 106 MBytes/s 2m13s

--------------------------------------------------

rclone serve http disk:/backup -vvv --stats 2s --addr :80 --no-modtime  --user teste --pass teste --vfs-read-chunk-size 10M --vfs-read-chunk-size-limit 10M

Results: 105 MBytes/s 2m15s

--------------------------------------------------

rclone serve http disk:/backup -vvv --stats 2s --addr :80 --no-modtime  --user teste --pass teste --vfs-read-chunk-size 20M --vfs-read-chunk-size-limit 20M

Results: 105 MBytes/s 2m15s

--------------------------------------------------

rclone serve http disk:/backup -vvv --stats 2s --addr :80 --no-modtime  --user teste --pass teste --vfs-read-chunk-size 10M --vfs-read-chunk-size-limit 10M --max-header-bytes 8192

Results: 107 MBytes/s 2m12s

Is there anything else I can do to improve speeds? Comparing with rclone mount + cp it usually takes double the time than rclone copy

[ncw (Nick Craig-Wood)]

@random404

rclone serve http disk:/backup -vvv

When I tried this locally I got

Transferred: 1.000G / 1.000 GBytes, 100%, 1.403 GBytes/s, ETA 0s

If I try it across a 1Gbit/s network I get

Transferred: 1.000G / 1.000 GBytes, 100%, 111.389 MBytes/s, ETA 0s

Which I guess is probably at the limit of what a 1Gbit/s network can do - this is 930 Mbit/s and that doesn't include any of the overheads...

This article states that the maximum throuhput using TCP over 1Gbit/s network is 118660598 bytes per second which is 113 MiByte/s (this is what rclone measures) so rclone is getting pretty close to the theoretical max transfer rate!

I tried the transfers from my laptop to an embedded computer with very low powered CPU.

[random404]

Nice links! And that makes sense! Thanks.

Rclone is so amazing and wonderful, I can use it for so many things and so many use cases.

With this great performance I think I'll not bother looking up NFS. I have been slowly replacing my servers away from google drive and moving to my http backend and nobody is noticing yet !

[asdffdsa (jojothehumanmonkey)]

yes, rclone is amazing

https://rclone.org/donate/
i do not profit financially from donations

[random404]

I'm already a github sponsor