curl http://192.168.10.1:8000/movies/test.mkv > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
1 34.4G 1 448M 0 0 2037k 0 4:55:29 0:03:45 4:51:44 949k
I've broken it down to my pfsense box, by replacing it with my old router, which works perfectly fine.
pfsense (latest version) runs on my proxmox node with dedicated lan and wan (virtio). I start from stock, assign lan and wan to my adapters and disable hardware checksum offloading.
At this point the problem already occurs and I have no idea what to do.
Things I tried without success: Disable Firewall, disable ipv6, traffic shaping
I've tried opnsense as well with the same results.
I've used OPNSense for 10-12 months and pfSense before that. If you have traffic shaping off, which I use actually for OPN, there really isn't much config to do as it really just works for things like this. I would think you have something else going on.
I just setup a caddy config identical to yours and tested a copy:
[felix@gemini ~]$ curl http://192.168.1.30:6767/Movies/10%20Cloverfield%20Lane%20%282016%29.mp4 > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
84 7694M 84 6491M 0 0 76.8M 0 0:01:40 0:01:24 0:00:16 84.6M
I'm maxing out my gigabit for the most part.
On the pfSense side, are you seeing any thing jumping out from cpu/memory/io?
# curl http://192.168.10.1:8000/movies/test.mkv > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
17 34.4G 17 6230M 0 0 58.6M 0 0:10:01 0:01:46 0:08:15 78.2M^C
pfSense:
# curl http://192.168.10.1:8000/movies/test.mkv > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 34.4G 2 707M 0 0 1958k 0 5:07:28 0:06:10 5:01:18 1713k
Speedtests are normal, I usually get Gigabit
My interrupt rate with 2-3%, peaking at 10 is higher. I also reduced core count to 2 as many others do. Didn't change anything.
Basically anuthing I found during reasearching is that most problems are caused by additional packages (which I don't have installed) or a vague guess that you should run pfSense bare metal.
Yeah, that's super strange as I haven't been using pfSense for a bit, but it's very similar to OPNSense.
If you are running straight out of the box, that has traffic shaping off and really shouldn't interfere with rclone as that's just a normal HTTPS connection out which would be the same as a speedtest for the most part.
You said the speedtest through returns gigabit so for me that rules out interfaces not connecting at the right speeds and quite a lot of things.
I'd probably check dmesg and look for anything in there. I'd make a rule for that device and log all the traffic out to see if anything is going on there. I'd turn on logging for all the other rules to make sure something else isn't blocking something.
Your interrupt times aren't awful. I had a i3 when I was shaping before and it would make out a core and be unable to handle the traffic using shaping which is why I asked to check there to see.
2.4.4 isn't super old but you could also try grabbing the latest, which is p3 released a few days back.
I was on pfsense 2.4.4-p3. I didn't manage to solve the issue. But I've noticed that I have the exact same speed dropoffs, when I set any DNS Server in my router's config. So as soon as I use a DNS other than my provider rclone runs at 1Mbit. Everything else works fine. I don't know what to test anymore.
DNS is always a funny thing as it depends on what your provider returns back.
I use CloudFlare's DNS for myself. Here is what is returns for 1.1.1.1 and 8.8.8.8
[felix@gemini ~]$ host www.googleapis.com
www.googleapis.com is an alias for googleapis.l.google.com.
googleapis.l.google.com has address 172.217.7.138
googleapis.l.google.com has address 172.217.8.10
googleapis.l.google.com has address 172.217.15.106
googleapis.l.google.com has address 172.217.164.138
googleapis.l.google.com has address 172.217.164.170
googleapis.l.google.com has address 172.217.5.234
googleapis.l.google.com has IPv6 address 2607:f8b0:4004:811::200a
[felix@gemini ~]$ host www.googleapis.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
www.googleapis.com is an alias for googleapis.l.google.com.
googleapis.l.google.com has address 172.217.11.42
googleapis.l.google.com has address 172.217.12.170
googleapis.l.google.com has address 172.217.12.202
googleapis.l.google.com has address 172.217.9.234
googleapis.l.google.com has address 172.217.10.42
googleapis.l.google.com has address 172.217.10.106
googleapis.l.google.com has address 172.217.10.138
googleapis.l.google.com has address 172.217.3.106
googleapis.l.google.com has address 172.217.12.138
googleapis.l.google.com has address 172.217.6.202
googleapis.l.google.com has address 172.217.6.234
googleapis.l.google.com has address 172.217.7.10
googleapis.l.google.com has IPv6 address 2607:f8b0:4006:815::200a
It all depends own which you get connected to and usually it gives you a location that is closest to you for best peering.
If you are having a pure pfSense issue, I'd post on the netgate forums and get someone to troubleshoot through that as that seems to be your case.
I don't know anymore whats causing what. Only thing I know after trying multiple routers. After I change my DNS (even to google), rclone downloads at 1mbit, where everything else works as expected with 1gbit.