Http download speeds low with pfsense or usg router with dns entry set

So I have different download speeds from my google drive via mount (30mb/s) + copy and http serve (same as caddy as server) with 1mb/s. Playing video files over samba mounted rclone mount

rclone mount:
RCLONE_CONFIG=/opt/rclone/rclone.conf /usr/bin/rclone mount cgdrive: /gdrive --allow-other --dir-cache-time 96h --drive-chunk-size 64M --vfs-read-chunk-size-limit 300M --vfs-cache-mode writes --buffer-size 150M

caddy config:
192.168.10.1:8000
root /gdrive
browse

download test (curl):

 curl http://192.168.10.1:8000/movies/test.mkv > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1 34.4G    1  448M    0     0  2037k      0  4:55:29  0:03:45  4:51:44  949k

copy test:

dd if=/test.mkv of=/dev/null status=progress
961397248 bytes (961 MB, 917 MiB) copied, 33.4164 s, 28.8 MB/s

I've broken it down to my pfsense box, by replacing it with my old router, which works perfectly fine.

pfsense (latest version) runs on my proxmox node with dedicated lan and wan (virtio). I start from stock, assign lan and wan to my adapters and disable hardware checksum offloading.

At this point the problem already occurs and I have no idea what to do.

Things I tried without success: Disable Firewall, disable ipv6, traffic shaping

I've tried opnsense as well with the same results.

Sure sounds like traffic shaping. But I know you said you've disabled it. The fact that some traffic isn't slow is reasonably odd.

I've used OPNSense for 10-12 months and pfSense before that. If you have traffic shaping off, which I use actually for OPN, there really isn't much config to do as it really just works for things like this. I would think you have something else going on.

I just setup a caddy config identical to yours and tested a copy:

[felix@gemini ~]$ curl http://192.168.1.30:6767/Movies/10%20Cloverfield%20Lane%20%282016%29.mp4 >  /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 84 7694M   84 6491M    0     0  76.8M      0  0:01:40  0:01:24  0:00:16 84.6M

I'm maxing out my gigabit for the most part.

On the pfSense side, are you seeing any thing jumping out from cpu/memory/io?

I'd check for high cpu/interrupts.

Ubiquity Box:

# curl http://192.168.10.1:8000/movies/test.mkv > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 17 34.4G   17 6230M    0     0  58.6M      0  0:10:01  0:01:46  0:08:15 78.2M^C

pfSense:

# curl http://192.168.10.1:8000/movies/test.mkv > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2 34.4G    2  707M    0     0  1958k      0  5:07:28  0:06:10  5:01:18 1713k

Speedtests are normal, I usually get Gigabit

My interrupt rate with 2-3%, peaking at 10 is higher. I also reduced core count to 2 as many others do. Didn't change anything.

last pid: 55600;  load averages:  0.17,  0.16,  0.16                            up 0+00:36:54  06:42:56
49 processes:  1 running, 48 sleeping
CPU:  1.2% user,  0.0% nice,  4.1% system,  8.0% interrupt, 86.7% idle
Mem: 63M Active, 51M Inact, 143M Wired, 16M Buf, 7657M Free
Swap: 1638M Total, 1638M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
  337 root          1  52    0 88516K 32500K accept  1   0:03   0.75% php-fpm
  338 root          1  20    0 88644K 32424K accept  0   0:03   0.41% php-fpm
34194 root          1  20    0 23596K  8836K kqread  1   0:00   0.08% nginx
82472 root          1  20    0  7816K  3696K CPU0    0   0:00   0.07% top
41554 root          1  20    0  6268K  2208K select  1   0:00   0.03% radvd
46724 root          5  52    0  6904K  2444K uwait   0   0:00   0.03% dpinger
63268 dhcpd         1  20    0 12584K  7480K select  1   0:00   0.02% dhcpd
89349 root          1  20    0 12908K  8132K select  1   0:00   0.01% sshd
73563 root          1  20    0  6404K  2548K select  1   0:00   0.01% syslogd
  336 root          1  20    0 88384K 22712K kqread  0   0:00   0.01% php-fpm
35222 root          1  20    0 12400K 12504K select  0   0:05   0.01% ntpd
11879 root          1  20    0  6604K  2356K bpf     1   0:00   0.00% filterlog
 8706 _dhcp         1  20    0  6452K  2412K select  1   0:06   0.00% dhclient
18907 root          1  52    0 88516K 30184K accept  1   0:01   0.00% php-fpm
58047 unbound       2  20    0 34076K 19104K kqread  1   0:01   0.00% unbound
71030 root          1  52   20  6968K  2856K wait    1   0:00   0.00% sh
 6389 root          1  52    0  6452K  2292K select  1   0:00   0.00% dhclient
30071 root          1  20    0  7280K  3796K pause   1   0:00   0.00% tcsh
16123 root          1  20    0 11912K  5228K piperd  1   0:00   0.00% sshg-parser
16174 root          2  20    0  6532K  2476K piperd  1   0:00   0.00% sshg-blocker
15661 root          1  36    0  6968K  2576K wait    1   0:00   0.00% sh
42230 root          1  21    0  6724K  2840K wait    1   0:00   0.00% login
  375 root          1  40   20  9244K  4440K kqread  1   0:00   0.00% check_reload_status
34424 root          1  20    0  6372K  2356K nanslp  1   0:00   0.00% cron
13988 root          1  52    0  6968K  2824K wait    0   0:00   0.00% sh
42750 root          1  52    0  6968K  2824K ttyin   1   0:00   0.00% sh
42544 root          1  21    0  6968K  2944K wait    0   0:00   0.00% sh
15783 root          1  20    0  6196K  2004K piperd  1   0:00   0.00% cat
15619 root          1  52    0  6312K  2148K ttyin   1   0:00   0.00% getty
14755 root          1  52    0  6312K  2148K ttyin   0   0:00   0.00% getty
14533 root          1  52    0  6312K  2148K ttyin   1   0:00   0.00% getty
16584 root          1  52    0  6968K  2564K piperd  1   0:00   0.00% sh
  414 root          1  20    0  9188K  4976K select  0   0:00   0.00% devd
15048 root          1  52    0  6312K  2148K ttyin   1   0:00   0.00% getty
15051 root          1  52    0  6312K  2148K ttyin   0   0:00   0.00% getty
15391 root          1  52    0  6312K  2148K ttyin   1   0:00   0.00% getty
28184 root          1  20    0 12616K  7572K select  1   0:00   0.00% sshd
14992 root          1  52    0  6312K  2148K ttyin   1   0:00   0.00% getty
33868 root          1  20    0 21548K  7468K kqread  0   0:00   0.00% nginx
33840 root          1  52    0 21548K  6992K pause   0   0:00   0.00% nginx


[2.4.4-RELEASE][admin@pfSense.localdomain]/root: vmstat -i
interrupt                          total       rate
irq1: atkbd0                          68          0
irq11: uhci0+                         18          0
irq15: ata1                          710          0
cpu0:timer                         39234         22
cpu1:timer                         25627         15
irq259: virtio_pci1                 5970          3
irq261: virtio_pci2               535376        307
irq262: virtio_pci2                    4          0
irq264: virtio_pci3               259033        148
irq265: virtio_pci3                    8          0
Total                             866048        496

Basically anuthing I found during reasearching is that most problems are caused by additional packages (which I don't have installed) or a vague guess that you should run pfSense bare metal.

Yeah, that's super strange as I haven't been using pfSense for a bit, but it's very similar to OPNSense.

If you are running straight out of the box, that has traffic shaping off and really shouldn't interfere with rclone as that's just a normal HTTPS connection out which would be the same as a speedtest for the most part.

You said the speedtest through returns gigabit so for me that rules out interfaces not connecting at the right speeds and quite a lot of things.

I'd probably check dmesg and look for anything in there. I'd make a rule for that device and log all the traffic out to see if anything is going on there. I'd turn on logging for all the other rules to make sure something else isn't blocking something.

Your interrupt times aren't awful. I had a i3 when I was shaping before and it would make out a core and be unable to handle the traffic using shaping which is why I asked to check there to see.

2.4.4 isn't super old but you could also try grabbing the latest, which is p3 released a few days back.

I was on pfsense 2.4.4-p3. I didn't manage to solve the issue. But I've noticed that I have the exact same speed dropoffs, when I set any DNS Server in my router's config. So as soon as I use a DNS other than my provider rclone runs at 1Mbit. Everything else works fine. I don't know what to test anymore.

DNS is always a funny thing as it depends on what your provider returns back.

I use CloudFlare's DNS for myself. Here is what is returns for 1.1.1.1 and 8.8.8.8

[felix@gemini ~]$ host www.googleapis.com
www.googleapis.com is an alias for googleapis.l.google.com.
googleapis.l.google.com has address 172.217.7.138
googleapis.l.google.com has address 172.217.8.10
googleapis.l.google.com has address 172.217.15.106
googleapis.l.google.com has address 172.217.164.138
googleapis.l.google.com has address 172.217.164.170
googleapis.l.google.com has address 172.217.5.234
googleapis.l.google.com has IPv6 address 2607:f8b0:4004:811::200a
[felix@gemini ~]$ host www.googleapis.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

www.googleapis.com is an alias for googleapis.l.google.com.
googleapis.l.google.com has address 172.217.11.42
googleapis.l.google.com has address 172.217.12.170
googleapis.l.google.com has address 172.217.12.202
googleapis.l.google.com has address 172.217.9.234
googleapis.l.google.com has address 172.217.10.42
googleapis.l.google.com has address 172.217.10.106
googleapis.l.google.com has address 172.217.10.138
googleapis.l.google.com has address 172.217.3.106
googleapis.l.google.com has address 172.217.12.138
googleapis.l.google.com has address 172.217.6.202
googleapis.l.google.com has address 172.217.6.234
googleapis.l.google.com has address 172.217.7.10
googleapis.l.google.com has IPv6 address 2607:f8b0:4006:815::200a

It all depends own which you get connected to and usually it gives you a location that is closest to you for best peering.

If you are having a pure pfSense issue, I'd post on the netgate forums and get someone to troubleshoot through that as that seems to be your case.

I don't know anymore whats causing what. Only thing I know after trying multiple routers. After I change my DNS (even to google), rclone downloads at 1mbit, where everything else works as expected with 1gbit.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.