pfsense. I think I just narrowed it down to rule 1 of IT: it’s always dns.
I had cloudflare’s DNSSEC enabled on pfsense and I just disabled it and went back to google’s DNS servers. The issue seems to be resolved. I’m thinking what happened is rclone had to resolve DNS every time it grabbed a new chunk of data, and the overhead of DNS over TLS was causing massive slowdowns. FML.
Edit: it looks like this wasn’t it. After a few minutes of glory, everything’s back to 4.5MBps
I’ve never set the shaping up before. I also use Unbound DNS resolver. I’m thinking it could be my netgear GS724TPv2 switch, but I never touched the QoS settings.
So couple things I would validate, but doesn’t sound like it would be the case. If you do check the Traffic Shaper, you don’t see anything there other than the blank section where the queues would be. You also haven’t setup any limiters but those generally just caps so you wouldn’t get a burst than a drop.
If you login to your pfsense router, run a top while the transfer is going on and make sure you aren’t seeing any high interrupts which might be slowing you down.
Mine jump up a bit when pulling down 500 Mbs, but nothing more than one core out of my 4 can handle. I traffic shape a gigabit link.
If you don’t see anything obvious, I’d go along with your simple route and limit the pieces of gear involved. If you can test just PFsense router with a machine plugged into it, start there and add pieces to your chain till it breaks.
DNS really shouldn’t be an issue as you’d see those in your logs and it only asks for the DNS on the initial connection out. You’d see a clear message saying it can’t lookup a DNS name.
I just ran continuous iperf tests along with speedtest.net and I’m not seeing and traffic shaping. I think google might just be pulling our legs and for whatever reason whenever a transfer of a file first starts, they give me burst speeds.
I think I have a better idea of what’s going on here @Vamp@Animosity022@ncw. This may very well be a peering issue of sorts with google’s newer v3 API endpoints. I just connected to my vpn provider and was able to get the max speed that the provider was showing on a speed test (roughly 200Mbps). If this is the case, the issue isn’t easily resolvable other than allowing users to use the v2 endpoints or if the user switches providers. This would explain why the issue varies so much from user to user and seems to disappear if using a VPS with a good backbone connection.
(it’s also possible that google’s web interface uses different endpoints to download files which would explain why that resolves the problem)
So did we just move past this? The version linked /w v1.41-075-ga193ccdb-drive_v2_download before isn’t available anymore and now I’m capped 100% of the time at 25MB/sec even /w gigabit connection. I don’t think @B4dM4n submitted a PR to add --drive-v2-download-min-size and now that version is gone.
I also haven’t seen this option in the latest release from @ncw . Any options to work around this issue anymore?