I wanted to open this topic because PiHole 6 has been released recently and I think there are ways to improve the way rclone works in general (not only with Pihole)
The main problem I currently see is the fact of the humongous amount of DNS requests rclone executes, when syncing with something like PCloud
1.7 Million DNS requests in 13h is quite the network traffic. Sure I use pihole and rclone on the same server, so there is only traffic on the lo interface but image this scenario on different servers.
I am not very accustomed to Go, so I can't really debug where the actuall DNS call is being done (or implicitly via just a HTTP call) but maybe there is a way to introduce a sort of "built-in" DNS cache for the currently executed sync command.
Aka only execute 1 DNS request for a given domain if the built in client hasn't resolved it yet. After that use that cache for all ongoing requests (of the current process) instead of re-quering the DNS server.
This would vastly decrease the network load when syncing larger amounts of datasets.
Sure this would break commands, where DNS records change mid execution, but then you have a whole set of other problems.
I agree with @Animosity022. On Linux you have multiple options to enable DNS caching. Bind9 or dnsmasq are two which I used.
I do not think it is good idea to bloat user space program like rclone with DNS functionality (even if it is only caching). And then troubleshoot it and maintain, Especially when there are well established and proven solutions available.
rclone is simply network traffic heavy:) you would probably see similar flood of DNS requests when running big transfer with aws-cli or similar.
Nothing rclone specific is broken, I just noticed, that DNS Requests are a bit wonky in general while the sync is running (which of course is a PiHole problem due to the "high load", not rclone)
Just thought maybe this is something that could be improved in rclone but seems like PiHole needs to improve their DNS resolver to handle "such loads"
And then it should not be really hitting Pi-hole. Unless you run rclone on the same machine?
Your computer DNS resolver you run rclone from should cache DNS requests too so only after TTL expires pihole would be hit by DNS request again.
Are you sure that your client computer (where you run rclone from) has DNS configured correctly? I think caching should be on by default (but not sure about Debian).
DNS caching is ON when using macOS, Windows or Linux Suse Tumbleweed. I would be surprised that Debian does not do it by default. It is not Arch Linux at the end where all features have to be manually configured:) Maybe during installation you chose some custom configuration?
ahaha. This is proper way:) I always try to run things similarly.
This is possible. But not sure why rclone would do it. Go networking libs should use system calls. They are not aware of pihole or anything special. just throw DNS request at port 53.
Maybe worth looking at it closer.
I think that pihole performance here might be not related to DNS queries at all. It is just that when rclone does massive fast transfer it stresses system a bit. Every file has to be hashed, read from storage, send to network. And all with certain parallelism (maybe lower number of simultaneous transfers?)
And maybe it is too much for your Pi. They are fantastic computers but have their limits.
Have a look at overall system utilisation when running your rclone transfer (top/htop). Maybe it is swapping? Or CPU is maxed out? And then as a result pihole slows down - not because this 40 local DNS requests per second.
As can be seen by the screenshot my nextcloud (occasionally) creates more load than the rclone script. I manually started the rclone script again to see how much load it creates.
Yes, i have limited the bandwith of the rclone script via --bwlimit=1M to not 100% saturate my internet/server (had that problem in the past).
