Workaround for rclone and pihone dns problems

[mlody11]

This post is more of an FYI or in case someone runs into this issue. This is an issue with pihole, not rclone, but it affects rclone behavior.

I had rclone backing up to onedrive and it wouldn't delete and uploads were odd. I checked the script result and I saw these entries:

2021/03/24 11:51:54 Failed to create file system for "onedrive:backups/some_folder_name": failed to get root: Get "...graph dot microsoft dot com / v1.0 /drives/blahblahblah/root": dial tcp: lookup graph dot microsoft dot com on 192.168.xx.xx:53 server misbehaving
2021/03/24 11:51:54 Failed to create file system for "onedrive:backups/some_folder_name": failed to get root: Get "...graph dot microsoft dot com / v1.0 /v1.0/drives/blahblahblah/root": dial tcp: lookup graph dot microsoft dot com on 192.168.xx.xx:53 server misbehaving

This meant that rclone was having DNS resolution issues. Well, I had a pihole serving up DNS requests. Apparently, they introduced rate limiting to the pihole and because rclone is chatty with DNS, it was hitting the rate limiting count when trying to backup/sync. You can see this by looking on your pihole when rclone runs on a sync/copy and you can see a spike on the graph.

The solution is to modify pihole's /etc/pihole/pihole-FTL.conf and set the RATE_LIMIT. I set it to RATE_LIMIT=0/0 because that disables it but I imagine you can tweak and tune as you want.

Hope it helps someone. Sorry if there is a better place to post this topic.

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

thanks, great first post!

perhaps re-post that under "Howto Guides"
you can click the pencil next to the topic title.

also, perhaps change the title to "Workaround for rclone and pihone dns problems" or something like that.

[asdffdsa (jojothehumanmonkey)]

good, i see you made the changes.

[mlody11]

Yup, thanks for the guidance.

[robinvn (Robin)]

Registered here just to say thanks for taking the time to post this here. I had been looking for a solution for quite some time and this helped me out.

[mlody11]

Thanks for letting me know and glad it helped Cheers!

[Jenssaibe]

Thanks for finding this solution.
Had the same problem, decided to exclude my server from Pi-hole. No big deal as it’s only for self hosted services.

[John_Delisle]

I can confirm this is an issue, but I suggest a different approach to solving it.

While disabling rate-limiting on pi-hole does work, it's not addressing the root cause of the problem (it's client-side) and it makes your pi-hole vulnerable to DOS attacks (if that concerns you). Also, and most critically, not everyone uses pi-holes for DNS

Based on packet captures I've looked at, rclone makes a TON of DNS requests for graph microsoft com during operations against OneDrive. The system where you're running rclone should have a local DNS resolver cache, and that cache should bear the brunt of the tens of thousands of requests rclone makes. What we're seeing with this problem is evidence of a failing or misconfigured local resolver. The DNS requests I observed in my packet capture had lots of TTL remaining, yet I was seeing the same request over and over and that is bad news! It means you're going to assault your DNS server with requests, and your DNS server might block you (like our pi-holes do).

To fix this the problem, I recommend enabling your system's local resolver cache. In my case I'm running FreeBSD 12 (TrueNAS 12), and run rclone in an iocage jail. It seems that FreeBSD's local resolver cache (called "Unbound") was not enabled. I enabled it as follows:

sysrc local_unbound_enable=YES
service local_unbound start

Then, I re-ran rclone and problem solved. You should be able to do the same in any OS, and caching is usually enabled by default. I'd bet I'm not the only TrueNAS iocage jail user reading this.