To many DNS -request pihole+unbound

What is the problem you are having with rclone?

Every time I try to sync I get some error's in my logfile. A look at the pihole shows, that rclone send so muche dns-request for the sync-process. I thought first it's a issue from the pi-hole, but then I writted in the pihol-forum and there was the answer, that rclone is the problem.
Why need rclone so muche unnecessary dns-request for sync?

What is your rclone version (output from rclone version)

rclone v1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Synology DSM7

Which cloud storage system are you using? (eg Google Drive)

Onedrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here
sudo rclone sync /volume1/homes/daniel synotoonedrive:/daniel --progress -vv --log-file=/var/log/rclone.log

A log from the command with the -vv flag

2021/08/30 21:21:30 ERROR : Encrypted drive 'synotoonedrive:/daniel': not deleting files as there were IO errors
2021/08/30 21:21:30 ERROR : Encrypted drive 'synotoonedrive:/daniel': not deleting directories as there were IO errors
2021/08/30 21:21:30 ERROR : Attempt 2/3 failed with 244 errors and: march failed with 219 error(s): first error: couldn't list files: Get "https://graph.microsoft.com/v1.0/drives/xxxxx/items/xxxxx!42725/children?$top=1000": dial tcp: lookup graph.microsoft.com on 192.168.178.222:53: server misbehaving

I've never noticed/had any DNS issues with rclone.

You may find this thread interesting:

Tip: You can search in the rclone forum with the magnifier in the top right corner next to the hamburger bar - there also some advanced options if needed.

I saw this thread but it's only a workaround rclone send hundrets of dns request per minute and if I change the thinks in the workaround rclone falsified the blocklist percent in the pihole.

because of this I make tje new thread.

hello,

keep in mind that if rclone get a high level error, it will retry the operation three times and for a low-level error rclone will try ten times.
that can inflate the total number of dns requests.

in the pi-hole post, you stated that the pi-hole on the pi4 works without error
it is the pi-hole on the synbox that is the problem.
i assume that you have tested rclone on both and only the synbox is the problem.
if my assumption correct?

tho i have seen dns errors in the past, never saws server misbehaving

in the pi-hole forum post, there was a second workaround, which would seem to prevent your concern.
"falsified the blocklist percent in the pihole"
did you try that and it failed?

if you want a deeper look into rclone, you can add this your command
--dump=headers --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log

Do you mean with synbox the synology?

I changed the RATE_LIMIT on the pihole, now there are no errors any more but I can see there are despite this hundrets of dns-request.

Is it possible that rclone make for every file a request? The folder I sync have about 11.000 files.

synbox is the synology device.

rclone does not make a dns request per file.

with rclone, does the dns issue happen with the synbox, the pi or both?

11,000 files is a small number of files.
i have a data set with over 1,000,000+ files and other rcloners have much larger sets of files.

at this point, if you want to diagnose this further, need to post /var/log/rclone.log

rclone most definitely makes a lot of requests, at least with OneDrive.

I just checked my DNS server logs (BIND; a real name server :slight_smile: ) and it made 3188 calls (half for AAAA records, half for A records) to graph.microsoft.com in a 7 minute window (9:23:00.815 and 9:30:13.589) this morning.

And this is to be expected. Each connection to graph to check a file's meta information is a unique http request, and this will lead to a resolver request for the domain name. An app does not (and IMHO should not!) cache DNS information. That's just asking for complication and failures.

On a host like Windows, Microsoft default-enable a DNS cache service (which has had problems in the past). Linux can be configured to run a caching daemon (e.g nscd or sssd) and nsswitch.conf modified to point to it. BSD systems have their own variation.

So if you're seeing too many DNS hits then the machine rclone is running on should be configured to do name lookup caching.

1 Like

I have now rclone tested on my Pc (Linux mint) there I saw also many DNS-Request:

and from the Synology:

Some lines from the rclone.log there are no Error:

2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/share.db: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/size.db: Size and modification time the same (differ by -582.619463ms, within tolerance 1s)
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/size.db: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/report.html: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/dup.db: Size and modification time the same (differ by -895.597519ms, within tolerance 1s)
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/dup.db: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/folder.db: Size and modification time the same (differ by -322.616081ms, within tolerance 1s)
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/folder.db: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/file.db: Size and modification time the same (differ by -138.613687ms, within tolerance 1s)
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/file.db: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/mtime.db: Size and modification time the same (differ by -651.620361ms, within tolerance 1s)
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/mtime.db: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/INFO: Unchanged skipping
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/atime.db: Size and modification time the same (differ by -742.621544ms, within tolerance 1s)
2021/08/31 18:57:15 DEBUG : berichte/synoreport/Berichte/2020-07-23_22-00-03/atime.db: Unchanged skipping
.
.
.
.
2021/08/31 18:57:22 DEBUG : Encrypted drive 'synotoonedrive:/daniel': Waiting for checks to finish
2021/08/31 18:57:22 DEBUG : Encrypted drive 'synotoonedrive:/daniel': Waiting for transfers to finish
2021/08/31 18:57:22 DEBUG : Waiting for deletions to finish
2021/08/31 18:57:22 INFO  : 
Transferred:   	  174.306Ki / 174.306 KiByte, 100%, 611 Byte/s, ETA 0s
Checks:             10955 / 10955, 100%
Transferred:            3 / 3, 100%
Elapsed time:      1m23.6s

2021/08/31 18:57:22 DEBUG : 5 go routines active

FWIW there is a GoLang library which could do the caching (I still don't think it's the right thing to do), and you can tell the http libraries to use it. See GitHub - viki-org/dnscache: A DNS Cache for Go for details.

The raspberry is my network-wide ad-blocker and dns server.

I was wondering why on my Synbox and Pc there are so muche DNS-request are.
But when @sweh said it's normal and by him there are also so muche request then I will deal with it, before I change to muche on my Synology and I breake more then I fix.

The whole thing didn't leave me alone yesterday, which is why I have now put on a night shift and found a solution that suits me.

On my Synology I installed the DNS-Server with following configuration.

DNS-Config

and changed the DNS-Server from the Synology to the own ip.

Network

I don't know if this is a right way or the best, but now it work's for me.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.