resolves across all the major DNS providers too
resolves across all the major DNS providers too
Rclone just calls the OS for a host name.
Guessing you are on Windows with some configuration perhaps blocking it? AV? Defender?
That's a general network error that you had a connection timeout to the provider.
That means DNS stopped working for rclone which could be AV/Defender/etc.
Rclone is just reporting the error it sees.
yes windows 10 - I don't have any third party AV/ad blocker etc just Windows Defender running which I can't seem to find any issues with
If no other person is reporting it, it's very unlikely rclone has a systemic DNS problem as it's more likely a local issue on your machine somewhere.
I'm not a Windows user so really can't offer much more to troubleshoot.
can you open https://api.pcloud.com in a web browser?
I don't think it is a DNS issue, it looks more like you are having transient network connectivity issues, that also sometimes happens to affect the connection to your DNS server.
Here is an observation supporting this (a timeout between two IP addresses is pure connectivity - no DNS involved):
Next you see this (which could be a DNS issue or a general connectivity issue):
which I would consider a general connectivity issue when seen immediately after the first issue.
Here are some posts with my suggestions to troubleshoot network issues like these on Windows:
Not nitpick but DNS is already resolved to an IP as that's just a network/connectivity timeout so that would be a general connectivity issue.
Fully agree, as I (tried to) communicate above it:
ping -t api.pcloud.com I see no issues thus far running this - yet while that same ping -t is running fine doing the rclone dedup command hangs and eventually throws the same errors as above - again meanwhile ping -t is progressing just fine
OK, that was unexpected.
I still suspect some transient connectivity issues, but it seems more subtle than my experience.
Do you see the same with other rclone commands to pcloud?
Do you see the same with other rclone remotes (e.g. Google Drive)?
I honestly only use rclone with pcloud - and its seemingly more of an issue with dedupe than it is with other things lsf and copy commands seem fine.
BUT I think part of the issue might be the dedupe command and how it actually runs b/c it waits at this stage for a LONG time before it starts throwing errors
2022/11/07 13:40:07 DEBUG : rclone: Version "v1.60.0" starting with parameters ["C:\\Program Files (x86)\\rclone\\rclone.exe" "dedupe" "--by-hash" "--dedupe-mode" "newest" "remote:" "--dry-run" "-vv" "--max-age" "2022-10-02" "--filter-from" "P:\\scripts\\filter-file.txt"] 2022/11/07 13:40:07 DEBUG : Creating backend with remote "remote:" 2022/11/07 13:40:07 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf" 2022/11/07 13:40:07 INFO : pcloud root '': Looking for duplicate md5 hashes using newest mode.
So rclone is doing a lot of network activity for a long time (I see 12+ hours in the first log).
It could be random and then you only see it with long running internet intensive commands after a random amount of time
Or it could be time dependent where it always happens after approximately the same amount of rclone execution time
Or it could be a certain hours of the day
Or something else
Which pattern do you see?
Are you using WiFi?
That seems maybe like a wonky home router or something dropping connections if it gets pushed too hard.
No - hard wired
Don't think its time of day nor hours -
here's the latest log run from today - thats over two hours from when it says its looking for duplicates vs actually showing ANY activity.
2022/11/07 13:40:07 INFO : pcloud root '': Looking for duplicate md5 hashes using newest mode. 2022/11/07 15:57:05 DEBUG : pacer: low level retry 1/10 (error Get "https://api.pcloud.com/checksumfile?fileid=11749612966": dial tcp 18.104.22.168:443: connectex: A socket operation was attempted to an unreachable host.) 2022/11/07 15:57:05 DEBUG : pacer: Rate limited, increasing sleep to 20ms 2022/11/07 15:57:05 DEBUG : pacer: Reducing sleep to 15ms 2022/11/07 15:57:05 ERROR : My Pictures/2015/06/2015_06_22_33.jpg: Failed to hash: failed to get hash: Get "https://api.pcloud.com/checksumfile?fileid=11749612966": dial tcp: lookup api.pcloud.com: no such host
meanwhile the ping running continuously during that same period!
Ping statistics for 22.214.171.124: Packets: Sent = 8736, Received = 8706, Lost = 30 (0% loss), Approximate round trip times in milli-seconds: Minimum = 43ms, Maximum = 796ms, Average = 58ms Control-C
hmm just noticed the 30 lost packets
Reply from 126.96.36.199: bytes=32 time=85ms TTL=43 Reply from 188.8.131.52: bytes=32 time=53ms TTL=43 Reply from 184.108.40.206: bytes=32 time=69ms TTL=43 Reply from 220.127.116.11: bytes=32 time=47ms TTL=43 No resources. No resources. General failure. General failure. General failure. Reply from 18.104.22.168: bytes=32 time=53ms TTL=43 Reply from 22.214.171.124: bytes=32 time=55ms TTL=43 Reply from 126.96.36.199: bytes=32 time=55ms TTL=43 Reply from 188.8.131.52: bytes=32 time=51ms TTL=43 Reply from 184.108.40.206: bytes=32 time=48ms TTL=43 Reply from 220.127.116.11: bytes=32 time=56ms TTL=43 Reply from 18.104.22.168: bytes=32 time=49ms TTL=43 Reply from 22.214.171.124: bytes=32 time=51ms TTL=43 Reply from 126.96.36.199: bytes=32 time=48ms TTL=43 Reply from 188.8.131.52: bytes=32 time=47ms TTL=43 Reply from 184.108.40.206: bytes=32 time=54ms TTL=43 Reply from 220.127.116.11: bytes=32 time=52ms TTL=43 Request timed out. Reply from 18.104.22.168: bytes=32 time=53ms TTL=43 Reply from 22.214.171.124: bytes=32 time=58ms TTL=43
Let me turn on ping timestamps and re-run - see ya in a few hours ---
I would like to figure out why it takes rclone hours to produce any verbose output - seems like it needs a super verbose (@ncw )
Seems like a bad network card/adapter/etc.
All the evidence points to a local isolated issue. Occam's Razor.
With ping, you've removed rclone from the equation.
I agree with @Animosity022, this pattern looks like a network adapter being reset by Windows, so you can probably find more info in the System Event Log seen in the Event Viewer. Good idea to add timestamps, that will make it easier.
I would disconnect/reconnect all network cables from your computer to the router to the wall and then restart router, switches and computer in that order. If this doesn't help then I would try updating all components and changing cables.
If you suspect an issue with rclone dedupe or pcloud, then I suggest you try reproducing on the smallest dataset possible. That is start with your smallest subfolder and then gradually move up in size until you see the issue.
You can add --dump headers if you would like some extra verbosity, it will show the communication between rclone and pcloud.
That seems to be correct behavior according to the code. First rclone collect a list of all files and then it collect the hashes for each of the files, that may take some time if you have a lot of data (or a slow/unstable connection):
Is the observed behavior different from your earlier dedupes or trials on smaller folders?
It would be easy to mark some checkers here then at least you'd see something going on in the stats. Worth doing?
Answering for myself: I can easily see the benefit, but my cost of learning this part of the functionality/code would be too high compared to the value I can create elsewhere.
I for one would welcome this - in the meantime I'll try the --dump headers
I was wondering whether I should put the effort in!
Give this a go. It will count
Checks in the stats. You may see file names rclone is working on also. You need
-v at minimum to see the stats and by default they print once per minute (adjust with --stats).
v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats on branch fix-dedupe-stats (uploaded in 15-30 mins)