Dedupe --by-hash throws error

Help and Support
1

This error only happens with v1.54 of rclone and the --by-hash param. Regular dedupe works fine so this appears to be an rclone issue with pcloud and byhash dedup.

2021/01/10 15:09:51 DEBUG : rclone: Version "v1.54.0-beta.5050.ef2bfb971" starting with parameters ["C:\\Program Files (x86)\\rclone-v1.54.0-beta.5050.ef2bfb971-windows-amd64\\rclone.exe" "dedupe" "remote:" "--dry-run" "--bind" "0.0.0.0" "-vv" "--by-hash" "--progress"]
2021/01/10 15:09:51 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2021/01/10 15:09:51 DEBUG : Creating backend with remote "remote:"
2021-01-10 15:09:51 INFO  : pcloud root '': Looking for duplicate MD5 hashes using interactive mode.
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:     41m34.5s
C:\Users\Tony>"C:\Program Files (x86)\rclone-v1.54.0-beta.5050.ef2bfb971-windows-amd64\rclone.exe" dedupe remote: --dry-run --bind "0.0.0.0" -vv --by-hash --progress --filter-from "C:\Program Files (x86)\rclone\filter-file.txt"
2021/01/10 15:51:37 DEBUG : rclone: Version "v1.54.0-beta.5050.ef2bfb971" starting with parameters ["C:\\Program Files (x86)\\rclone-v1.54.0-beta.5050.ef2bfb971-windows-amd64\\rclone.exe" "dedupe" "remote:" "--dry-run" "--bind" "0.0.0.0" "-vv" "--by-hash" "--progress" "--filter-from" "C:\\Program Files (x86)\\rclone\\filter-file.txt"]
2021/01/10 15:51:37 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2021/01/10 15:51:37 DEBUG : Creating backend with remote "remote:"
2021-01-10 15:51:37 INFO  : pcloud root '': Looking for duplicate MD5 hashes using interactive mode.
2021-01-10 16:40:10 DEBUG : pacer: low level retry 1/10 (error Get "https://api.pcloud.com/checksumfile?fileid=11577473147": dial tcp 0.0.0.0:0->74.120.9.121:443: i/o timeout)
2021-01-10 16:40:11 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021-01-10 16:40:11 DEBUG : pacer: Reducing sleep to 15ms
2021-01-10 16:40:11 ERROR : My Pictures/2010/08/2010_08_05_16.jpg: Failed to hash: failed to get hash: Get "https://api.pcloud.com/checksumfile?fileid=11577473147": dial tcp: lookup api.pcloud.com: no such host
2021-01-10 16:40:11 DEBUG : pacer: Reducing sleep to 11.25ms
2021-01-10 16:40:11 ERROR : My Pictures/2010/08/2010_08_05_16.nef: Failed to hash: failed to get hash: Get "https://api.pcloud.com/checksumfile?fileid=11577474633": dial tcp: lookup api.pcloud.com: no such host
2

It looks like things are going wrong when calling the API for reading the hash.

Can you try

rclone md5sum remote: --bind "0.0.0.0" -vv

to see if that replicates the problem?

This looks like the checksum API call timed out. Are the files you are checksumming really big?

This is definitely some kind of networking problem as we discussed earlier.

3

rclone md5sum remote: --bind "0.0.0.0" -vv

runs without issue and almost immediately starts returning the checksums

4

That is good...

Can you try the original problem using a different resolver? So 1.1.1.1 or 8.8.8.8 ?

5

trying that oddly enough doesn't create any api errors but also doesn't seem to do anything else and is super quick to "finish"?

C:\Users\Tony>"C:\Program Files (x86)\rclone-v1.54.0-beta.5050.ef2bfb971-windows-amd64\rclone.exe" dedupe remote: -vv --dry-run --progress
2021/01/14 09:59:06 DEBUG : rclone: Version "v1.54.0-beta.5050.ef2bfb971" starting with parameters ["C:\\Program Files (x86)\\rclone-v1.54.0-beta.5050.ef2bfb971-windows-amd64\\rclone.exe" "dedupe" "remote:" "-vv" "--dry-run" "--progress"]
2021/01/14 09:59:06 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2021/01/14 09:59:06 DEBUG : Creating backend with remote "remote:"
2021-01-14 09:59:06 INFO  : pcloud root '': Looking for duplicate names using interactive mode.
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:        57.1s
2021/01/14 10:00:03 DEBUG : 3 go routines active

something seems off I know there's hash duplicates.

6

I think you forgot the --by-hash parameter

7

doh! - ok its running now and no errors thus far 48min in.

QQ - does it show dupes as it finds them or all at once at the end?

in case anyone is looking for the answer to the api error indeed changing my DNS from comcast to google 8.8.8.8 worked which is odd.

8

It will show them at the end when it has collected all the data.

All the symptoms pointed towards sketchy DNS, I don't know what is wrong with Comcast's.

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.