Answering for myself: I can easily see the benefit, but my cost of learning this part of the functionality/code would be too high compared to the value I can create elsewhere.
I for one would welcome this 
- in the meantime I'll try the --dump headers
Good!
I was wondering whether I should put the effort in!
Give this a go. It will count Checks in the stats. You may see file names rclone is working on also. You need -v at minimum to see the stats and by default they print once per minute (adjust with --stats).
v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats on branch fix-dedupe-stats (uploaded in 15-30 mins)
silly question but how do I self-update to the beta? I tried
rclone selfupdate --beta v1.61.0-beta.6543.effbcf5d3
rclone selfupdate --beta v1.61.0-beta.6543.effbcf5d3.fix-dedupe-statsrclone selfupdate --version v1.61.0-beta.6543.effbcf5d3
rclone selfupdate --version v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats
You don't.
You download it as it's not part of the beta yet.
Running now - QQ does the dedupe command not support maxAge?
2022/11/08 15:58:56.604174 DEBUG : --max-age 1.2569199861782407M to 2022-10-02 00:00:00 -0400 EDT m=-3257936.538362699
2022/11/08 15:58:56.617902 DEBUG : rclone: Version "v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats" starting with parameters ["C:\\Users\\Tony\\Downloads\\rclone-v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats-windows-amd64\\rclone-v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats-windows-amd64\\rclone.exe" "dedupe" "--by-hash" "--dedupe-mode" "newest" "remote:" "--dry-run" "--max-age" "2022-10-02" "--filter-from" "P:\\scripts\\filter-file.txt" "--dump" "bodies" "--log-file=P:\\scripts\\log_superDebug.txt" "--log-level" "DEBUG" "--log-format" "date,time,microseconds"]
I see it scanning files that are way old
I don't know for sure but max-age in my mind wouldn't work for dedupe as the goal is to find duplicates.
It was originally created for Google Drive and other remotes that allow duplicate file names / folders in the same spots.
By that logic, it would have a dupe but if max-age was greater than 1 day, it would ignore it? Not sure what the flow would be.
my thought was that I give it a max age and it looks for all the files that have dupes within that age.
You have a valid point, the fact that 4 regular forum members participated in this thread without anybody commenting on this:
tells me that the dedupe documentation needs a warning that filters aren't supported.
Another option is to support filters, but that may be easier said than done, I don't know.
I mean, the PC is broken so that's really the bigger point
I can't see why you'd filter or age a dedupe as it defeats the purpose. The dedupe has been extended quite a bit though since it started though and I don't use it so I really am not familiar with it at this stage.
The connectivity issue is also important and could be due to many other things than a broken pc.
@tb582 Any updates on that? Anything in the Event Viewer?
It is actually quite handy to find duplicates, try something like this:
rclone dedupe --by-hash --dedupe-mode=list --dry-run /your/home/folder/
when using like this, it is a little surprising that I cannot add "--include=*.jpg" to just check my photos.
--dry-run really isn't needed when using --dedupe-mode=list, but better safe than sorry.
Agree.
I'm old and stuck in my ways though so I only have what it was built for as it dealt with duplicates on Google Drive on inception. I don't think rclone as a 'dedupe' tool even though many folks use it for it much like folks use it for a backup tool. Getting a bit on a tangent as I do always say, whatever works best for a person, use that tool. If dedupe works for the use case, use it.
I'll open a separate thread for this issue - b/c it would be really nice for my use-case if dedupe supported the maxAge/filters etc
Everything was going fine:
2022/11/08 21:01:48.642735 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/08 21:01:48.905254 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/08 21:01:48.911860 DEBUG : HTTP RESPONSE (req 0xc0032f0200)
2022/11/08 21:01:48.911860 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 704
Cache-Control: private, max-age=0
Content-Type: application/json; charset=utf-8
Date: Wed, 09 Nov 2022 02:01:48 GMT
Etag: "yF0N3I084JY1cXlIbTqWdYACJREX"
Server: CloudHTTPd-API v1.1
Vary: Accept-Encoding
{
"result": 0,
"md5": "909a31179fb508a8dd17002d0605ee81",
"sha1": "7fe9acbecd161350748928b017545716413352bd",
"metadata": {
"name": "20150126_205422.jpg",
"created": "Tue, 27 Jan 2015 01:54:22 +0000",
"thumb": true,
"modified": "Tue, 27 Jan 2015 01:54:22 +0000",
"canread": true,
"exifdatetime": 1422305661,
"isfolder": false,
"height": 3120,
"width": 4160,
"fileid": 21282908694,
"userid": 11789355,
"ismine": false,
"hash": 14908214687534043973,
"comments": 0,
"category": 1,
"candelete": true,
"id": "f21282908694",
"isshared": false,
"canmodify": true,
"size": 2919121,
"parentfolderid": 5877875849,
"contenttype": "image\/jpeg",
"icon": "image"
}
}
2022/11/08 21:01:48.911860 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/08 21:01:48.911860 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/08 21:01:48.911860 DEBUG : HTTP REQUEST (req 0xc003143200)
2022/11/08 21:01:48.911860 DEBUG : GET /checksumfile?fileid=21282909753 HTTP/1.1
Host: api.pcloud.com
User-Agent: rclone/v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats
Authorization: XXXX
Accept-Encoding: gzip
2022/11/08 21:01:48.911860 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/08 21:01:49.184331 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/08 21:01:49.187523 DEBUG : HTTP RESPONSE (req 0xc003143200)
2022/11/08 21:01:49.187523 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 704
Cache-Control: private, max-age=0
Content-Type: application/json; charset=utf-8
Date: Wed, 09 Nov 2022 02:01:48 GMT
Etag: "1IPj0FsmkvmCvKDu3KNuyfC0sGb7"
Server: CloudHTTPd-API v1.1
Vary: Accept-Encoding
{
"result": 0,
"md5": "89bf4586a3cedfc9a6d0e0745aeb0d89",
"sha1": "b2793581b229cc4e06d8d08fe8bd007e26d682a8",
"metadata": {
"name": "20150126_210547.jpg",
"created": "Tue, 27 Jan 2015 02:05:47 +0000",
"thumb": true,
"modified": "Tue, 27 Jan 2015 02:05:47 +0000",
"canread": true,
"exifdatetime": 1422306346,
"isfolder": false,
"height": 3120,
"width": 4160,
"fileid": 21282909753,
"userid": 11789355,
"ismine": false,
"hash": 11112966277481492445,
"comments": 0,
"category": 1,
"candelete": true,
"id": "f21282909753",
"isshared": false,
"canmodify": true,
"size": 3062796,
"parentfolderid": 5877875849,
"contenttype": "image\/jpeg",
"icon": "image"
}
}
2022/11/08 21:01:49.187523 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/08 21:01:49.188052 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/08 21:01:49.188052 DEBUG : HTTP REQUEST (req 0xc003143600)
2022/11/08 21:01:49.188052 DEBUG : GET /listfolder?folderid=5877875715 HTTP/1.1
Host: api.pcloud.com
User-Agent: rclone/v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats
Authorization: XXXX
Accept-Encoding: gzip
2022/11/08 21:01:49.188052 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/08 21:01:49.202740 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/08 21:01:49.202740 DEBUG : HTTP REQUEST (req 0xc0030cfa00)
2022/11/08 21:01:49.202740 DEBUG : GET /checksumfile?fileid=21239644324 HTTP/1.1
Host: api.pcloud.com
User-Agent: rclone/v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats
Authorization: XXXX
Accept-Encoding: gzip
The issues started here:
2022/11/08 21:01:49.202740 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 09:37:29.142979 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:29.142979 DEBUG : HTTP RESPONSE (req 0xc0030cfa00)
2022/11/09 09:37:29.142979 DEBUG : Error: net/http: TLS handshake timeout
2022/11/09 09:37:29.142979 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:29.142979 DEBUG : pacer: low level retry 1/10 (error Get "https://api.pcloud.com/checksumfile?fileid=21239644324": net/http: TLS handshake timeout)
2022/11/09 09:37:29.142979 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/09 09:37:29.143977 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 09:37:29.143977 DEBUG : HTTP REQUEST (req 0xc00327b200)
2022/11/09 09:37:29.143977 DEBUG : GET /checksumfile?fileid=21239644324 HTTP/1.1
Host: api.pcloud.com
User-Agent: rclone/v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats
Authorization: XXXX
Accept-Encoding: gzip
2022/11/09 09:37:29.143977 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 09:37:29.144881 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:29.144881 DEBUG : HTTP RESPONSE (req 0xc003143600)
2022/11/09 09:37:29.144881 DEBUG : Error: net/http: TLS handshake timeout
2022/11/09 09:37:29.144881 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:29.144881 DEBUG : pacer: low level retry 1/10 (error Get "https://api.pcloud.com/listfolder?folderid=5877875715": net/http: TLS handshake timeout)
2022/11/09 09:37:29.144881 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/11/09 09:37:29.144881 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:29.144881 DEBUG : HTTP RESPONSE (req 0xc00327b200)
2022/11/09 09:37:29.144881 DEBUG : Error: dial tcp: lookup api.pcloud.com: no such host
2022/11/09 09:37:29.144881 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:29.144881 DEBUG : pacer: Reducing sleep to 30ms
2022/11/09 09:37:29.144881 ERROR : Mom/My Pictures/2015-01-30/.picasa.ini: Failed to hash: failed to get hash: Get "https://api.pcloud.com/checksumfile?fileid=21239644324": dial tcp: lookup api.pcloud.com: no such host
2022/11/09 09:37:29.179017 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 09:37:29.179017 DEBUG : HTTP REQUEST (req 0xc00327b600)
2022/11/09 09:37:29.179017 DEBUG : GET /listfolder?folderid=5877875715 HTTP/1.1
Host: api.pcloud.com
User-Agent: rclone/v1.61.0-beta.6543.effbcf5d3.fix-dedupe-stats
Authorization: XXXX
Accept-Encoding: gzip
and the last log lines:
2022/11/09 09:37:31.811625 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 09:37:31.811625 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:31.811625 DEBUG : HTTP RESPONSE (req 0xc000ef4900)
2022/11/09 09:37:31.811625 DEBUG : Error: dial tcp: lookup api.pcloud.com: no such host
2022/11/09 09:37:31.811625 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 09:37:31.811625 ERROR : Mom/My Pictures/wedding: error listing: couldn't list files: Get "https://api.pcloud.com/listfolder?folderid=5877855419": dial tcp: lookup api.pcloud.com: no such host
2022/11/09 09:37:31.812142 DEBUG : 4 go routines active
2022/11/09 09:37:31.812142 Failed to dedupe with 109 errors: last error was: couldn't list files: Get "https://api.pcloud.com/listfolder?folderid=5877855419": dial tcp: lookup api.pcloud.com: no such host
sooo giving the timing between it going fine and then not:
2022/11/08 21:01:49.202740 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 09:37:29.142979 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
FML I think this points to some PC wake-on-lan or something that happens when it goes to "sleep" its the only thing that makes any sense! so I need to figure out what in windows land causes this - I'll probably post on stackExchange somewhere and see what PC folks think...
The PC is going to sleep in mid action here? I'm not sure I'm following that bit.
No it doesn't...
However it is very easy to make it support the filters.
Try this which has the previous change and the filter support for dedupe.
v1.61.0-beta.6544.91aeb4e36.fix-dedupe-stats on branch fix-dedupe-stats (uploaded in 15-30 mins)
Looks like you PC doesn't support Modern Standby (S0 low power idle) and therefore is unable to gracefully "sleep" while programs like rclone have ongoing network activity.
Modern standby isn't truly sleeping the CPU and network card. They are kept alive, but the power is reduced to almost nothing while keeping network connections open and occasionally checking for mails, updates etc.
You can check your PC with this command (may need Admin terminal):
powercfg -a
More info on Windows sleep modes here:
https://learn.microsoft.com/en-us/windows/win32/power/system-power-states
https://learn.microsoft.com/en-us/windows-hardware/design/device-experiences/modern-standby
If your PC doesn't support Modern Standby, then the best you can do is to keep the PC from sleeping while rclone is active. That is to increase the time before Windows goes into sleep in Settings; or set it to Never.
The above still looks like a network adapter being reset. Was the PC entering sleep when this happened?
oh how interesting...
The following sleep states are available on this system:
Standby (S3)
Hibernate
Hybrid Sleep
Fast Startup
The following sleep states are not available on this system:
Standby (S1)
The system firmware does not support this standby state.
Standby (S2)
The system firmware does not support this standby state.
Standby (S0 Low Power Idle)
The system firmware does not support this standby state.
this is the mobo that I'm using: https://www.asrock.com/mb/Intel/B460M-ITXac/
Nice PC, now let's find out if you have a problem with your Giga PHY Inte® I219V, cabling, router or ISP.
Was the PC entering sleep when the below happened?
I THINK so but can't find the right event logs to back that up but the timing with my sleep schedule seems to line up
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.