Dedupe Should support filters

[tb582]

for whatever reason the dedupe command doesn't support maxAge nor includesFiles - I'd like for this to be supported and going by the docs it should be, at least it doesn't mention any global flags not supported. (Hence filing as a bug)

rclone dedupe

Dial tcp: lookup api.pcloud.com: no such host

You have a valid point, the fact that 4 regular forum members participated in this thread without anybody commenting on this: tells me that the dedupe documentation needs a warning that filters aren't supported. Another option is to support filters, but that may be easier said than done, I don't know.

[Animosity022]

It's not a bug as it's an enhancement as every global flag doesn't work on every command as trying to put every single exception in every spot would be brutal.

Feel free to submit a feature request on the github.

Issues · rclone/rclone

[ncw (Nick Craig-Wood)]

Actually I had a go at implementing this.

I had to change two true values in the code to false so it wasn't hard and seems like a good idea. This has the previous dedupe fix in too (to show checks).

v1.61.0-beta.6544.91aeb4e36.fix-dedupe-stats on branch fix-dedupe-stats (uploaded in 15-30 mins)

[tb582]

SWEET! thanks @ncw

[tb582]

confirmed this is working as expected!

2022/11/09 13:45:22.908845 DEBUG : --max-age 1.2871616156726466M to 2022-10-02 00:00:00.0010215 -0400 EDT m=-3336322.828519599
2022/11/09 13:45:22.910375 DEBUG : rclone: Version "v1.61.0-beta.6544.91aeb4e36.fix-dedupe-stats" starting with parameters ["C:\\Users\\Tony\\Downloads\\rclone-v1.61.0-beta.6544.91aeb4e36.fix-dedupe-stats-windows-amd64\\rclone-v1.61.0-beta.6544.91aeb4e36.fix-dedupe-stats-windows-amd64\\rclone.exe" "dedupe" "--by-hash" "--dedupe-mode" "newest" "remote:" "--dry-run" "--max-age" "2022-10-02" "--filter-from" "P:\\scripts\\filter-file.txt" "--dump" "bodies" "--log-file=P:\\scripts\\log_superDebug.txt" "--log-level" "DEBUG" "--log-format" "date,time,microseconds"]

results:

2022/11/09 13:46:27.620818 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 13:46:27.621381 NOTICE: a3512c98c9e159c021ebbb76b238707e: Found 2 files with duplicate md5 hashes
2022/11/09 13:46:27.621381 NOTICE: My Pictures/Tony/Automatic Upload/Tony’s iPhone/2022-10-08 21-46-21 (2).mov: Skipped delete as --dry-run is set (size 477.127Mi)
2022/11/09 13:46:27.621381 NOTICE: a3512c98c9e159c021ebbb76b238707e: Deleted 1 extra copies
2022/11/09 13:46:27.621381 NOTICE: 14a59c9c2e6d4725fe1326e1a3a1d74f: Found 2 files with duplicate md5 hashes
2022/11/09 13:46:27.621381 NOTICE: My Pictures/Tony/Automatic Upload/Tony’s iPhone/2022-10-08 21-58-26 (2).mov: Skipped delete as --dry-run is set (size 354.166Mi)
2022/11/09 13:46:27.621381 NOTICE: 14a59c9c2e6d4725fe1326e1a3a1d74f: Deleted 1 extra copies
2022/11/09 13:46:27.621381 NOTICE: b5de27aa5bba0f280ebf7eb7c3e2f768: Found 2 files with duplicate md5 hashes
2022/11/09 13:46:27.621915 NOTICE: My Pictures/Tony/Automatic Upload/Tony’s iPhone/2022-10-21 17-02-06 (2).jpeg: Skipped delete as --dry-run is set (size 6.223Mi)
2022/11/09 13:46:27.621915 NOTICE: b5de27aa5bba0f280ebf7eb7c3e2f768: Deleted 1 extra copies
2022/11/09 13:46:27.621992 DEBUG : 3 go routines active

Anything I need to do to ensure this gets in the next build?

[Animosity022]

That should be it.

He'll merge into the beta and should arrive on the next release.

[ncw (Nick Craig-Wood)]

@tb582

confirmed this is working as expected!

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.61