2020/03/10 06:03:36 DEBUG : rclone: Version "v1.47.0" starting with parameters [
"rclone" "dedupe" "-vv" "--dedupe-mode" "newest" "D:\gallery-dl\gallery-dl"]
2020/03/10 06:03:36 DEBUG : Using config file from "MYREALNAME.conf"
2020/03/10 06:03:36 INFO : Local file system at \?\D:\gallery-dl\g
allery-dl: Looking for duplicates using newest mode.
2020/03/10 06:03:36 DEBUG : 2 go routines active
2020/03/10 06:03:36 DEBUG : rclone: Version "v1.47.0" finishing with parameters
["rclone" "dedupe" "-vv" "--dedupe-mode" "newest" "D:\gallery-dl\gallery-dl"]
note: this folder is full of 100s of duplicate files, and the process ran instantly, I don't think it checked a single md5.
note2: none of the files have the same name because windows wouldn't allow that, but all the duplicates have been renamed to _001 or _002. I guess dedupe doesn't compmare md5's between files without identical names? It would be useful if it would, because, my computer won't let any such file exist, so my cloud backup is unlikely to either.
I uploaded the same folder to google drive, and I ran the dedupe again, it did nothing at all, and completed instantly.
This is a problem small enough that I could just scan like 500 duplicates and delete them myself but I'd like to try and take this as a chance to learn.
I've had this problem literally 1000s and 1000s of times. sometimes I solve it, and sometimes I just let my cloud backup get cluttered.
I know running deduping on a million files without caring about filename would be impossible, but for a few 100 small files it actually wouldn't take that long to just make all the md5's and compare all of them to all of them. I guess rclone doesn't have a command for that yet though.
Well I deleted the 250 or so duplicates one at a time by hand. Now i'm reuploading them to googledrive, but really, it'd be great if i could find a tool that would do this for me. It'd be great if it worked via rclone too, because like I said, in the past, literally maybe 1000 times, I've had this problem, and given up, and just sent the files to googledrive.
Thing is, i don't need to deduplicate the whole cloud drive, just within specific folders. I know that the files in different folders won't be duplicates, but often files within one folder ARE duplicates that I stupidly let windows make before cloning that error to googledrive.