Rclone dedupe - global or folder-based?

Quick question - I’m seeing a huge number of dupe files in my pictures folder, so running dedupe to clean them up. It’s been running for ages and hasn’t done anything yet. :slight_smile:

But I’m wondering whether the dedupe is global (i.e., does rclone build a huge map of all files each with an MD5 hash) or is it folder-based (i.e., does rclone iterate each folder, and then enumerate any duplicate file entries within that folder structure)?

So in essence, if I have

/root/folder1/somefile.jpg
/root/folder1/somefile.jpg
/root/folder2/somefile.jpg
/root/folder2/somefile.jpg

will I end up with:

/root/folder1/somefile.jpg

and the folder2 being empty, or will I end up with

/root/folder1/somefile.jpg
/root/folder2/somefile.jpg

Im pretty sure it goes by folder. You’re allowed to have duplicates in different folders as those aren’t duplicates really in the context of this fixing them. I also be believe in recent betas it will also dedup folders themselves like this:

/root/folder1
/root/folder1

becomes just /root/folder1 and moves the files that were contained in each into a combined one. This just fixes google wierdness.

It does - it is looking for identical file names in the identical folder.

However for historical reasons it actually builds a map of the entire directory tree first which it really shouldn’t.

I made an issue #1700 to fix this in the next release!