Possible to dedupe a union and give path/back end remote priority as mode?

Not sure if this is possible but figured it can't hurt to ask. If I have a union remote named union: consisting of /mnt/path1 and /mnt/path2, and I run rclone dedupe union:, is there a flag or option to say only delete dupes from /mnt/path2?

Essentially I'm trying to find hash matches across two paths, the first path is read only, the second path is rw, so obviously, I can only delete from the second path. I'm not sure if there is a way to do this with rclone. Thanks in advance for any help/guidance.


you have a union consisting of two local paths?

Yes. But they can be remotes I'd that's easier to achieve what I'm trying to do.

if both paths are local, then you can use any depude tool supported by your operating system.

I primarily use rclone but this scenario is a bit more advanced than previous ones. I'll google some linux utilities.

Are you Linux?

rdfind is a nice one that I've used before.

This might give you a few ideas too:

1 Like

I am! Thank you for the pointer, will def check it out . Might be better than my rclone lsf hash lists into excel lol :slight_smile:

You can also do this with the relatively new rclone dedupe --by-hash which is probably pretty similar to your hashes into excel but gives tools for which one to delete.

1 Like

Exactly what I needed. Thanks! I ran it, and it appears to only spit out hashes, not file names. Is their a flag to spit out file names with the hashes? Thanks again!

What version of rclone are you using?

I see this with 1.56.0

$ rclone dedupe --by-hash a
2021/07/31 12:03:08 NOTICE: b1946ac92492d2347c6235b4d2611184: Found 2 files with duplicate md5 hashes
b1946ac92492d2347c6235b4d2611184: 2 duplicates
  1:            6 bytes, 2021-07-31 12:02:35.129599017, one
  2:            6 bytes, 2021-07-31 12:02:37.265480512, two
s) Skip and do nothing
k) Keep just one (choose which in next step)

where the two files are called one and two.

1.55 I believe. I'll double check. However I was trying to do so unattended and spit out to a file so I could parse and mass delete. The output in non interactive doesn't seem to provide file names. But I'll try again with 1.56. Thanks again.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.