Deduping identical files and listing the rest

Malam · October 2, 2022, 5:28pm

What is the problem you are having with rclone?

I'm wondering if there's a way to combine --dedupe-mode skip and --dedupe-mode list to remove identical duplicates and list the rest. Currently I'm doing both with && in the middle, but that requires going through the directory twice

Run the command 'rclone version' and share the full output of the command.

`rclone v1.57.0

os/version: Microsoft Windows 10 Home 2009 (64 bit)
os/kernel: 10.0.19043.2006 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.17.2
go/linking: dynamic
go/tags: cmount`

Which cloud storage system are you using? (eg Google Drive)

Gdrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone dedupe "Drive:dir" --drive-use-trash=false --dedupe-mode skip -v --no-traverse --transfers=32 && rclone dedupe "Drive:dir" --drive-use-trash=false --dedupe-mode list -v --no-traverse --transfers=32
(Do my flags make sense?)

Ole · October 3, 2022, 6:26am

Hi Malam,

I don't think so, the dedupe-mode is translated to an integer value internally, so something like --dedupe-mode="skip,list" isn't possible and not easily added to rclone (if at all possible).

github.com

rclone/rclone/blob/a3c06b9bbe148ac4995c26dac3a5ce1b603a176e/fs/operations/dedupe.go#L181-L195


      
          // DeduplicateMode is how the dedupe command chooses what to do
          type DeduplicateMode int
          
          
// Deduplicate modes
          const (
          	DeduplicateInteractive DeduplicateMode = iota // interactively ask the user
          	DeduplicateSkip                               // skip all conflicts
          	DeduplicateFirst                              // choose the first object
          	DeduplicateNewest                             // choose the newest object
          	DeduplicateOldest                             // choose the oldest object
          	DeduplicateRename                             // rename the objects
          	DeduplicateLargest                            // choose the largest object
          	DeduplicateSmallest                           // choose the smallest object
          	DeduplicateList                               // list duplicates only
          )

ncw · October 3, 2022, 10:16am

Did you try running with -vv? That will given you more info, but not sure it it will be enough.

system · November 2, 2022, 10:16am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Deduping identical files and listing the rest

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)