Rclone dedupe question how long for 500tb

I am trying to dedupe 500tb, 150k~ files. Any idea on the time? Days? Weeks? Months? I have no actual duplicates, I just need to merge 2 Media folders, where 1/2 my data is in each.

Also, is it possible to do on drive cache? At the moment, I am just doing --tpslimit .1

Also my directory structure is :slight_smile:

Media
   TV
      ShowName
          file1
   Movies
      MovieName1
          moviefile1

Media
  TV
      ShowName
          file2
   Movies
      MovieName2
          moviefile2

These will all get correctly merged to the output:

Media
  TV
      ShowName
          file1
          file2
   Movies
      MovieName1
          moviefile1
      MovieName2
          moviefile2

correct?

This is hard to predict. You should add the --fast-list flag to improve the listing phase of dedupe. Reading my gdrive with over 700k files and 40k folders took 18 min via the lsjson command.

But this is the fast phase. Depending on the number of items that need to be moved in the second phase, this will take substantially longer.
If you can, run dedupe on a small subset of your data and measure how long it takes for a known number of items.

I’m not sure if it is possible, but it would also give no benefit.

Exactly, this is the result.