[Drive] How to tell rclone to *not* ignore "Duplicate object found in source". If the file/foulder exsists twice in source, we want it to exsist twice in dest

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I am trying to migrate a partner orgs google drive to a new google drive. They would like us to, to the greatest extent possible, re-create their old drive in a new place, including shortcuts and duplicates.

Seting shortcuts aside for the moment, I cant figure out how to tell rclone to not ignore duplicates. If there are duplicates in the source, we want rclone to create duplicates in the destination. But not sure how to do that other then manually hunting each instance of duplication down.

So if drive1 looks like:

- foulder
  - file1.txt
  - file1.txt
  - file2.txt

We want it to look exactly the same in drive2. But instead it looks like:

- foulder
  - file1.txt
  - file2.txt

So drive2 ends up missing a bunch of data.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: debian 11 (64 bit)
  • os/kernel: redacted
  • os/type: linux
  • os/arch: amd64
  • go/version: redacted
  • go/linking: redacted
  • go/tags: redacted

Which cloud storage system are you using? (eg Google Drive)

Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v -P sync drive1:'The Foulder' drive2:'TheFoulder2' --drive-shared-with-me --create-empty-src-dirs --drive-skip-shortcuts --server-side-across-configs

The rclone config contents with secrets removed.

[drive1]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = {"redacted","expiry":"redacted"}
team_drive = 
root_folder_id = 

[drive2]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = {"redacted","expiry":"redacted"}
team_drive = 
root_folder_id =

A log from the command with the -vv flag

redacted NOTICE: redacted/redacted1.docx: Duplicate object found in source - ignoring
redacted NOTICE: redacted/redacted2.docx: Duplicate object found in source - ignoring
2

You can't.

You have to dedupe the source.

3

I see. Deduping the source might not be possible in this case (it seems some of the duplication's is deliberate, and I dont have write access to entire source drive) so I guess for now I'll try to do the same thing I'm doing with foulder shortcuts and use the lsjson flag to try to make a list of duplicates in the source so I can use a script or a checklist.

I should make this a feature request I suppose. Since from my perspective if the task is "migrate data from driveA to driveB", and I use a tool called rclone, I feel like its reasonable to expect that when your done driveB has all the same data as driveA, warts (and duplicates) and all. Or at least for there to be an option to do so.

4

Most major OS'es/filesystems don't support duplicate files so it is very unlikely it'll change. There are only a handful of remotes that support duplicates.

5

For this specific case (gdrive -> gdrive) wouldn't it be easier just share content and copy inside gdrive? no need for rclone.

And gdrive is unique with allowing for duplicates.

6

That's not really relevant to the average user who uses gdrive and needs to migrate from one gdrive to another, especially if you've been using the source gdrive for a while and the fact that you can have files with the same name baked into how you use it. If these folks have dupes in their source, and they are migrating, its totally reasonable for them to expect duplications in the dest. It would be one thing if the place they are migrating to does not allow duplicated file names, but thats not the case here.

gdrive does not let you make copies of directories, so I cant just go to the "shared with me" section and copy a directory thats been shared with me. Are you suggesting folks go through and manually make copies of 2TB worth of files? Even if we where just focusing on cleanup and making the dest a true copy of the source after using rclone, there are more then a 100 of these (a bunch of which are duplicated directories) and jury is still out on whether manually hunting through a sprawling gdrive to manually copy a bunch of files is going to be quicker / simpler then trying to use a script to do it. Although I'm hoping rclone lsjson + some !cp commands in https://colab.research.google.com will do the trick.

7

It's actually very, very relevant as I've been reading through forums for more than I care to admit as the scenario you describe is very much an edge case and can be solved by not having duplicates. Rclone works with a plethora of cloud backends and the 'general' movement that people do is handled the majority of the time.

The best part about this is that rclone is open source.

If you think it's something that you want to use and it's critical for you, there are a few options:

  1. Code it
  2. Convince some that can code to code it
  3. Sponsor it

And I'm not saying your use case isn't very relevant to you and something you'd like to see.

8

It's actually very, very relevant as I've been reading through forums for more than I care to admit as the scenario you describe is very much an edge case

Its not relevant to my users, so its not relevant to the rational behind the feature request. Perhapes its an edge for standard rclone users, but rclone users are very much not normal google drive users, and sysadmins dont always get to pick their users.

I have no expectations here, If the devs want to throw my feature requests and bug reports in the bin thats totally fine. I have no insight into where this would sit on the endless list of things to do facing every open source project. I just post the things in the hopes folks find em useful.

Although that's not necessarily true, I do have some expectations that if I raise an issue, to not be treated as if my users are doing something wrong when they in no sense of the word are doing anything wrong.

the scenario you describe is very much an edge case and can be solved by not having duplicates.

Hard to read that as anything as blaming my users to some degree. "you would not be having this issue if your users did not do X" as if X was some stupid or incorrect thing to do when it is clearly not since its explicitly allowed by google.

9

Ok - I'm checking out as I clearly said.

So if you reading any blame, that's not me.

Good luck.

1 Like
10

So if you reading any blame, that's not me.

I think reading "the scenario you describe is very much an edge case and can be solved by not having duplicates." as blaming my users is a fair reading of what you wrote. In the first place the message is tautological. "you would not have problem x if you did not have problem x".

Might not have been your intent, but the implied message is equally present: "you would not have problem x if your users did not do x".

EDIT, going to close this thread, as rclone simply can not do what we need it to do (that I can see).

11

That's called projection and you are projecting that as I've said multiple times now that wasn't the case and you keep at it so it's clear where the intent is.

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.