About duplicate objects in source when there's native + non-native docs on Drive

[estan (Elvis Stansvik)]

What is the problem you are having with rclone?

When I rclone copy from Drive to a local directory, I sometimes get messages like

2024/07/11 17:38:26 NOTICE: Data Science/OXT_Template.pptx: Duplicate object found in source - ignoring

I looked into the particular case above and found that it was when there's a Google Docs native sheet called OXT_Template, but also an actual pptx file called OXT_Template.pptx at the source.

It manifests itself in the Drive UI as:

Perhaps it is considered a "duplicate" because the Docs native OXT_Template document is on-the-fly converted to a pptx format file OXT_Template.pptx when rclone copy is doing its thing, and that name-clashed with the existing OXT_Template.pptx?

Is there any way to control how conflict resolution is handled for this particular case, whether to copy the actual .pptx file, or whatever file is generated by the native Docs document? I think I would prefer the latter if possible (though it won't always be the right thing to do.. our Drive is a bit of a mess with these "duplicates" but nothing to do about that now..).

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-91-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy name-of-our-remote: "some_name"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[name-of-our-remote]
type = drive
client_id = XXX
client_secret = XXX
scope = drive.readonly
team_drive = XXX
token = XXX
root_folder_id =

A log from the command that you were trying to run with the -vv flag

I'm sorry, I wasn't using the the -vv flag when initially running this, and I don't want to re-run the exact rclone copy command again right now since it will take quite some time. Let me know if it's crucial that I provide output from -vv and I can run that later.

Below is an excerpt of the output of rclone copy name-of-our-remote: "some_name" which is what I ran:

2024/07/11 17:38:26 NOTICE: Data Science/OXT_Template.pptx: Duplicate object found in source - ignoring

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

@estan

to copy the actual .pptx file

might try --include='*.pptx'

[estan (Elvis Stansvik)]

Thanks, but hmm, the issue is not that I'm not getting any .pptx, the issue is that I want some sort of control of which file is picked by rclone copy when there is both a native Docs spreadsheet called foo and also an actual foo.pptx file next to it. It seems in these cases, rclone copy gives a duplicate file notice, and I'm not sure which one I am getting - Is it the auto-converted native Docs sheet, or the .pptx file next to it?

I haven't checked yet which files I've typically been getting, but I can't find any options to control how the disambiguation is done - I.e. to control which of the two files rclone copy downloads.

[asdffdsa (jojothehumanmonkey)]

@estan

control how the disambiguation is done - I.e. to control which of the two files rclone copy downloads.

from the docs, "Google docs will appear as size -1"

rclone lsf --format=ps remote:
file.pptx;-1
file.pptx;32583

# download native
rclone copy remote: ./ --max-size=0 -v 
INFO  : file.pptx: Copied (Rcat, new)

# download non-native
rclone copy remote: ./ --min-size=0  -v
INFO  : file.pptx: Copied (new)

[estan (Elvis Stansvik)]

Ah, thanks, using --max-size like that is a neat trick. However if I was to use --max-size=0, to prefer the converted Google docs, then I would only get those, not all the other files that I want.

This is part of an rclone copy command to copy all files, not just the Google docs.

Perhaps I can make some sort of combination of two rclone copy commands...

[estan (Elvis Stansvik)]

Do you know what the default choice is? Arbitrary? Or is rclone copy looking at modified dates or similar to decide which one to download when there is a naming clash?

[estan (Elvis Stansvik)]

Actually, it seems I can just run rclone copy like I first did, to make sure I get all files, then finish off with another rclone copy with the --max-size=0 flag like you suggested, to make sure that I get the native versions when there's situations like above, which is what I want. Thanks!

[asdffdsa (jojothehumanmonkey)]

@estan

Thanks!

nice, you exported a neat trick into a good solution ;wink