Copying Shared Google Drive Folder Containing Files with Same Names

Help and Support
1

What is the problem you are having with rclone?

I'm trying to understand how to best handle copying a folder with multiple files with the same name from Google Drive to my local machine. In some cases, three or more files will have the same name.

I am not the owner of the Google Drive share, and cannot run rclone dedupe

Also, as I understand, the suffix flag only works once, so if I use suffix .old, if there are three files with the same name, I'll only get two copies, correct?

Is there a way to instruct rclone to use a different suffix per file encountered (like old.1, old.2, etc)?

Run the command 'rclone version' and share the full output of the command.

rclone v1.56.2
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19045.3324 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.16.8
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.\rclone.exe copy --drive-root-folder-id REDACTED GDrive: ./Backup

Note: I didn't actually run the command yet - the folder is large and I wanted to understand the behavior before I started the copy (and potentially had to handle duplicates manually).

Thank you for any help/advice.

2

This is very old version of rclone - update to the latest one to avoid some already fixed old issues.

What flag are you talking about? Maybe I am missing something but I have never heard about such option for duplicates.

Nope

The question is what you want to achieve. Do you want to preserve all duplicates? (as old1, old2 etc) or you know that they are all the same so you want to deduplicate them first?

3

What flag are you talking about? Maybe I am missing something but I have never heard about such option for duplicates.

The --suffix flag allows you to specify that, if a file would be overwritten by something in the copy, apply a suffix first. So if there were multiple copies of fileA.png and I used --suffix "-new", the first would be written as fileA.png and the second would be fileA.png-new.

It was brought up in another thread about this problem.

That doesn't help me though, given there are sometimes three or four files that have the same name, and the suffix is applied only once (if fileA.png-new already exists, it will overwrite it rather than generate fileA.png-new-new).

I was hoping there was a way to do something like what Chrome/Firefox do with downloads and apply a numbered suffix that incremented with each copy.

Do you want to preserve all duplicates? (as old1, old2 etc)

Yes, that's the goal - I'm trying to archive everything in the shared folder, and there are multiple images that have the same names.

4

Ahh OK. Got it now. Yes it only works for two copies.

This does not exist today but indeed it would be useful in cases like this.

But there is solution you can try.

  1. rclone lsf --format "pi" drive: - this will give you list of all files with their full paths and unique gdrive IDs

  2. Loop over the list from the previous step and copy files using their IDs - rclone backend copyid drive: ID dest_path. It can be simple bash script where you detect that dest_path already exist so you add prefix 1 - if exist too prefix 2 etc.

If you have a lot of files you can rclony copy everything first (duplicates won't be copied) and then use described method only to process duplicated files.

5

Loop over the list from the previous step and copy files using their IDs - rclone backend copyid drive: ID dest_path .

This works! It's not fast, but it's functional!

Here's the script I ended up using, in case someone else needs it (I wrote the output of rclone lsf --format "pi" drive: to filelist.txt)

from pathlib import Path
import os, sys

records = Path("./filelist.txt").read_text()
files = Path("./files")

for line in records.splitlines():
    filename, id = line.split(";")
    filename = Path(filename)

    print(f"Will fetch {filename}")

    targetPath = files / filename
    if (targetPath.exists()):
        oldPath = targetPath

        count = 1
        while (targetPath.exists()):
            targetPath = files / (filename.stem + f"_{count}" + filename.suffix)
            count += 1
        
        print(f"Rewrote path from {oldPath.name} to {targetPath.name}")
    
    os.system(f".\\rclone.exe backend copyid GDrive: {id} \"{targetPath}\"")

Thank you for your help!

1 Like
6

Thx for sharing your script.

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.