This is my first day using rclone, so pardon my ignorance if it shows.
I have some massives folders I need to download from google drive and a lot of files have the same filename and are on the same folder. The thing is, the folders are only shared with me, so I can't use dedupe to solve this by renaming the files on the cloud since I have read only permissions set.
Is there any way I can apply a kind of dedupe during copy, so files with same file name are not ignored, but simply renamed?
You'd want to ask the owner and get that deduped as I'm not sure a way to pick the right file if there are duplicates.
Not sure if there are any other creative solutions, but nothing I'm aware of as majority of things don't allow for duplicates.
Unfortunately that's not a possible solution in this case.
Anyway I don't quite understand the implementation here, I've never seen a program that copy files that instead of asking if I want to replace it or to rename it automatically just ignores duplicates.
Unfortunately that's not a possible solution in this case 
Anyway I don't quite understand the implementation here, I've never seen a program that copy files that instead of asking if I want to replace it or to rename it automatically just ignores duplicates.
I'm not following what you are saying.
If rclone see a duplicate (in this case a duplicate is a folder and file with the same name), it would toss an error complaining about a duplicate.
Example.
felix@gemini:~/test$ rclone copy GD:test /home/felix/test -vv
2020/07/24 15:54:48 DEBUG : rclone: Version "v1.52.2-214-g67098511-beta" starting with parameters ["rclone" "copy" "GD:test" "/home/felix/test" "-vv"]
2020/07/24 15:54:48 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/07/24 15:54:48 DEBUG : Local file system at /home/felix/test: Waiting for checks to finish
2020/07/24 15:54:48 DEBUG : Local file system at /home/felix/test: Waiting for transfers to finish
2020/07/24 15:54:48 DEBUG : hosts: MD5 = 157c632175c4251012e651322736f0ae OK
2020/07/24 15:54:48 INFO : hosts: Copied (new)
2020/07/24 15:54:49 DEBUG : Local file system at /home/felix/test: File to upload is small (6084 bytes), uploading instead of streaming
2020/07/24 15:54:49 ERROR : hosts/testodc.docx: Failed to copy: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:49 DEBUG : hosts/testodc.docx: Updating size of doc after download to 6084
2020/07/24 15:54:49 ERROR : hosts/testodc.docx: Failed to copy: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:49 ERROR : Attempt 1/3 failed with 1 errors and: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:49 DEBUG : hosts: Size and modification time the same (differ by 0s, within tolerance 1ms)
2020/07/24 15:54:49 DEBUG : hosts: Unchanged skipping
2020/07/24 15:54:49 DEBUG : Local file system at /home/felix/test: Waiting for checks to finish
2020/07/24 15:54:49 DEBUG : Local file system at /home/felix/test: Waiting for transfers to finish
2020/07/24 15:54:49 DEBUG : Local file system at /home/felix/test: File to upload is small (6084 bytes), uploading instead of streaming
2020/07/24 15:54:49 ERROR : hosts/testodc.docx: Failed to copy: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:49 DEBUG : hosts/testodc.docx: Updating size of doc after download to 6084
2020/07/24 15:54:49 ERROR : hosts/testodc.docx: Failed to copy: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:49 ERROR : Attempt 2/3 failed with 1 errors and: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:49 DEBUG : hosts: Size and modification time the same (differ by 0s, within tolerance 1ms)
2020/07/24 15:54:49 DEBUG : hosts: Unchanged skipping
2020/07/24 15:54:50 DEBUG : Local file system at /home/felix/test: Waiting for checks to finish
2020/07/24 15:54:50 DEBUG : Local file system at /home/felix/test: Waiting for transfers to finish
2020/07/24 15:54:50 DEBUG : Local file system at /home/felix/test: File to upload is small (6084 bytes), uploading instead of streaming
2020/07/24 15:54:50 ERROR : hosts/testodc.docx: Failed to copy: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:50 DEBUG : hosts/testodc.docx: Updating size of doc after download to 6084
2020/07/24 15:54:50 ERROR : hosts/testodc.docx: Failed to copy: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:50 ERROR : Attempt 3/3 failed with 1 errors and: mkdir /home/felix/test/hosts: not a directory
2020/07/24 15:54:50 INFO :
Transferred: 18.228k / 18.228 kBytes, 100%, 16.797 kBytes/s, ETA 0s
Errors: 1 (retrying may help)
Checks: 2 / 2, 100%
Transferred: 1 / 1, 100%
Elapsed time: 1.0s
2020/07/24 15:54:50 DEBUG : 9 go routines active
2020/07/24 15:54:50 Failed to copy: mkdir /home/felix/test/hosts: not a directory
So my local file system does not support duplicates so it errors out trying to make a directory with the same name as an already existing file.
If I try to copy an existing file with same file name in linux it gives an option to automatically rename. Same in Windows. Same in all file managers I've even seen. I just find it odd rclone just ignores the file and doesn't give me the option to rename the duplicate automatically, that's all.
I'm trying to see if other google drive managers allow me do that, if I find it out I'll post it here, rclone is a great tool but in my particular case this problem makes it useless.
That's not what we are talking about here.
Those are the same file names not duplicates.
A duplicate is the same directory name and a file named the same. Linux would not work with that as you can't have that.
Check the image I shared above as it shows an example.
I mean just that. If I have a "test.pdf" on a folder on linux, and try to copy a file named "test.pdf" to the same folder on linux, it will give me the option to rename it automatically, same on windows, and most file managers.
Right, those are not duplicates.
You can choose how you want to handle that as there are multiple options for picking which side wins.
You can use checksums, file sizes, mod times, etc.
It works similar to cp or mv on Linux as you can't rename either.
If you want to use a file manager, you can always mount and try that way as well.
Well, when I try to copy a folder from google drive which has two files in the same location named the same rclone will just detect it as duplicates and ignore one of them. Any way to bypass this?
Also, mounting doesn't work, it will just show a single version of the file, not both.
As my example, a folder has two "test.pdf" files, how to I force rclone to copy both of them by renaming one of them automatically instead of just ignoring one of them?
The specific error is: "duplicate object found in source - ignoring"
You don't. You dedupe the source as rclone doesn't deal with duplicates.
Found an workaround with a different program:
It can mount duplicates if you set it that way on the config file, doing that the mounted folder already has duplicates automatically renamed.
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.