I was hoping someone here may have some additional insight into the behavior outlined in this issue:
In short, Google Drive (and other some other storage providers) support multiple files with the same name, e.g. two files name foo.txt in the same dir. The backend we have for Google Drive has some logic to try and handle duplicates by essentially marking the original for deletion before uploading the new copy, and if successful, deleting the original.
My main question is why are we actually doing this? If the platform supports some specified behavior, why are we creating a backend API that is more restrictive?
As it stands though, the current logic fails to handle duplicates beyond the first copy though. If I already have 2 files on the remote named test.txt, and I go to upload a new copy from local, only one of them is actually replaced. The second is never seen.
I'm sure there's a good reason for this behavior that I'm not aware of, so any insight here is appreciated. There were some suggestions made in the linked issue (as well as in the referenced issues) to support this.
At the very least, I'd be open to working on a patch to bulletproof the duplicate checking if my understanding is correct.
In general, the use case was to bridge the gap between local file systems and cloud storage. Only a few providers offer duplicate file name and to my knowledge, you can’t do that on any local file systems.
So things like a mount would break if you had duplicates.
Sure, yeah I totally get that. To be clear, my point is that I feel like there may be better solutions to this problem, i.e. what was discussed in the linked issue.
Each provider that's currently supported by rclone has it's own backend API that leverages the core network/file IO logic. These backends have their own proprietary implementation for methods like copy so that they can wrap the core copy logic while also adding some proprietary business logic based on the target platform's requirements.
In other words, even though both iCloud and Google Drive have a copy method, their implementations are different, which leaves room for us to more explicitly handle a case like the behavior in Google Drive.
I think the main reason is that rclone presents a path-based filesystem view, even when the backend is not really path-unique. If two Drive objects have the same parent+name, operations like mount, check, sync destination matching, and path-only logs become ambiguous. For a patch, I would lean toward making duplicate detection more explicit: detect multiple matches for the destination path and either warn/error or handle all matching IDs behind a flag. Silent partial replacement is the awkward bit here.
Mm okay that makes sense. I'll explore the duplicate detection a bit and if I can come up with something reasonable, I'll submit a patch. Thanks for the insight.
Yes, this is precisely the case where the "common interface" is starting to crack.
The idea of rclone as an abstraction over various cloud APIs works well overall, but problems arise precisely in places where the operation is formally the same (copy/move), but the semantics are already diverging across providers. Google Drive, Dropbox, iCloud—each has its own nuances: in some cases, it's essentially a server-side copy, in others, it's practically a "re-upload under the hood," in others, there are limitations on shared drives, deduplication, or metadata.