Google Drive Potential Bug - Unlimited Duplicates

Suspected Bug
1

Hi,

Bit of an odd one. I’m using rclone for the first time to backup Google drive to Backblaze b2.

I run it with this:

rclone sync gdrive:Photos backblaze:JamesFiles/Photos --copy-links --log-level info

The backup seems to be fine but seems to be creating unlimited duplicates. When I run it I get something like this:

2017/08/03 07:23:28 INFO : 2016/IMG_1561.JPG: Copied (replaced existing)
2017/08/03 07:23:35 INFO : 2016/IMG_1563.JPG: Copied (new)
2017/08/03 07:23:36 INFO : 2016/IMG_1628.JPG: Copied (new)
2017/08/03 07:23:39 INFO : 2016/IMG_1670.JPG: Copied (new)
2017/08/03 07:23:41 INFO : 2016/IMG_1670.JPG: Copied (replaced existing)
2017/08/03 07:23:47 INFO : 2016/IMG_1565.JPG: Copied (replaced existing)
2017/08/03 07:23:48 INFO : 2016/IMG_1572.JPG: Copied (new)
2017/08/03 07:23:50 INFO : 2016/IMG_1628.JPG: Copied (replaced existing)
2017/08/03 07:23:51 INFO : 2016/IMG_1720.JPG: Copied (new)
2017/08/03 07:23:54 INFO : 2016/IMG_1720.JPG: Copied (replaced existing)

When I look at these files in backblaze, for each of these there are duplicates, a lot of them.

The source of the bug is that in google drive there are two files with the same name in the same folder (IMG_1670.JPG) for example, and so this duplication only occurs in those. For situations when there are two files with the same name in Google Drive in the same folder, Rclone seems to keep copying more and more duplicates to backblaze. So far I have about 15 of each.

This infinite loop of transfer to the destination only happens when there are two files in the source with the same name in the same folder, even if they are different photos.

So, is there a way to stop this? The photos are different, despite having the same name, so should have different hashes.

2

Sorry two more points. Using the command rclone dedupe backblaze:JamesFiles/Photos does nothing (just finishes within 0.5 seconds, no messages).

In addition my version is: 1.36

3

Do a rclone dedupe on the Google drive that should fix the source of the issue.

You can use the rename option to make them have different names.

B2 can’t have two files with the same name so you have to fix the source.

4

Hi NCW,

Thank you for the reply. Just a quick follow up question. Is this an issue purely with B2, if I moved to something like S3 would I run into this issue?

I only ask because this is the “Google Photos” folder in drive and I’ve been recommended not to rename the photos within this folder, as then it could cause issues within photos.google.com

5

It is an issue with drive - no other cloud storage systems support duplicate names so you’ll have this problem copying it anywhere, even to your local disk.

Where did you see that recommendation? I’ve deduped my “Google Photos” directory without problems, though I didn’t have too many duplicates.