Check following sync from Google Drive keeps producing "MD5 differ" errors

[felciano (Ramon Felciano)]

What is the problem you are having with rclone?

• I am trying to sync contents of a folder of photos in Google Drive.
• The sync works for about 99% of the files, but for several dozen it produces a MD5 differ log error.
• Running rclone check reports these same errors again.
• Strangely, nanually downloading the files from Google Drive and placing them in the destination folder still yields the MD5 differ error.

What is your rclone version (output from rclone version)

rclone v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS Catalina 10.15.4, 64-bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

For the initial cloning:

rclone -v copy gdrive:/GDriveGPhotos /Volumes/Google/GDriveGPhotos

For the checking:

rclone --log-file 2020-04-17-check-GDriveGPhotos.txt -v check gdrive:/GDriveGPhotos /Volumes/Google/GDriveGPhotos

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Here's an example from a single nested directory that highlights the issue (just showing the relevant excerpts):

2020/04/18 17:51:52 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "--log-file" "/Volumes/Google/2020-04-18-check-GDriveGPhotos-one-folder.txt" "-vv" "check" "gdrive:/GDriveGPhotos/2016/12/2016-12-31" "/Volumes/Google/GDriveGPhotos/2016/12/2016-12-31"]
2020/04/18 17:51:52 DEBUG : Using config file from "/Users/username/.config/rclone/rclone.conf"
2020/04/18 17:51:54 INFO  : Local file system at /Volumes/Google/GDriveGPhotos/2016/12/2016-12-31: Waiting for checks to finish
2020/04/18 17:51:54 DEBUG : 2016-12-31 02.18.28.mp4: MD5 = fddf4cabdd5820ab4d355837a3fa282f OK
2020/04/18 17:51:54 DEBUG : 2016-12-31 02.18.28.mp4: OK
2020/04/18 17:51:54 DEBUG : 2016-12-31 09.56.59.mp4: MD5 = a6816d3e6e146d6cfc61c6798893c714 OK
2020/04/18 17:51:54 DEBUG : 2016-12-31 09.56.59.mp4: OK
...
2020/04/18 17:51:54 DEBUG : 2016-12-31 17.53.44 (iPhone 6s Plus).jpg: MD5 = b83780716b77981b17ccc8eb1548630d (Google drive root 'GDriveGPhotos/2016/12/2016-12-31')
2020/04/18 17:51:54 DEBUG : 2016-12-31 17.53.44 (iPhone 6s Plus).jpg: MD5 = 52165d5c12847c3a9453ffa4a4fe6d10 (Local file system at /Volumes/Google/GDriveGPhotos/2016/12/2016-12-31)
2020/04/18 17:51:54 ERROR : 2016-12-31 17.53.44 (iPhone 6s Plus).jpg: MD5 differ
2020/04/18 17:51:55 DEBUG : 2016-12-31 20.51.56 (iPhone 6s Plus).jpg: MD5 = 46f651d66d1661fa59ea20cb5ddf36d7 (Google drive root 'GDriveGPhotos/2016/12/2016-12-31')
2020/04/18 17:51:55 DEBUG : 2016-12-31 20.51.56 (iPhone 6s Plus).jpg: MD5 = 96f06efc0fe94257de0a5b267e1f4c95 (Local file system at /Volumes/Google/GDriveGPhotos/2016/12/2016-12-31)
2020/04/18 17:51:55 ERROR : 2016-12-31 20.51.56 (iPhone 6s Plus).jpg: MD5 differ
...
2020/04/18 17:51:55 DEBUG : IMG_5530.jpg: MD5 = 1b54ad5c877deb750c7b6aa8dde4ad09 OK
2020/04/18 17:51:55 DEBUG : IMG_5530.jpg: OK
2020/04/18 17:51:55 NOTICE: Local file system at /Volumes/Google/GDriveGPhotos/2016/12/2016-12-31: 2 differences found
2020/04/18 17:51:55 NOTICE: Local file system at /Volumes/Google/GDriveGPhotos/2016/12/2016-12-31: 50 matching files
2020/04/18 17:51:55 Failed to check with 3 errors: last error was: 2 differences found

[calisro (Rob)]

Have you verified you don't have duplicates? Run rclone dedupe on the remote.

[felciano (Ramon Felciano)]

I do have duplicates but no for the files that are throwing the MD5 error, so I assumed those were unrelated. I'll give the dedup a try anyway -- thanks for the suggestion

[ncw (Nick Craig-Wood)]

I have a memory that we did see files that were synced by Google Photos to drive very ocassionally having the wrong MD5SUM.

[felciano (Ramon Felciano)]

Uh oh. The run did find about 15 pairs of legitimate duplicates that it removed. However it also fully deleted 2 large folders of photos from Google Drive! Here's the head of the log file:

2020/04/18 21:39:57 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "--log-file" "/Volumes/Google/2020-04-18-dedup-GDriveGPhotos.txt" "-vv" "--dedupe-mode" "rename" "dedupe" "gdrive:/GDriveGPhotos"]
2020/04/18 21:39:57 DEBUG : Using config file from "/Users/username/.config/rclone/rclone.conf"
2020/04/18 21:39:57 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2020/04/18 21:39:57 DEBUG : gdrive: Saved new token in config file
2020/04/18 21:39:58 INFO  : Google drive root 'GDriveGPhotos': Looking for duplicates using rename mode.
2020/04/18 21:41:57 INFO  : 2012b: Merging contents of duplicate directories
2020/04/18 21:41:58 INFO  : 2012b: merging "2012-11-05"
2020/04/18 21:41:59 INFO  : 2012b: merging "2012-11-10"
2020/04/18 21:41:59 INFO  : 2012b: merging "2012-11-30"
2020/04/18 21:42:00 INFO  : 2012b: merging "2012-11-18"
2020/04/18 21:42:00 INFO  : 2012b: merging "2012-11-03"

This continues, iterating through every day in the 2012b and 2013b folders. At the end of the run, neither folder is present in Google Drive anymore.

I suspect this is because these folders are originally in my Google Backup and Sync folders, where I selected the add this folder to Google Drive and it showed up under Google Drive. I note that the folders are no longer under Google Backup and Sync anymore either.

In my particular situation I'm OK as I have copies of these files elsewhere, but obviously the above behavior is destructive on the Google Drive side in a way that I don't think fits the usual semantic interpretation of "deduplication".

This is purely speculation on my part, but if the way the "add this folder to Google Drive" functions is by placing aliases in Google Drive, and those aliases are all small file pointers of the same file size, maybe they were considered duplicates and thus merged?

Please let me know how to proceed, including whether I should file a ticket in the repo. Also LMK if forum etiquette is to migrate this to a new thread since this is a separate issue from my original post.

Thanks,

Ramon

[felciano (Ramon Felciano)]

I should also highlight that I "added" these Google Backup and Sync folders to Google Drive before the recent conversion to the shortcuts model. The menu option I used (wording was something like "add this folder to Google Drive") is no longer available, and has been replaced with an "Add Shortcut to Drive" menu item.

I ran a quick test with a simple folder of files uploaded to Google Backup and Sync, and then added as an alias to Google Drive. The problem doesn't seem to manifest itself because rclone complains that the Failed to dedupe: find duplicate dirs: directory not found.

[calisro (Rob)]

Hm. Glad you have a backup. I wonder if dedupe should have ignored shortcuts by default and warn instead. @ncw

[Edward_Barker (Edward)]

Backups are great! Google Drive's trashcan might be all you need though. I would be very wary of the Google Drive dedupe in rclone 1.51. Maybe wait for the next version, or downgrade if you need to run dedupe now.

[calisro (Rob)]

@Edward_Barker

I would be very wary of the Google Drive dedupe in rclone 1.51

Why exactly? Other than the issue that he brought up.

[Edward_Barker (Edward)]

@calisro, I was routinely running rclone dedupe across my Google Drive whenever rclone sync identified duplicates and encountered issues with 1.51.0

Google Drive dedupe behaviour change?

Problem A bash script I have used daily by cron for ages which does an rclone sync google drive to local and dedupe if duplicates are found and then a second sync has suddenly found a lot of duplicates to chew over although all except one of the files had not been touched since yesterday. Until now identical named files in the same folder were distinctively named by the dedupe to make them compatible with the locally synced ext4 copy. I am away from the computer and only have access to a sort …

One issue was the sudden deletion of many files with complex identities in terms of being in multiple folders, not just duplicates in the same folder. My understanding is that the glitch leading to his arose in 1.51and has now been fixed.

Again, in my own case I had backups and the Google Drive trashcan, but have suspended using dedupe until the next release.

[calisro (Rob)]

Thanks!. I see. I'm running the beta so it's "fixed" there. @felciano anyway to recreate your issue and see if it is also fixed in the beta? I see you're running the last release instead. I wonder if you ran into the same issue.

[ncw (Nick Craig-Wood)]

@felciano

Uh oh. The run did find about 15 pairs of legitimate duplicates that it removed. However it also fully deleted 2 large folders of photos from Google Drive! H

Ouch

This is a bug due to things having multiple parents which is fixed in the latest beta - I better get that into a release ASAP.

I'm sorry about the deletions but really glad you had a backup!

Rclone (in the beta) now refuses to delete files that have multiple parents.

[felciano (Ramon Felciano)]

I haven't been able to reproduce this, likely because Google seems to have shifted to the "alias" model rather than what sounds like the "multiple parents" issue

[felciano (Ramon Felciano)]

Thanks @ncw. Glad it is a known issue and fixed. Looking forward to the new release.