I’ve been using rclone for well over a year to pull down md5sums of various directories of backup files that I have uploaded.
To this end, my procedure is:
Build a local text file containing all md5sums on the local side for a particular backup directory (using rhash), and I run this periodically to pick up updates as new backup files are created within the directory.
I use the Syncovery app which on a scheduled basis, does a sync – to mirror the local files to the remote Google Drive directory.
To see what’s at Google, I use “rclone md5sum” to download the md5sums for a particular directory on Google Drive
To assume nothing – that everything has made it’s way to Google successfully – and not to have to trust anyone else about, it I compare the two md5sum files to see if there is a difference – do all files on the local side that rhash logged exist at Google? Do all files on Google I am pulling with “rclone md5sum” exist in the local directory?
This has all worked very well for over a year now, without any notable wrinkles. However, something has changed in the past month or so. When I use:
“rclone md5sum”, it’s not pulling down all md5sums. That is, rclone is not putting out a complete list of files. Sometimes one file is missing from the list, sometimes two or three, say out of 1200 files or so.
How do I know this? Because when I go to the Google Drive web interface, the file is there, and I can download it, and the md5sum even checks out.
I’ve also tried:
“rclone ls”, and verified that the file is not making it’s way into this list.
In addition, if I use another 3rd-party tool such as “odrive”, it ALSO cannot see the file that rclone is “missing”.
But all along, both Google Drive, and Syncovery can see the file.
Now, to perhaps give a little more insight into this, if I upload another file (say via web browser) to Google Drive into that same directory, the behavior changes. Now, rclone can see BOTH this newly uploaded file as well as the one that previously was not listed on the remote end.
What happens when I delete the newly uploaded file? The old file goes “missing” again on the remote end. You can literally watch an odrive folder show the file appear / disappear.
I contacted Google, but they were generally uninterested because the problem involved 3rd party tools.
I contacted the Syncovery developer, but he didn’t really have any new insight.
Any tips here?
BTW, I am running rclone v1.34, and I have also tried the most beta version as of two days ago:
The result was the same.
My OS is OS X 10.11.6.
Syncovery version 7.68.
I must have changed one of the apps in my workflow, either that or perhaps something has changed in the Drive API, or a bug has crept in somewhere?
Any insights or things to try would be appreciated.