`rclone md5sum` on a large Encrypted GDrive dirtree is generating multiple errors

[durval (Durval Menezes)]

@ncw

Do these files (the ones that do the same thing every time or do they only sometimes give the corrupted error?

I'm trying to avoid doing other things with this remote in parallel to this rclone md5sum to so as not to 'rock the boat', but I just did an rclone cat for some of these files and they all aborted with the same exact error -- so these are probably permanent errors.

• file corrupted on drive - in which case it would do the exact same thing every time

This is looking more and more to be the case.

And all these files were checked when they were first put into Google Drive, about 6 years ago -- therefore the corruption happened while they were stored in Google Drive.

It seems Google Drive has been quietly 'eating' my data... haven't destroyed much so far ('only' 19 files in the ~6.5M md5sum'ed so far is about 0.0002%), but this is one more incentive to get my data the F out of there ASAP.

I don't know what that (Bad Request, failedPrecondition) means! Did a file change?

I was pretty sure this particular file has not changed since I stored it many years ago, and I'm even more sure now after checking the file's modTime in the subjacent remote:

rclone cryptdecode --reverse ENCRYPTED_GOOGLE_DRIVE: REDACTED20/REDACTED21/REDACTED22.JPG
    REDACTED20/REDACTED21/REDACTED22.JPG        REDACTED101/REDACTED102/REDACTED103

rclone lsl GOOGLE_DRIVE:REDACTED10/REDACTED101/REDACTED102/REDACTED103
       8743504 2017-07-09 16:02:03.000000000 REDACTED103

That is, Google Drive itself reports the (base) file as unmodified since 2017... so we can affirm that the file has not changed during the execution of the above rclone md5sum that reported this failedPrecondition error.

The good news is that I just tried calculating that file's data again:

rclone md5sum --download ENCRYPTED_GOOGLE_DRIVE: REDACTED20/REDACTED21/REDACTED22.JPG
    9de005985fda7216b4985d2a37f57f90  REDACTED22.JPG

So the error isn't permanent. Perhaps just some piece of crap stuck in one of Google's 'tubes' but has since then got loose?

@ncw

That (Invalid Credentials, authError) is probably a race condition between the token renewer and the hashing. Could be clock skew between you and Google maybe?

Clock skew is very improbable on my end, as:

1. the machine runs ntpd
2. This ntpd has been in sync since December 10th; from this time and the time the error occurred (Feb 19th) there were no logs of anything going on with ntpd, so it almost certainly remained in sync (yes, I log and save everything that happens at this machine at syslog level DEBUG).

Therefore the only chance for any clock skew would be in Google's end -- which I don't think is significantly probable either.

The good news is, just like the other single error above, this one also seems to have been transient:

rclone md5sum --download ENCRYPTED_GOOGLE_DRIVE:REDACTED30/REDACTED31/REDACTED32/REDACTED33/REDACTED34/REDACTED35/REDACTED36/REDACTED37/REDACTED38.pdf
     0fe0e04974dcf14a65e3e67375a1fa84  REDACTED38.pdf

So, yay! It seems Google has indeed only eaten 19 of my files so far
(all of which I also have on two different local disks -- so they can easily be restored)

When this is over, I will come back and report a final tally (and also post a "PSA: Google Drive is silently corrupting files!" topic to warn the unwary).