I’ve successfully finished copying my first 8TB or so of local data to Google Drive (on my unlimited academic account) and now I’m in the process of verifying it.
This data is divided in volumes (which locally are entire hard-disks, and remotely are top-level directories on GDrive), and for each volume I have generated a local .md5 file (via the traditional “find … -type f -print0 | xargs md5sum” method), and I’m using it to check the remote GDrive volume by accessing it locally over an “rclone mount” and then running “md5sum -c” on it, like this:
mkdir egd rclone --low-level-retries=100 mount --no-modtime --no-seek --read-only egd: egd & (cd egd/VOLUME; md5sum -c /tmp/VOLUME.md5)
This mostly works, but for about 0.01% of the files, I get “Input/Output” errors, like these:
md5sum: ./REDACTED/REDACTED.bni: Input/output error ./REDACTED/REDACTED.bni: FAILED open or read
If I simultaneously check (on another interactive shell) that same file that gave the error above, using “rclone cat”, it always succeeds:
rclone cat egd:VOLUME/REDACTED/REDACTED.bni | md5sum - fc2751e43fb21aa12156322643c0491a -
And, the checksum is correct:
grep REDACTED/REDACTED.bni /tmp/VOLUME.md5 fc2751e43fb21aa12156322643c0491a ./REDACTED/REDACTED.bni
So, it seems that whatever’s happening is some sort of “transient” problem with the rclone mount…
Are any of you seeing similar issues? What are the recommended options for “rclone mount” for such a use case, in order to minimize those errors (I’m not specially concerned about speed here).