It seems silent corruption occured on my data stored on GDrive. One of the file failed to decrypt after 583073792 byte into the file. I downloaded this file directly from GDrive and have tried to locally decrypt it. rclone cat generates a truncated file whereas rclone copyto just quits when it can't get past the corruption. Is there a way to force rclone to push past the corrupted bits and greedily decrypt the rest? It's a video so loosing a few frames won't be too bad...
Run the command 'rclone version' and share the full output of the command.
v1.63.1
Which cloud storage system are you using? (eg Google Drive)
GDrive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
I think the copyto command shows more clearly where the rot took place...Maybe?
rclone copyto -vv --config=./localconfig/temp.conf --crypt-pass-bad-blocks localvideos:"Wedding.avi" file2.avi
2023/08/20 13:23:15 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "copyto" "-vv" "--config=./localconfig/temp.conf" "--crypt-pass-bad-blocks" "localvideos:Wedding.avi" "file2.avi"]
2023/08/20 13:23:15 DEBUG : Creating backend with remote "localvideos:Wedding.avi"
2023/08/20 13:23:15 DEBUG : Using config file from "/home/user/eEternalStores/Multimedia/bitrot/localconfig/temp.conf"
2023/08/20 13:23:15 DEBUG : localvideos: detected overridden config - adding "{hu89K}" suffix to name
2023/08/20 13:23:15 DEBUG : Creating backend with remote "/home/user/eEternalStores/Multimedia/bitrot/localcrypt/92bd8ahs1n8gvtjrlg6n2unjlo"
2023/08/20 13:23:15 DEBUG : fs cache: adding new entry for parent of "/home/user/eEternalStores/Multimedia/bitrot/localcrypt/92bd8ahs1n8gvtjrlg6n2unjlo", "/home/user/eEternalStores/Multimedia/bitrot/localcrypt"
2023/08/20 13:23:15 DEBUG : fs cache: adding new entry for parent of "localvideos:Wedding.avi", "localvideos{hu89K}:Wedding.avi"
2023/08/20 13:23:15 DEBUG : Creating backend with remote "."
2023/08/20 13:23:15 DEBUG : fs cache: renaming cache item "." to be canonical "/home/user/eEternalStores/Multimedia/bitrot"
2023/08/20 13:23:15 DEBUG : Wedding.avi: Need to transfer - File not found at Destination
2023/08/20 13:23:15 DEBUG : Wedding.avi: Starting multi-thread copy with 3 parts of size 269.688Mi
2023/08/20 13:23:15 DEBUG : Wedding.avi: multi-thread copy: stream 3/3 (565575680-848233878) size 269.564Mi starting
2023/08/20 13:23:15 DEBUG : Wedding.avi: multi-thread copy: stream 2/3 (282787840-565575680) size 269.688Mi starting
2023/08/20 13:23:15 DEBUG : Wedding.avi: multi-thread copy: stream 1/3 (0-282787840) size 269.688Mi starting
2023/08/20 13:23:15 DEBUG : Wedding.avi: multi-thread copy: write buffer set to 128Ki
2023/08/20 13:23:15 DEBUG : Wedding.avi: multi-thread copy: write buffer set to 128Ki
2023/08/20 13:23:15 DEBUG : Wedding.avi: multi-thread copy: write buffer set to 128Ki
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:16 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:17 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/20 13:23:33 DEBUG : Wedding.avi: multi-thread copy: stream 2/3 (282787840-565575680) size 269.688Mi finished
2023/08/20 13:23:33 DEBUG : Wedding.avi: multi-thread copy: stream 3/3 (565575680-848233878) size 269.564Mi finished
2023/08/20 13:23:33 DEBUG : Wedding.avi: multi-thread copy: stream 1/3 (0-282787840) size 269.688Mi finished
2023/08/20 13:23:33 DEBUG : Wedding.avi: Finished multi-thread copy with 3 parts of size 269.688Mi
2023/08/20 13:23:33 DEBUG : file2.avi.tumuqix4.partial: renamed to: file2.avi
2023/08/20 13:23:33 INFO : Wedding.avi: Multi-thread Copied (new) to: file2.avi
2023/08/20 13:23:33 INFO :
Transferred: 808.939 MiB / 808.939 MiB, 100%, 41.154 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 17.8s
2023/08/20 13:23:33 DEBUG : 6 go routines active
Crypted file was 848441014 bytes while decrypted file is 848233878 bytes, so maybe lost of 207136 bytes?
[user@yellowtrain bitrot]$ ls
total 1656728
-rw-r--r--. 1 user 848233878 Aug 20 12:36 file
-rw-r--r--. 1 user 848233878 Aug 19 23:49 file2.avi
drwxr-xr-x. 2 user 4096 Aug 20 12:33 localconfig/
drwxr-xr-x. 2 user 4096 Aug 20 02:16 localcrypt/
[user@yellowtrain bitrot]$ ls localcrypt/
total 828560
-rw-r--r--. 1 user 848441014 Aug 19 23:49 92bd8ahs1n8gvtjrlg6n2unjlo
ERROR : crypt: ignoring: failed to authenticate decrypted block - bad
When I say lose - that 64k will be set to all zeros so you can see them in the output with a hex editor. I count 48 of them which is 3MB. So you'll find there is a 3 MB chunk of zeroes in your video. Most video players will cope with that kind of thing just find, so you'll have a glitch with a few seconds missing most likely.
You might want to try copying it with --multi-thread-streams 0 also - that might not make a difference, but it might so it is worth a try.
were you ever able to read the file of gdrive? If yes, maybe then its bitro (though I have my doubts), but if this is your first time reading it, I'd hazzard to say, it might be more likely to be a problem on your side (i.e. memory corruption or the like at the time of copying to gdrive) and might want to check out the system that was used to copy to gdrive to see if there are any hardware problems on it.
multi-thread-stream 0 does not make a difference. But doesn't matter since the recovered video is sufficiently intact, which is very good for me. Thanks
rclone copyto -vv --config=./localconfig/temp.conf --crypt-pass-bad-blocks --multi-thread-streams 0 localvideos:"Wedding.avi" file3.avi
2023/08/22 06:11:08 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "copyto" "-vv" "--config=./localconfig/temp.conf" "--crypt-pass-bad-blocks" "--multi-thread-streams" "0" "localvideos:Wedding.avi" "file3.avi"]
2023/08/22 06:11:08 DEBUG : Creating backend with remote "localvideos:Wedding.avi"
2023/08/22 06:11:08 DEBUG : Using config file from "/home/user/eEternalStores/Multimedia/bitrot/localconfig/temp.conf"
2023/08/22 06:11:08 DEBUG : localvideos: detected overridden config - adding "{hu89K}" suffix to name
2023/08/22 06:11:08 DEBUG : Creating backend with remote "/home/user/eEternalStores/Multimedia/bitrot/localcrypt/92bd8ahs1n8gvtjrlg6n2unjlo"
2023/08/22 06:11:08 DEBUG : fs cache: adding new entry for parent of "/home/user/eEternalStores/Multimedia/bitrot/localcrypt/92bd8ahs1n8gvtjrlg6n2unjlo", "/home/user/eEternalStores/Multimedia/bitrot/localcrypt"
2023/08/22 06:11:08 DEBUG : fs cache: adding new entry for parent of "localvideos:Wedding.avi", "localvideos{hu89K}:Wedding.avi"
2023/08/22 06:11:08 DEBUG : Creating backend with remote "."
2023/08/22 06:11:08 DEBUG : fs cache: renaming cache item "." to be canonical "/home/user/eEternalStores/Multimedia/bitrot"
2023/08/22 06:11:08 DEBUG : Wedding.avi: Need to transfer - File not found at Destination
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:11 ERROR : crypt: ignoring: failed to authenticate decrypted block - bad password?
2023/08/22 06:11:12 DEBUG : file3.avi.vujurup6.partial: renamed to: file3.avi
2023/08/22 06:11:12 INFO : Wedding.avi: Copied (new) to: file3.avi
2023/08/22 06:11:12 INFO :
Transferred: 808.939 MiB / 808.939 MiB, 100%, 192.479 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 4.3s
2023/08/22 06:11:12 DEBUG : 6 go routines active
Of course, the video was able to be played off gdrive with some frames missing. It's been about 5 years since I stored this particular video on GDrive. I know it's bit rot because I remember doing a thorough checksum of this particular file and recalled that they were identical. It's just too bad that at the time I wasn't in the habit of making and keeping local parity archives of things I stored on GDrive, which is usually way smaller than the source. This is a relatively rare incident though as out of the 9 TB or so that I had stored only this file had any bit rot. So take note and always keep some parity archives of things you hold to be precious in the case some corrupted bytes need to be reconstructed.
2023/08/19 10:23:14 INFO : Starting bandwidth limiter at 30Mi Byte/s
2023/08/19 10:23:14 INFO : Starting transaction limiter: max 12 transactions/s with burst 1
2023/08/19 10:23:14 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "ls" "tresplas_Collections_Videos:/Events/Weddings" "-vv" "--transfers" "12" "--checkers" "12" "--tpslimit" "12" "--fast-list" "--checksum" "--bwlimit" "30M:30M" "--progress" "--rc" "--rc-addr=localhost:6000" "--ask-password=false"]
2023/08/19 10:23:14 NOTICE: Serving remote control on http://127.0.0.1:6000/
2023/08/19 10:23:14 DEBUG : Creating backend with remote "tresplas_Collections_Videos:/Events/Weddings"
2023/08/19 10:23:14 DEBUG : Using RCLONE_CONFIG_PASS password.
2023/08/19 10:23:14 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2023/08/19 10:23:14 DEBUG : Creating backend with remote "tresplas:/Collections/Videos/m8s85ibih5pvifrgl8rkr0se04/a1h43c7nqc3jkt0ttr3eqdv88k"
2023/08/19 10:23:15 DEBUG : fs cache: renaming cache item "tresplas:/Collections/Videos/m8s85ibih5pvifrgl8rkr0se04/a1h43c7nqc3jkt0ttr3eqdv88k" to be canonical "tresplas:Collections/Videos/m8s85ibih5pvifrgl8rkr0se04/a1h43c7nqc3jkt0ttr3eqdv88k"
2023/08/19 10:23:15 DEBUG : fs cache: switching user supplied name "tresplas:/Collections/Videos/m8s85ibih5pvifrgl8rkr0se04/a1h43c7nqc3jkt0ttr3eqdv88k" for canonical name "tresplas:Collections/Videos/m8s85ibih5pvifrgl8rkr0se04/a1h43c7nqc3jkt0ttr3eqdv88k"
848441014 Wedding.avi
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1.2s
2023/08/19 10:23:15 DEBUG : 8 go routines active
I already vacated most of the storage on GDrive, so I can not use rclone to download. When I tried to use Google's restoration service, this particular file was not found within the restored set.
What I did was I went to GDrive and downloaded the file directly from the web interface. So the process should have copied the crypted file as is, warts and all. This is what I did to prepare the file for local decryption.
just wanted to see if google's idea of the MD5SUM was different from the actual MD5SUM of the file (including the bit-rot). That would tell us something about how google look after your files. But if it isn't on google drive any more we can't do that!
Interesting. I wonder if that was because it had bit-rotted and the exporter noticed.