OneDrive sync never completes with SHA-1 corrupted transfer with latest rclone

$ rclone copy OneDriveSweharris2:Backups/penfold/20220403._FastData.0.gz_ag\ 1.bin .
$ rclone copy OneDriveSweharris2:Backups/penfold/20220403._FastData.0.gz_ag\ \(1\).bin .
$ rclone copy OneDriveSweharris2:Backups/penfold/20220403._FastData.0.gz_ag\ \(2\).bin .
$ sha1sum *
5008bfd43fa526566d221897d1d6bf7cbf2096f2  20220403._FastData.0.gz_ag (1).bin
c33bffe167ee08f8d85d621dce5d776387a41745  20220403._FastData.0.gz_ag (2).bin
1741774b18a839226697804c6b7e4186ac6ccbdb  20220403._FastData.0.gz_ag 1.bin

We can see these one match the errors

2022/04/03 13:24:55 ERROR : penfold/20220403._FastData.0.gz_ag: Failed to copy: corrupted on transfer: sha1 crypted hash differ "fe9c8b9ebaf512cac3a8ef74bcf5b5a5e28981ab" vs "5008bfd43fa526566d221897d1d6bf7cbf2096f2"

2022/04/03 12:45:19 ERROR : penfold/20220403._FastData.0.gz_ag: Failed to copy: corrupted on transfer: sha1 crypted hash differ "e43bee9ac8c35a64b7cc3c1ad62ac3014b12841c" vs "c33bffe167ee08f8d85d621dce5d776387a41745"

2022/04/03 13:35:59 ERROR : penfold/20220403._FastData.0.gz_ag: Failed to copy: corrupted on transfer: sha1 crypted hash differ "d976cd211051bbb71961716aa6786f98c6bfcd71" vs "1741774b18a839226697804c6b7e4186ac6ccbdb"

So Microsoft's hashes appear to be correct.

The files are all of the right size (including the "good" version for comparison).

-rw-r--r-- 1 sweh sweh 5370019872 Apr  3 03:09 20220403._FastData.0.gz_ag (1).bin
-rw-r--r-- 1 sweh sweh 5370019872 Apr  3 03:09 20220403._FastData.0.gz_ag (2).bin
-rw-r--r-- 1 sweh sweh 5370019872 Apr  3 03:09 20220403._FastData.0.gz_ag 1.bin
-rw-r--r-- 1 sweh sweh 5370019872 Apr  3 03:09 20220403._FastData.0.gz_ag.bin

So I ran strings on 20220403._FastData.0.gz_ag 1.bin and looked for any string longer than 30 characters long (grep .....). And... well...

C6jzLl8yC0Necw7InUZ9k9vBtR/nwXJd3BPxqpN1mBZ1vY88z2XEnyGn9kbcDxPdLXUIr9OwICiNcf3dWPiPZ1/klN/AvP8n28
....
....
....
ps7zhrsWCdht11NRF2gLHqunFhPL5Fwhm6kbKGwg39e24TZQ++q2xjVIf5pRdeX8FNt7KQceFNZ/79K+G  

That block is around 6934 bytes long all in one long line. It was at offset 248199391.

Similarly 20220403._FastData.0.gz_ag (1).bin had, at offset 16753810, a chunk of around 9961 characters, and 20220403._FastData.0.gz_ag (2).bin had, at offset 252537251, a chunk of 5860 bytes.

I have no idea if there were low level retries. But this looks like it might either have been doubly-encoded base64 by rclone, or else not decoded by OneDrive.

1 Like