ERROR : test.json: corrupted on transfer: sizes differ 898 vs 283
I am setting up a backup of Cloudflare R2 (S3 compatible) to Google Drive, and I noticed right away that json files were not transferring and unexpectedly failing.
I saw that there was a similar ticket here Error about corrupted transfer and that there's a new feature in 1.60 which I just updated to https://forum.rclone.org/t/rclone-1-60-0-release/33646:
Add --s3-decompress flag to decompress gzip-encoded files (Nick Craig-Wood)
With this flag, the JSON files are now transferring, so it looks like it's a necessary solution for S3 sources, at least for Cloudflare R2. I'm posting this to
a) validate that the flag was the right thing to enable in my case
b) ponder whether perhaps it should be defaulted to on, or if rclone can autodetect when it needs to be enabled and do so without having to do it manually
c) wonder if this ticket would have been the correct solution to Error about corrupted transfer as well.
rclone version rclone v1.60.0 - os/version: opensuse-leap 15.3 (64 bit) - os/kernel: 5.19.2-x86_64-linode156 (x86_64) - os/type: linux - os/arch: amd64 - go/version: go1.19.2 - go/linking: static - go/tags: none
Cloudflare R2 (S3 compatible) -> Google Drive.
[Drive] type = drive client_id = client_secret = scope = drive token = root_folder_id = team_drive = [Cloudflare R2] type = s3 provider = Other env_auth = false access_key_id = secret_access_key = endpoint = acl = private bucket_acl = private upload_cutoff = 5G force_path_style = false
2022-10-24 16:13:04 NOTICE: test.json: Not decompressing 'Content-Encoding: gzip' compressed file. Use --s3-decompress to override 2022-10-24 16:13:06 ERROR : test.json: corrupted on transfer: sizes differ 898 vs 283 2022-10-24 16:13:06 INFO : test.json: Removing failed copy