Copy files over 8Mb to Google Drive fail

DavideCarando · April 15, 2023, 8:01pm

Hello, when I try copy file over 8Mb to Google Drive fail while all files under 8Mb coping fine.
I use rclone version 1.62.2.0 on windows machine and the internet connection is behind proxy server with basic authentication without ssl inspection.
Thank you very much for the support.

rclone v1.62.2

os/version: Microsoft Windows Server 2016 Standard 1607 (64 bit)
os/kernel: 10.0.14393.1884 Build 14393.1884.1884 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: cmount

rclone.exe copy c:\tools\rclone\dati\9M.txt Drive:

[Drive]
type = drive
scope = drive
service_account_file = c:/tools/rclone/dotted-signer-xxxxxxxx.json
team_drive =

C:\Tools\rclone>rclone.exe -vv copy c:\tools\rclone\dati\9M.txt Drive:
2023/04/15 21:33:14 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone.exe" "-vv" "copy" "c:\\tools\\rclone\\dati\\9M.txt" "Drive:"]
2023/04/15 21:33:14 DEBUG : Creating backend with remote "c:\\tools\\rclone\\dati\\9M.txt"
2023/04/15 21:33:14 DEBUG : Using config file from "C:\\Tools\\rclone\\rclone.conf"
2023/04/15 21:33:14 DEBUG : fs cache: adding new entry for parent of "c:\\tools\\rclone\\dati\\9M.txt", "//?/c:/tools/rclone/dati"
2023/04/15 21:33:14 DEBUG : Creating backend with remote "Drive:"
2023/04/15 21:33:14 DEBUG : Drive: detected overridden config - adding "{rAyzR}" suffix to name
2023/04/15 21:33:14 DEBUG : fs cache: renaming cache item "Drive:" to be canonical "Drive{rAyzR}:"
2023/04/15 21:33:15 DEBUG : 9M.txt: Need to transfer - File not found at Destination
2023/04/15 21:33:16 DEBUG : 9M.txt: Sending chunk 0 length 8388608
2023/04/15 21:33:20 ERROR : 9M.txt: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdvj79HhxsEmRpskTnBR-fEAdjgHbZy1q_pYzuTqLXfavI6k4J5HlxTJncW4PYszWcQebeRVum_GD4_NcQQWsMyA76YGyJ05": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"
2023/04/15 21:33:20 ERROR : Attempt 1/3 failed with 1 errors and: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdvj79HhxsEmRpskTnBR-fEAdjgHbZy1q_pYzuTqLXfavI6k4J5HlxTJncW4PYszWcQebeRVum_GD4_NcQQWsMyA76YGyJ05": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"
2023/04/15 21:33:20 DEBUG : 9M.txt: Need to transfer - File not found at Destination
2023/04/15 21:33:21 DEBUG : 9M.txt: Sending chunk 0 length 8388608
2023/04/15 21:33:27 ERROR : 9M.txt: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycduVY8Rh98mkrhNrrpUcH00rNkAjsEq-_Gf-3hsEPc88RYVh43FLDx7WsC9A55Hf0-pWnEd2ICxoXg30PwSjUTylgOtUYCCN": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"
2023/04/15 21:33:27 ERROR : Attempt 2/3 failed with 1 errors and: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycduVY8Rh98mkrhNrrpUcH00rNkAjsEq-_Gf-3hsEPc88RYVh43FLDx7WsC9A55Hf0-pWnEd2ICxoXg30PwSjUTylgOtUYCCN": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"
2023/04/15 21:33:28 DEBUG : 9M.txt: Need to transfer - File not found at Destination
2023/04/15 21:33:29 DEBUG : 9M.txt: Sending chunk 0 length 8388608
2023/04/15 21:33:33 ERROR : 9M.txt: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdtlHukn0uamBKARhSYnqB-6UYhxegjZTxrUzNp7XjfT2IkAB08HkkNSJcPLMKKN4cRCsrFR9S4PQ4IyNsrZXf7oUVcP33oP": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"
2023/04/15 21:33:33 ERROR : Attempt 3/3 failed with 1 errors and: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdtlHukn0uamBKARhSYnqB-6UYhxegjZTxrUzNp7XjfT2IkAB08HkkNSJcPLMKKN4cRCsrFR9S4PQ4IyNsrZXf7oUVcP33oP": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"
2023/04/15 21:33:33 INFO  :
Transferred:           24 MiB / 24 MiB, 100%, 1.327 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:        19.3s

2023/04/15 21:33:33 DEBUG : 5 go routines active
2023/04/15 21:33:33 Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdtlHukn0uamBKARhSYnqB-6UYhxegjZTxrUzNp7XjfT2IkAB08HkkNSJcPLMKKN4cRCsrFR9S4PQ4IyNsrZXf7oUVcP33oP": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\xc0\xe3\xff\xff\xff\x7f\x00\x00n/J\x00\x00\x00\x00\x00"

Animosity022 · April 15, 2023, 8:02pm

Looks like your proxy is not working as expected. I'd explore that.

DavideCarando · April 15, 2023, 8:18pm

I've already checked the proxy but it seems to work fine even because if I increase the chunk size using --drive-chunk-size flag the copy the copy working fine.

Animosity022 · April 15, 2023, 8:42pm

The proxy in the middle is most definitely the problem as if you test somewhere else, you'll see it works fine.

DavideCarando · April 15, 2023, 8:49pm

I believe so but what could I check specifically?

Animosity022 · April 15, 2023, 8:54pm

What proxy server is it? Check the logs on it for any issues?

DavideCarando · April 16, 2023, 9:41am

The proxy is Cisco Secure Web Appliance, on the logs there aren't issues about rclone activity.

ncw · April 17, 2023, 11:32am

There is something about the HTTP transaction rclone is doing with the proxy that it doesn't understand.

Can you report a bug to them?

ncw · April 17, 2023, 3:19pm

The request which is upsetting the proxy is this one

2023/04/17 16:31:46 DEBUG : HTTP REQUEST (req 0xc000141700)
2023/04/17 16:31:46 DEBUG : POST /upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdv2iFm_ne9992XDkI2F_aYfOduIXyHWS8-NayEitk3KZoSPi2f6qVg6aShaB_d-s7yLQcYeIbrD7qXuqJMw7kKAUTMfUmOJ HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.62.2
Content-Length: 8388608
Authorization: XXXX
Content-Range: bytes 16777216-25165823/30000000
Content-Type: text/plain; charset=utf-8
Accept-Encoding: gzip

It looks like it gets out of sequence in its HTTP 1.1 streaming, maybe.

You could try experimenting with these flags (one at a time) to see if you can workaround it

  --disable-http-keep-alives   Disable HTTP keep-alives and use each connection once.
  --disable-http2              Disable HTTP/2 in the global transport

DavideCarando · April 17, 2023, 7:07pm

As you suggest I have try use --disable-http-keep-alives and --disable-http2 flags one at a time but the result is always same.

Animosity022 · April 17, 2023, 7:17pm

You'd be best to work with the vendor to see what's going on as it's a proxy issue.

ncw · April 18, 2023, 10:13am

Ok it was worth a try.

I think you'll have to take this up with the vendor.

DavideCarando · April 18, 2023, 12:33pm

We have verified that the problem was caused by ssl proxy inspection. Removing it solved the problem.
Thank you very much at all for the support.

"If you are using any form of web filtering this traffic needs to be de-crypted and re-encrypted on a edge security device if you want to peak inside the packet contents.
Google recommend that you bypass packet level inspect on data heading to Google and publish a list of site that need to be white listed."

system · April 21, 2023, 12:33pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.