Chunked uploads to Google Drive fail for Shared Drives

Suspected Bug
1

What is the problem you are having with rclone?

Large files uploading to Google Drive that trigger the chunk (multipart) upload are failing to actually save. Increasing the chunk trigger amount so the file is uploaded in a single POST works (until you get really big files). Chunked uploads to MyDrive work fine, just not shared drives.

Run the command 'rclone version' and share the full output of the command.

Reproducible Linux and Windows

rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.13.0-27-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive - Shared Drives

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# This fails (Shared Drive)
rclone --config rclone.conf --drive-upload-cutoff=1M --drive-stop-on-upload-limit --progress --log-file "debug_chunks.log" -vv --dump responses sync test/ 'Engage-and-Equip-Fundraising:/testing/test/'
# This works (MyDrive)
rclone --config rclone.conf --drive-upload-cutoff=1M --drive-stop-on-upload-limit --progress --log-file "debug_chunks_mydrive.log" -vv --dump responses sync test/ 'TimGoogleDrive:/testing/test/'

The rclone config contents with secrets removed.

[Engage-and-Equip-Fundraising]
type = drive
scope = drive
service_account_file = SAFILE.json
impersonate = google-sync-service@ourdomain.com
team_drive = 0AMBrU9Ir5R6PUk9PVA
root_folder_id =
client_id = XXXXX (own client ID and secret)
client_secret = XXXXX

[TimGoogleDrive]
type = drive
scope = drive
token = XXXX

A log from the command with the -vv flag

2022/02/03 08:32:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/03 08:32:20 DEBUG : HTTP REQUEST (req 0xc000796200)
2022/02/03 08:32:20 DEBUG : POST /upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Content-Length: 223
Authorization: XXXX
Content-Type: application/json; charset=UTF-8
X-Upload-Content-Length: 14793506
X-Upload-Content-Type: application/zip
Accept-Encoding: gzip

2022/02/03 08:32:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/03 08:32:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/03 08:32:20 DEBUG : HTTP RESPONSE (req 0xc000796200)
2022/02/03 08:32:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: text/plain; charset=utf-8
Date: Thu, 03 Feb 2022 00:32:20 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Location: https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdvb0F24O9Kx1hKzxhHm41CBDjy09bpLK1iF_W7t_X3RY_xdgJK-x2tqHfxmaeueSFmMr6ytuNbKhXET3iPEysJmRcNoZw
Pragma: no-cache
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Guploader-Uploadid: ADPycdvb0F24O9Kx1hKzxhHm41CBDjy09bpLK1iF_W7t_X3RY_xdgJK-x2tqHfxmaeueSFmMr6ytuNbKhXET3iPEysJmRcNoZw

2022/02/03 08:32:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/03 08:32:20 DEBUG : rclone-v1.56.2-windows-amd64.zip: Sending chunk 0 length 8388608
2022/02/03 08:32:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/03 08:32:20 DEBUG : HTTP REQUEST (req 0xc0005fa900)
2022/02/03 08:32:20 DEBUG : POST /upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdvb0F24O9Kx1hKzxhHm41CBDjy09bpLK1iF_W7t_X3RY_xdgJK-x2tqHfxmaeueSFmMr6ytuNbKhXET3iPEysJmRcNoZw HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Content-Length: 8388608
Authorization: XXXX
Content-Range: bytes 0-8388607/14793506
Content-Type: application/zip
Accept-Encoding: gzip

2022/02/03 08:32:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/03 08:32:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/03 08:32:25 DEBUG : HTTP RESPONSE (req 0xc0005fa900)
2022/02/03 08:32:25 DEBUG : HTTP/1.1 308 Resume Incomplete
Content-Length: 0
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Content-Type: text/plain; charset=utf-8
Date: Thu, 03 Feb 2022 00:32:25 GMT
Range: bytes=0-8388607
Server: UploadServer
X-Guploader-Uploadid: ADPycdvb0F24O9Kx1hKzxhHm41CBDjy09bpLK1iF_W7t_X3RY_xdgJK-x2tqHfxmaeueSFmMr6ytuNbKhXET3iPEysJmRcNoZw
X-Range-Md5: 3277fb8f87fdc369953c802d656b1314

2022/02/03 08:32:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/03 08:32:25 DEBUG : rclone-v1.56.2-windows-amd64.zip: Sending chunk 8388608 length 6404898
2022/02/03 08:32:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/03 08:32:25 DEBUG : HTTP REQUEST (req 0xc000796200)
2022/02/03 08:32:25 DEBUG : POST /upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdvb0F24O9Kx1hKzxhHm41CBDjy09bpLK1iF_W7t_X3RY_xdgJK-x2tqHfxmaeueSFmMr6ytuNbKhXET3iPEysJmRcNoZw HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Content-Length: 6404898
Authorization: XXXX
Content-Range: bytes 8388608-14793505/14793506
Content-Type: application/zip
Accept-Encoding: gzip

2022/02/03 08:32:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/03 08:32:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/03 08:32:29 DEBUG : HTTP RESPONSE (req 0xc000796200)
2022/02/03 08:32:29 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 306
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Content-Type: application/json; charset=UTF-8
Date: Thu, 03 Feb 2022 00:32:29 GMT
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Guploader-Uploadid: ADPycdvb0F24O9Kx1hKzxhHm41CBDjy09bpLK1iF_W7t_X3RY_xdgJK-x2tqHfxmaeueSFmMr6ytuNbKhXET3iPEysJmRcNoZw

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "File not found: 1aJTtUYkh2iG3RCIW8UO6Yv4GbOVtruIR.",
    "locationType": "parameter",
    "location": "fileId"
   }
  ],
  "code": 404,
  "message": "File not found: 1aJTtUYkh2iG3RCIW8UO6Yv4GbOVtruIR."
 }
}
2022/02/03 08:32:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/03 08:32:29 ERROR : rclone-v1.56.2-windows-amd64.zip: Failed to copy: googleapi: Error 404: File not found: 1aJTtUYkh2iG3RCIW8UO6Yv4GbOVtruIR., notFound

I'm wondering if this is another Google API issue and not rclone.

2

Can you share a full debug log from the start? The default cutoff is 8M so that would mean anyone uploading anything bigger than 8M would not work to a shared drive.

Just pick one file and copy that / share the full debug log.

3

With upload bodies? That becomes a rather large file, and with the default retry it ends up at 40Mb+. Or is --dump responses ok you just want from earlier in the debug log?
If you do want a large file, is pastebin the best way to share it with you guys? (The file contents isn't sensitive, for testing I actually used the rclone zip file).

4

Nah, no bodies yet, I just want to see a full log and see if I spot anything.

You can use any file you want.

https://jell.yfish.us/media/jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv is a nice test file I use quite a bit.

5

Phew, I didn't think you'd want to scroll through 40Mbs of upload bodies. Here is the complete debug log without auth tokens.

And Successful chunked upload mydrive - Pastebin.com is the chunked upload that works to MyDrive. They were too big for the char limit on a post.

6

Sorry as I was just looking for debug and no dumps. I’ll take a look through.

1 Like
7

This must be something to do with your setup as we haven't had any reports of this I can remember and multipart uploads are really common!

I looked through your log and what appears to happen is that Drive accepts the first chunk, tells rclone to carry on, then on uploading the second chunk returns "File not found" as if the partially uploaded file just disappeared.

I conjecture there is some permissions problem where the uploading user can't read files in the directory you are uploading to so can't find the partially uploaded file to add the next chunk to.

What permissions does your user have? Can you try giving it more permissions?

8

The pattern is shared drive, service account, impersonation between both the fast list issue and uploading.

If chunked uploaded on shared drives was broken, there would be many, many reports of that.

Can you try without the service account use a regular user? See if that has the same issue?

Does you shared drive have more than 400k files or something?

9

I've just come back to debugging and had come to the same conclusion as you, so I did more debugging trying to find what the common issue was, was it the SA or the impersonation or something else.... So apparently you can impersonate a disabled user and most things work. :man_facepalming:
One our automated scripts disabled my user that I impersonate (our "service" user) 2 weeks ago, the only things that broke were fast-list and chunked uploads (somewhere between 250Mb and 300Mb seemed to be the limit where it broke).

Thanks for all your help with debugging things. Now to see if Google thinks its a bug that most things work for a disabled user that you can impersonate.

1 Like
10

Neat. Thanks for sharing as that's a heck of an issue and a very odd set of broken items.

Glad you found it!

11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.