GDrive Sync broken with fast list on large numbers of files?

What is the problem you are having with rclone?

Using Rclone to sync large numbers of files off windows file servers to Google Drive, using a services account, has issues when using --fast-list. There is an error about needing teamDriveMembershipRequired which the impersonate account has at the top level of the shared drive. This error is not present when running without --fast-list, and because of the error an "IO error" prevents a proper sync with delete. Running without --fast-list works correctly, but is obviously much slower.

The note about the clocks being wrong isn't related, the same error is occurring on other servers that have NTP sync working, this one just happens to be the one I'm debugging today and it's NTP upstream is currently broken.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19042.1466 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive, own client_id, service account with impersonation of an admin user with access to the Shared Drives

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.\rclone.exe sync "S:\AllData\Admin\Entertainment Books" "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books" --fast-list --drive-stop-on-upload-limit --progress --log-file "C:\synclogs\debug.log" --log-level DEBUG

The rclone config contents with secrets removed.

[Engage-and-Equip-Fundraising]
type = drive
scope = drive
service_account_file = services_account_file.json
impersonate = google-sync-service@ourdomain.com
team_drive = XXXXXXXX
root_folder_id =
client_id = XXXXXXX
client_secret = XXXXXX

A log from the command with the -vv flag

2022/02/01 12:53:26 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\Users\\timw\\Desktop\\rclone-ps-migration\\rclone.exe" "sync" "S:\\AllData\\Admin\\Entertainment Books" "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books" "--fast-list" "--drive-stop-on-upload-limit" "--progress" "--log-file" "C:\\synclogs\\debug.log" "--log-level" "DEBUG"]
2022/02/01 12:53:26 DEBUG : Creating backend with remote "S:\\AllData\\Admin\\Entertainment Books"
2022/02/01 12:53:26 DEBUG : Using config file from "C:\\Users\\timw\\Desktop\\rclone-ps-migration\\rclone.conf"
2022/02/01 12:53:26 DEBUG : fs cache: renaming cache item "S:\\AllData\\Admin\\Entertainment Books" to be canonical "//?/S:/AllData/Admin/Entertainment Books"
2022/02/01 12:53:26 DEBUG : Creating backend with remote "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books"
2022/02/01 12:53:26 DEBUG : Engage-and-Equip-Fundraising: detected overridden config - adding "{-LSY5}" suffix to name
2022/02/01 12:53:27 NOTICE: Time may be set wrong - time from "oauth2.googleapis.com" is -6m26.6829799s different from this computer
2022/02/01 12:53:28 NOTICE: Time may be set wrong - time from "www.googleapis.com" is -6m27.4969998s different from this computer
2022/02/01 12:53:33 DEBUG : fs cache: renaming cache item "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books" to be canonical "Engage-and-Equip-Fundraising{-LSY5}:SA/Admin/Entertainment Books"
2022/02/01 12:53:34 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Disabling ListR to work around bug in drive as multi listing (6) returned no entries
2022/02/01 12:53:34 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Recycled 6 entries
2022/02/01 12:54:06 ERROR : Google drive root 'SA/Admin/Entertainment Books': error reading destination root directory: couldn't list directory: googleapi: Error 403: The attempted action requires shared drive membership., teamDriveMembershipRequired
2022/02/01 12:54:06 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Waiting for checks to finish
2022/02/01 12:54:06 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Waiting for transfers to finish
2022/02/01 12:54:06 ERROR : Google drive root 'SA/Admin/Entertainment Books': not deleting files as there were IO errors
2022/02/01 12:54:06 ERROR : Google drive root 'SA/Admin/Entertainment Books': not deleting directories as there were IO errors
2022/02/01 12:54:06 ERROR : Attempt 1/3 failed with 1 errors and: couldn't list directory: googleapi: Error 403: The attempted action requires shared drive membership., teamDriveMembershipRequired
<snip> Lots of lines where it actually checks individual files and copies new ones etc </snip>

It's a shared drive? What are the user's permissions in the shared drive?

(Also, can you fix the time)

and share the same results without --fast-list

I wrongly assumed it was only an issue when lots of files are being synced. In this case there are only 35 files being synced. The account doing the syncing has "Content manager" permissions.

I unfortunately can't fix the time, it's synced to an AD server which is having issues syncing the time (and about to be decommissioned, so not worth spending lots of time fixing). It's only 6 minutes off now, it was ~30 minutes off before I tried to fix it!

I should have posted logs without --fast-list, here they are.

2022/02/01 16:19:50 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\Users\\timw\\Desktop\\rclone-ps-migration\\rclone.exe" "sync" "S:\\AllData\\Admin\\Entertainment Books" "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books" "--drive-stop-on-upload-limit" "--progress" "--log-file" "C:\\synclogs\\debug2.log" "--log-level" "DEBUG"]
2022/02/01 16:19:50 DEBUG : Creating backend with remote "S:\\AllData\\Admin\\Entertainment Books"
2022/02/01 16:19:50 DEBUG : Using config file from "C:\\Users\\timw\\Desktop\\rclone-ps-migration\\rclone.conf"
2022/02/01 16:19:50 DEBUG : fs cache: renaming cache item "S:\\AllData\\Admin\\Entertainment Books" to be canonical "//?/S:/AllData/Admin/Entertainment Books"
2022/02/01 16:19:50 DEBUG : Creating backend with remote "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books"
2022/02/01 16:19:50 DEBUG : Engage-and-Equip-Fundraising: detected overridden config - adding "{-LSY5}" suffix to name
2022/02/01 16:19:51 NOTICE: Time may be set wrong - time from "oauth2.googleapis.com" is -6m27.9085489s different from this computer
2022/02/01 16:19:51 NOTICE: Time may be set wrong - time from "www.googleapis.com" is -6m28.236966s different from this computer
2022/02/01 16:19:52 DEBUG : fs cache: renaming cache item "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books" to be canonical "Engage-and-Equip-Fundraising{-LSY5}:SA/Admin/Entertainment Books"
2022/02/01 16:19:53 DEBUG : Consolidated sales lists 2009-2011.xls: Size and modification time the same (differ by -160µs, within tolerance 1ms)
<snip>Lots of files</snip>
2022/02/01 16:19:54 DEBUG : 2018-2019/2018-2019 Sales Record.xls: Unchanged skipping
2022/02/01 16:19:54 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Waiting for checks to finish
2022/02/01 16:19:54 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Waiting for transfers to finish
2022/02/01 16:19:54 DEBUG : Waiting for deletions to finish
2022/02/01 16:19:54 INFO  : There was nothing to transfer
2022/02/01 16:19:54 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                35 / 35, 100%
Elapsed time:         4.2s

2022/02/01 16:19:54 DEBUG : 20 go routines active

I suspect that the more powerful listing query that fast list uses is somehow tripping the permissions set on the tream drive.

If you run your test with -vv --dump bodies can you capture a REQUEST and the matching RESPONSE which shows the error and post them here? That will give some idea of the problem.

I've tried to find the relevant sections that lead to the 403, skipping the auth section, and then skipping packets between the request that results in the 403 and it's response

2022/02/02 10:07:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:18 NOTICE: Time may be set wrong - time from "oauth2.googleapis.com" is -6m28.9605045s different from this computer
2022/02/02 10:07:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:18 DEBUG : HTTP REQUEST (req 0xc000020800)
2022/02/02 10:07:18 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AMBrU9Ir5R6PUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%270AMBrU9Ir5R6PUk9PVA%27+in+parents%29+and+%28name%3D%27SA%27%29+and+%28mimeType%3D%27application%2Fvnd.google-apps.folder%27+or+mimeType%3D%27application%2Fvnd.google-apps.shortcut%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

2022/02/02 10:07:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:18 DEBUG : HTTP RESPONSE (req 0xc000020800)
2022/02/02 10:07:18 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/json; charset=UTF-8
Date: Tue, 01 Feb 2022 23:43:48 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

183
{"incompleteSearch":false,"files":[{"id":"1z8m5Zxz1xrXcJ3suVlAEmGcsgsRLrpbK","name":"SA","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["0AMBrU9Ir5R6PUk9PVA"],"webViewLink":"https://drive.google.com/drive/folders/1z8m5Zxz1xrXcJ3suVlAEmGcsgsRLrpbK","createdTime":"2022-01-14T06:07:32.960Z","modifiedTime":"2022-01-14T06:07:32.960Z"}]}
0

2022/02/02 10:07:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:18 NOTICE: Time may be set wrong - time from "www.googleapis.com" is -6m29.3194987s different from this computer
2022/02/02 10:07:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:18 DEBUG : HTTP REQUEST (req 0xc0006c2d00)
2022/02/02 10:07:18 DEBUG : GET /drive/v3/about?alt=json&fields=exportFormats%2CimportFormats&prettyPrint=false HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

2022/02/02 10:07:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:18 DEBUG : HTTP RESPONSE (req 0xc0006c2d00)
2022/02/02 10:07:18 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Tue, 01 Feb 2022 23:43:48 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: ESF
Vary: Origin, X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

11e4
{"importFormats":{"application/x-vnd.oasis.opendocument.presentation":["application/vnd.google-apps.presentation"],"text/tab-separated-values":["application/vnd.google-apps.spreadsheet"],"image/jpeg":["application/vnd.google-apps.document"],"image/bmp":["application/vnd.google-apps.document"],"image/gif":["application/vnd.google-apps.document"],"application/vnd.ms-excel.sheet.macroenabled.12":["application/vnd.google-apps.spreadsheet"],"application/vnd.openxmlformats-officedocument.wordprocessingml.template":["application/vnd.google-apps.document"],"application/vnd.ms-powerpoint.presentation.macroenabled.12":["application/vnd.google-apps.presentation"],"application/vnd.ms-word.template.macroenabled.12":["application/vnd.google-apps.document"],"application/vnd.openxmlformats-officedocument.wordprocessingml.document":["application/vnd.google-apps.document"],"image/pjpeg":["application/vnd.google-apps.document"],"application/vnd.google-apps.script+text/plain":["application/vnd.google-apps.script"],"application/vnd.ms-excel":["application/vnd.google-apps.spreadsheet"],"application/vnd.sun.xml.writer":["application/vnd.google-apps.document"],"application/vnd.ms-word.document.macroenabled.12":["application/vnd.google-apps.document"],"application/vnd.ms-powerpoint.slideshow.macroenabled.12":["application/vnd.google-apps.presentation"],"text/rtf":["application/vnd.google-apps.document"],"application/vnd.oasis.opendocument.spreadsheet":["application/vnd.google-apps.spreadsheet"],"text/plain":["application/vnd.google-apps.document"],"application/x-vnd.oasis.opendocument.spreadsheet":["application/vnd.google-apps.spreadsheet"],"application/x-vnd.oasis.opendocument.text":["application/vnd.google-apps.document"],"image/png":["application/vnd.google-apps.document"],"application/msword":["application/vnd.google-apps.document"],"application/pdf":["application/vnd.google-apps.document"],"application/x-msmetafile":["application/vnd.google-apps.drawing"],"application/vnd.openxmlformats-officedocument.spreadsheetml.template":["application/vnd.google-apps.spreadsheet"],"application/vnd.ms-powerpoint":["application/vnd.google-apps.presentation"],"application/vnd.ms-excel.template.macroenabled.12":["application/vnd.google-apps.spreadsheet"],"image/x-bmp":["application/vnd.google-apps.document"],"application/rtf":["application/vnd.google-apps.document"],"application/vnd.openxmlformats-officedocument.presentationml.template":["application/vnd.google-apps.presentation"],"image/x-png":["application/vnd.google-apps.document"],"text/html":["application/vnd.google-apps.document"],"application/vnd.oasis.opendocument.text":["application/vnd.google-apps.document"],"application/vnd.openxmlformats-officedocument.presentationml.presentation":["application/vnd.google-apps.presentation"],"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet":["application/vnd.google-apps.spreadsheet"],"application/vnd.google-apps.script+json":["application/vnd.google-apps.script"],"application/vnd.openxmlformats-officedocument.presentationml.slideshow":["application/vnd.google-apps.presentation"],"application/vnd.ms-powerpoint.template.macroenabled.12":["application/vnd.google-apps.presentation"],"text/csv":["application/vnd.google-apps.spreadsheet"],"application/vnd.oasis.opendocument.presentation":["application/vnd.google-apps.presentation"],"image/jpg":["application/vnd.google-apps.document"],"text/richtext":["application/vnd.google-apps.document"]},"exportFormats":{"application/vnd.google-apps.document":["application/rtf","application/vnd.oasis.opendocument.text","text/html","application/pdf","application/epub+zip","application/zip","application/vnd.openxmlformats-officedocument.wordprocessingml.document","text/plain"],"application/vnd.google-apps.spreadsheet":["application/x-vnd.oasis.opendocument.spreadsheet","text/tab-separated-values","application/pdf","application/vnd.openxmlformats-officedocument.spreadsheetml.sheet","text/csv","application/zip","application/vnd.oasis.opendocument.spreadsheet"],"application/vnd.google-apps.jam":["application/pdf"],"application/vnd.google-apps.script":["application/vnd.google-apps.script+json"],"application/vnd.google-apps.presentation":["application/vnd.oasis.opendocument.presentation","application/pdf","application/vnd.openxmlformats-officedocument.presentationml.presentation","text/plain"],"application/vnd.google-apps.form":["application/zip"],"application/vnd.google-apps.drawing":["image/svg+xml","image/png","application/pdf","image/jpeg"],"application/vnd.google-apps.site":["text/plain"]}}
0

2022/02/02 10:07:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:18 DEBUG : HTTP REQUEST (req 0xc000020f00)
2022/02/02 10:07:18 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AMBrU9Ir5R6PUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271z8m5Zxz1xrXcJ3suVlAEmGcsgsRLrpbK%27+in+parents%29+and+%28name%3D%27Admin%27%29+and+%28mimeType%3D%27application%2Fvnd.google-apps.folder%27+or+mimeType%3D%27application%2Fvnd.google-apps.shortcut%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

2022/02/02 10:07:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:19 DEBUG : HTTP RESPONSE (req 0xc000020f00)
2022/02/02 10:07:19 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/json; charset=UTF-8
Date: Tue, 01 Feb 2022 23:43:49 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

194
{"incompleteSearch":false,"files":[{"id":"1kkyM341jdCEwcSpeYTSOgzULKb_72RGj","name":"Admin","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1z8m5Zxz1xrXcJ3suVlAEmGcsgsRLrpbK"],"webViewLink":"https://drive.google.com/drive/folders/1kkyM341jdCEwcSpeYTSOgzULKb_72RGj","createdTime":"2022-01-14T06:07:34.364Z","modifiedTime":"2022-01-14T06:07:34.364Z"}]}
0

2022/02/02 10:07:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:19 DEBUG : HTTP REQUEST (req 0xc0006c3300)
2022/02/02 10:07:19 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AMBrU9Ir5R6PUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271kkyM341jdCEwcSpeYTSOgzULKb_72RGj%27+in+parents%29+and+%28name%3D%27Entertainment+Books%27%29+and+%28mimeType%3D%27application%2Fvnd.google-apps.folder%27+or+mimeType%3D%27application%2Fvnd.google-apps.shortcut%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

2022/02/02 10:07:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:19 DEBUG : HTTP RESPONSE (req 0xc0006c3300)
2022/02/02 10:07:19 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/json; charset=UTF-8
Date: Tue, 01 Feb 2022 23:43:49 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

1a2
{"incompleteSearch":false,"files":[{"id":"1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5","name":"Entertainment Books","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1kkyM341jdCEwcSpeYTSOgzULKb_72RGj"],"webViewLink":"https://drive.google.com/drive/folders/1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5","createdTime":"2022-01-14T06:07:35.816Z","modifiedTime":"2022-01-14T06:07:35.816Z"}]}
0

2022/02/02 10:07:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:19 DEBUG : fs cache: renaming cache item "Engage-and-Equip-Fundraising:/SA/Admin/Entertainment Books" to be canonical "Engage-and-Equip-Fundraising{-LSY5}:SA/Admin/Entertainment Books"
2022/02/02 10:07:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:19 DEBUG : HTTP REQUEST (req 0xc000021400)
2022/02/02 10:07:19 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AMBrU9Ir5R6PUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5%27+in+parents%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

2022/02/02 10:07:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:21 DEBUG : HTTP RESPONSE (req 0xc000021400)
2022/02/02 10:07:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/json; charset=UTF-8
Date: Tue, 01 Feb 2022 23:43:51 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

1b06
{"incompleteSearch":false,"files":[{"id":"1mMPFnPVN8JfgLBnMF04y1dQfIPb5zqPP","name":"Registered Purchasers List","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1mMPFnPVN8JfgLBnMF04y1dQfIPb5zqPP","createdTime":"2022-01-14T06:09:21.254Z","modifiedTime":"2022-01-14T06:09:21.254Z"},{"id":"1IdhJN2FLh8-M8uDGvHzJfGYDByO_zzav","name":"2019-2020","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1IdhJN2FLh8-M8uDGvHzJfGYDByO_zzav","createdTime":"2022-01-14T06:09:11.013Z","modifiedTime":"2022-01-14T06:09:11.013Z"},{"id":"1BDdQ8zYYx79uxtzibLjw7e_nx2BHheFL","name":"Promotions","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1BDdQ8zYYx79uxtzibLjw7e_nx2BHheFL","createdTime":"2022-01-14T06:08:12.319Z","modifiedTime":"2022-01-14T06:08:12.319Z"},{"id":"1k5Q0ARuFztDZ4DHCHmt9Zn3DjRBqyF7r","name":"2018-2019","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1k5Q0ARuFztDZ4DHCHmt9Zn3DjRBqyF7r","createdTime":"2022-01-14T06:08:05.947Z","modifiedTime":"2022-01-14T06:08:05.947Z"},{"id":"1RyehTIXe7gL1DnfhQZCKYytmVAuu4PaQ","name":"2017-2018","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1RyehTIXe7gL1DnfhQZCKYytmVAuu4PaQ","createdTime":"2022-01-14T06:08:03.440Z","modifiedTime":"2022-01-14T06:08:03.440Z"},{"id":"16Kr-4BY6Vjj0WoJqAi3egIl5lS6xwsUQ","name":"2015-2016","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/16Kr-4BY6Vjj0WoJqAi3egIl5lS6xwsUQ","createdTime":"2022-01-14T06:08:01.207Z","modifiedTime":"2022-01-14T06:08:01.207Z"},{"id":"1Mp6q1FLPKMUgrukJQl5Bfl-JF4HKljMc","name":"2011-2012","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1Mp6q1FLPKMUgrukJQl5Bfl-JF4HKljMc","createdTime":"2022-01-14T06:07:55.964Z","modifiedTime":"2022-01-14T06:07:55.964Z"},{"id":"10YDueOEJBAHoeF_bOF2Vhe7oooRxFJB1","name":"2010-2011","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/10YDueOEJBAHoeF_bOF2Vhe7oooRxFJB1","createdTime":"2022-01-14T06:07:53.970Z","modifiedTime":"2022-01-14T06:07:53.970Z"},{"id":"1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx","name":"2009-2010","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx","createdTime":"2022-01-14T06:07:49.313Z","modifiedTime":"2022-01-14T06:07:49.313Z"},{"id":"1efx-h-hhZgKox7wSnyor4_g8ua-m6GHt","name":"2014-2015","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YPHzQSurhM8BGIJgTw8VT9tv9zp9Wjp5"],"webViewLink":"https://drive.google.com/drive/folders/1efx-h-hhZgKox7wSnyor4_g8ua-m6GHt","createdTime":"2022-01-14T06:07:47.207Z","modifiedTime":"2022-01-14T06:07:47.207Z"},
<snip some folders and files>
0

2022/02/02 10:07:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/02/02 10:07:21 DEBUG : HTTP REQUEST (req 0xc0006c3a00)
2022/02/02 10:07:21 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AMBrU9Ir5R6PUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%2710YDueOEJBAHoeF_bOF2Vhe7oooRxFJB1%27+in+parents+or+%271BDdQ8zYYx79uxtzibLjw7e_nx2BHheFL%27+in+parents+or+%271LlsxZOJ0uunb_u83Asa_csXL9hQRRG8-%27+in+parents+or+%271OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx%27+in+parents+or+%271efx-h-hhZgKox7wSnyor4_g8ua-m6GHt%27+in+parents+or+%271xpR78gp4WbMEqKBrQkd81yzB8pdXYWCY%27+in+parents%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

2022/02/02 10:07:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

snip packets until we get the reply from above that's a 403

2022/02/02 10:07:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:21 DEBUG : HTTP RESPONSE (req 0xc0006c3a00)
2022/02/02 10:07:21 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private, max-age=0
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/json; charset=UTF-8
Date: Tue, 01 Feb 2022 23:43:51 GMT
Expires: Tue, 01 Feb 2022 23:43:51 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

e2
{"error":{"errors":[{"domain":"global","reason":"teamDriveMembershipRequired","message":"The attempted action requires shared drive membership."}],"code":403,"message":"The attempted action requires shared drive membership."}}
0

2022/02/02 10:07:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/02/02 10:07:21 DEBUG : Google drive root 'SA/Admin/Entertainment Books': Disabling ListR to work around bug in drive as multi listing (6) returned no entries

Lets look at the error query and response first

2022/02/02 10:07:21 DEBUG : HTTP REQUEST (req 0xc0006c3a00)
2022/02/02 10:07:21 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AMBrU9Ir5R6PUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%2710YDueOEJBAHoeF_bOF2Vhe7oooRxFJB1%27+in+parents+or+%271BDdQ8zYYx79uxtzibLjw7e_nx2BHheFL%27+in+parents+or+%271LlsxZOJ0uunb_u83Asa_csXL9hQRRG8-%27+in+parents+or+%271OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx%27+in+parents+or+%271efx-h-hhZgKox7wSnyor4_g8ua-m6GHt%27+in+parents+or+%271xpR78gp4WbMEqKBrQkd81yzB8pdXYWCY%27+in+parents%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.17.2 gdcl/20210812
Accept-Encoding: gzip

Decoding the query, rclone is trying to run this query to list those 6 directories in parallel.

q=trashed=false and
('10YDueOEJBAHoeF_bOF2Vhe7oooRxFJB1' in parents or
 '1BDdQ8zYYx79uxtzibLjw7e_nx2BHheFL' in parents or
 '1LlsxZOJ0uunb_u83Asa_csXL9hQRRG8-' in parents or
 '1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx' in parents or
 '1efx-h-hhZgKox7wSnyor4_g8ua-m6GHt' in parents or
 '1xpR78gp4WbMEqKBrQkd81yzB8pdXYWCY' in parents)

So the question is, is one of those directories not visible to you for some reason, or is is just the complicated query that is the problem.

Can you try listing each one of those directory IDs and see what happens?

So

rclone lsf Engage-and-Equip-Fundraising,root_folder_id=10YDueOEJBAHoeF_bOF2Vhe7oooRxFJB1:

Add -vv --dump bodies if you aren't sure what is happening.

If you find one folder which gives the same error "The attempted action requires shared drive membership." then can you look back through your original log and find the ID and then trace back where rclone got it from.

rclone lsf worked on all those folder_id's. I fired up postman and chucked the query in, as soon as I have an or statement in the query with another parent check, it fails with the 403.
In fact, the query

trashed=false+and+'1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx'+in+parents

works and

trashed=false+or+'1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx'+in+parents

doesn't.

name+contains+'random'+or+'1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx'+in+parents fails and name+contains+'random'+and+'1OTvCfpFAnWgwjAXdaOW3A4v28i8LzHnx'+in+parents works. So something about the or operator is not working.

The user we are impersonating has 'Content Manager' permissions on the top level of the shared drive, and I've even given it 'Manager' level to test with no change. I'm starting to guess it's a bug with Google's API.

Any other suggestions? We have support contracts with Google for GCP, so I may be able to raise an issue through that.

Even this fails. It's something about the or part of the query. The query works fine if we don't hit the shared drive and just hit the users drive, I've tried opening a ticket with Google support to see what they come back with.

name+contains+'random'+or+trashed=false

That is excellent debugging.

So it sounds like it is the complexity of the query that is causing the problem rather than the permissions of the folders or anything like that.

I would say this is a bug in the Google API.

API bugs get reported here: https://issuetracker.google.com/ - I had a brief look but I couldn't see a relevant one. There are quite a few bugs in the API rclone has to work around unfortunately.

Let us know what Google support says :slight_smile:

I've opened an issue at Google Issue Tracker
Google Support only covers GCP it seems, so I have to wait for the public "support" process. Thanks

Ooh, you got a reply asking for more info, so that is a good start!

Thanks for all your help @ncw and @Animosity022 for both issues. I updated Chunked uploads to Google Drive fail for Shared Drives - #9 by Tim_SU with what turned out to be the root cause. A service account lets you impersonate disabled users, and disabled users can use the Drive API for 99% of things, just some edge cases where stuff is broken.

I've updated the Google issue so we'll see what they say the expected behaviour for a disabled user is. It would have been nice if 2 weeks ago when it was disabled my rclone just stopped working, I'd have been able to find the issues much easier!

Again, thanks for all your help and the countless hours your pour into rclone. I've been using it for years and I think it's time to try get work to make a donation.

1 Like

That is super weird! And must surely be a) a bug and b) a security issue.

In fact it might be worth reporting it to goo.gl/vulnz

:slight_smile:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.