Google Drive listing shortcuts of folders cost ~200x time more than normal folders

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

When I run "rclone lsjson remote:/path" on Google Drive, it cost ~200x times more if the path contains shortcuts of folders, each shortcut points to a folder which contains sub-folders and around 50 files.

Test result:
1, run on path contains 2407 folders
|real|0m12.248s|
|user|0m0.143s|
|sys|0m0.036s|

2, run on path contains 2407 shortcuts (point to folders of above test)
|real|47m33.551s|
|user|0m0.036s|
|sys|0m0.036s|

3, run on path contains 106 shortcuts (point to folders of above test)
|real|0m12.248s|
|user|0m0.143s|
|sys|0m0.036s|

Is this expected, or I did anything wrong?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2-purple

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.15.0-71-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

time rclone lsjson media:/media/K-TV-bas/

The rclone config contents with secrets removed.

[media]
type = drive
scope = drive
service_account_file = 
service_account_file_path = 
team_drive =

A log from the command with the -vv flag

2023/05/13 15:45:28 DEBUG : rclone: Version "v1.62.2-purple" starting with parameters ["rclone" "-vv" "lsjson" "media:/media/K-TV-bas/"]
2023/05/13 15:45:28 DEBUG : Creating backend with remote "media:/media/K-TV-bas/"
2023/05/13 15:45:28 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/13 15:45:29 DEBUG : fs cache: renaming cache item "media:/media/K-TV-bas/" to be canonical "media:media/K-TV-bas"
[
... omitted json output ...
]
2023/05/13 15:46:06 DEBUG : 6 go routines active
2

hello and welcome to the forum,

to get a deeper look, to see if the delays are from rclone or gdrive, might add something like
--dump=headers --log-format=microseconds

3

Here is part of log, not sure if it's enough.

00:07:57.046893 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

00:07:57.088467 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

00:07:57.088489 DEBUG : HTTP REQUEST (req 0xc000ae8a00)

00:07:57.088494 DEBUG : GET /drive/v3/files/139jsQxHE6bX5ldhaLJ9_X5o3TQzEuzlA?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&prettyPrint=false&supportsAllDrives=true HTTP/1.1

Host: www.googleapis.com

User-Agent: rclone/v1.62.2-purple

Authorization: XXXX

X-Goog-Api-Client: gl-go/1.20.3 gdcl/0.112.0

Accept-Encoding: gzip

00:07:57.088503 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

00:07:57.602639 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

00:07:57.602662 DEBUG : HTTP RESPONSE (req 0xc000ae8a00)

00:07:57.602712 DEBUG : HTTP/1.1 200 OK

Transfer-Encoding: chunked

Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Cache-Control: no-cache, no-store, max-age=0, must-revalidate

Content-Type: application/json; charset=UTF-8

Date: Sat, 13 May 2023 16:07:57 GMT

Expires: Mon, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: ESF

Vary: Origin, X-Origin

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Xss-Protection: 0

00:07:57.602734 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
00:07:57.603103 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

4

Rclone has to resolve each shortcut which uses an extra http transaction per shortcut. Normal listing the directory only takes one transaction.

So yes, I'd say this is expected.

5

I see, thanks for the answer.

6

Is there anyway (within current version) to get the shortcuts list only, and without resolving shortcuts?

7

The problem that rclone has is that without resolving them, it doesn't know whether they point to files or directories. Do they point to files or directories?

You can skip shortcuts with

  --drive-skip-shortcuts                       If set skip shortcut files
8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.