Some GDrive folders invisible for rclone until user opens them in web UI

What is the problem you are having with rclone?

When listing the contents of a directory shared with me, some of its subdirectories are not listed, and I see no errors in the logs. If I login in the GDrive webUI using the same user that is authorized in the rclone remote and navigate into those folders, they become visible for rclone and I can list and copy them.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0-beta.6141.d206761d2.fix-drive-resourcekey
- os/version: debian 10.4 (64 bit)
- os/kernel: 4.19.0-9-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsd pro:"/Premsa"

The rclone config contents with secrets removed.

[pro]
type = drive
scope = drive
token = {"access_token":"****","token_type":"Bearer","refresh_token":"****","expiry":"2022-05-18T12:08:04.024825358+02:00"}
client_id = 89****.apps.googleusercontent.com
client_secret = ****
root_folder_id = 1cv****
team_drive =

A log from the command with the -vv flag

~# rclone lsd pro:"/Premsa" -vv
2022/05/18 16:06:08 DEBUG : rclone: Version "v1.59.0-beta.6141.d206761d2.fix-drive-resourcekey" starting with parameters ["rclone" "lsd" "pro:/Premsa" "-vv"]
2022/05/18 16:06:08 DEBUG : Creating backend with remote "pro:/Premsa"
2022/05/18 16:06:08 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/05/18 16:06:08 DEBUG : pro: Loaded invalid token from config file - ignoring
2022/05/18 16:06:08 DEBUG : Saving config "token" in section "pro" of the config file
2022/05/18 16:06:08 DEBUG : pro: Saved new token in config file
2022/05/18 16:06:08 DEBUG : fs cache: renaming cache item "pro:/Premsa" to be canonical "pro:Premsa"
          -1 2022-04-26 12:23:23        -1 BC
          -1 2022-04-26 12:22:49        -1 Ca
2022/05/18 16:06:09 DEBUG : 6 go routines active

The directory listing should show a 3rd directory (I only realized because a user complained about it missing). I went to GDrive via web with the same user account that I authorized for this remote, opened the invisible directory and now I can list it and copy it with rclone.

Thanks!

I have noticed these too.

I suspect these are from folders shared with a resourcekey in the URL.

I just added a fix to the latest beta so that you can actually download files.

So first can you try the beta to see if you can download files with it.

Was this directory from one which was originally shared with a link?

It might be that rclone needs to be using the resourcekey when listing the directory too - documentation is scant on the google drive API docs about this.

Not sure if the beta should be able to list (and copy) these directories before I navigate into them in the web UI, or only copy them after?

I downloaded the whole share via web UI and I'm looking at the diff between the outputs of the ~# find command on that downloaded tree and the one I get with the rclone beta (rclone-v1.59.0-beta.6154.fcec4bedb-linux-amd64).

I'm still finding some directories in the tree downloaded via the web UI that rclone didn't get. Is there any useful output I can provide if I find more cases of this?

The beta should be able to list the directory once you've opened it in the web. I don't know whether it will be able to list subdirectories of the main one though.

Try opening them in the web interface, then see if rclone copies them would be a good first step.

And if so did it have resourcekey in the URL?

Yes, I can list and copy the contents of the directory after opening it via web, but subdirectories are not listed until I navigate into each one of them.

Yeah, that works.

Yup! The URL is something like:
https://drive.google.com/drive/folders/0B2...TkU?resourcekey=0-S...8pw&usp=sharing

OK, give this a go

v1.59.0-beta.6160.73ea6188f.fix-resourcekey on branch fix-resourcekey (uploaded in 15-30 mins)

This should pick up the resource keys for the sub directories automatically.

Can you try rclone lsf -R pro:"/Premsa" to see if that works to find all the files.. If it doesn't try rclone lsf --disable ListR -R pro:"/Premsa".

This should also be able to list a drive without opening it in the web interface first. You'll need to supply the resourcekey as --drive-resource-key or in the config as resource_key.

It says in the google docs

This resource key requirement only applies to a subset of old files.

Are these old files?

1 Like

On a first test this seems to be working.

I downloaded the whole share using the "Download" functionality of the web UI. Then I counted the downloaded files and directories:

 find Premsa/ | wc -l
33535

I added the resourcekey of the parent shared folder to my config file:

[pro]
type = drive
scope = drive
token = ***
client_id = ***
client_secret = ***
root_folder_id = ***
team_drive = 
resource_key = ***

Then I list all files and count how many were listed:

~# rclone lsf -R pro:"/Premsa" | wc -l
12453
~# rclone lsf --disable ListR -R pro:"/Premsa" | wc -l
34168

Using --disable ListR I get a higher count than on the downloaded archive, which I guess can be due to duplicates.

I'll try to look into detail at these differences, but the web UI download renames many paths (e.g. those using special characters) which makes it difficult to use diff to compare them with the ones obtailed by rclone lsf.

I wasn't able to use the command line option, I get this error:

~# rclone lsf --drive-resource-key="3Q***Tg" -R pro:"/Premsa" 
Error: invalid argument "3Q***Tg" for "--drive-resource-key" flag: parsing "3Q***Tg" as bool failed: expected newline
Usage:
  rclone lsf remote:path [flags]
(...)
2022/05/23 23:26:38 Fatal error: invalid argument "3Q***Tg" for "--drive-resource-key" flag: parsing "3Q***Tg" as bool failed: expected newline

Yes, I've seen files as old as 2014.

Thanks again!

Great.

The --disable ListR looks to be the correct value.

I was concerned that ListR wouldn't work with resource keys as it lists multiple directories at once which means specifying multiple resource IDs. This should work but there have been a lot of corner cases in ListR.

The best way to see the differences is

rclone lsf -R pro:"/Premsa" | sort > with-listr
rclone lsf --disable ListR -R pro:"/Premsa" | sort > without-listr
comm -23 without-listr with-listr > missing-in-without-listr
comm -13 without-listr with-listr > missing-in-with-listr

What I suspect you are going to see is that whole directories are missing from the missing-in-without-listr. Assuming that is the case and that I constructed the headers correctly then this is probably a(nother) bug in the google search query handling. It might be better to disable ListR if resourcekeys are in use.

Oops, my mistake - try this:

v1.59.0-beta.6164.37d3308f3.fix-resourcekey on branch fix-resourcekey (uploaded in 15-30 mins)

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.