Google Drive with service account and restricted scope


#1

Hi,

I’m trying to set up a Google Drive remote with a service account as described here, the only difference being that I want to use the https://www.googleapis.com/auth/drive.appfolder scope instead of the https://www.googleapis.com/auth/drive one, and it doesn’t work.

If I put https://www.googleapis.com/auth/drive.appfolder in the “One or More API Scopes” field (last point under “2.” in the description) and use this config:

[drive-appfolder]
type = drive
scope = drive.appfolder
service_account_file = <path to json-file>
root_folder_id = appDataFolder

I get this error:

$ rclone -v --drive-impersonate <email> ls drive-appfolder:
2019/01/05 12:59:12 Failed to ls: couldn't list directory: Get https://www.googleapis.com/drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%27appDataFolder%27+in+parents%29&spaces=appDataFolder: oauth2: cannot fetch token: 401 Unauthorized
Response: {
  "error": "unauthorized_client",
  "error_description": "Client is unauthorized to retrieve access tokens using this method."
}

If, on the other hand, I put https://www.googleapis.com/auth/drive in the “One or More API Scopes” field and use this config:

[drive]
type = drive
scope = drive
service_account_file = <path to same json-file>

things work as expected:

$ rclone -v --drive-impersonate <email> ls drive:
$ echo $?
0

I’m using:

$ rclone version
rclone v1.45
- os/arch: linux/amd64
- go version: go1.10.5

Any ideas?


#2

Hmm, I suspect --drive-impersonate isn’t compatible with appfolder. I tried searching for a combination of the two and I couldn’t find anything though.


#3

So, a bug, a missing feature, or both? :slight_smile:

At the moment, I need to choose between a) using a service account with the
drive scope and having rclone data exposed via the web interface and b) using the OAuth 2 token flow with the drive.appfolder scope and not having rclone data exposed via the web interface.

Neither option is ideal, but (unless you have other ideas) I’ll probably stick with a) since all of my rclone data is encrypted so having it exposed via the web interface is more of a nuisance than a potential security threat…


#4

A good question!

I think it might be a bug - I’m not passing the correct scopes in to the JWT token…

try this

https://beta.rclone.org/branch/v1.45-057-g90aae624-fix-drive-scope-beta/ (uploaded in 15-30 mins)


#5

Wow! I’m impressed. I just tried the linux/amd64 version and both of the above configs now work as expected with their respective scopes. Excellent. Thank you very much. :smile:

When do you expect an official release with this fix to be out?


#6

Great :smile:

I’ve merged that to master now, it will be in the latest beta in 15-30 mins then be released in v1.46 on 7th Feb or there abouts!


#7

Sweet. :smile:

Anyway, I’ve done a bit more testing, with the latest beta and this config (and https://www.googleapis.com/auth/drive.appfolder as API scope):

[drive]
type = drive
scope = drive.appfolder
service_account_file = <path to json-file>
root_folder_id = appDataFolder
impersonate = <email>

Any idea whether this is another scope-related issue? Or something else entirely?

$ rclone about drive:
Used:    0
Trashed: 0
Other:   1.394G
$ rclone ls drive:
$ ls -s file
4 file
$ rclone copy file drive:
$ rclone ls drive:
        4 file
$ rclone cleanup drive:
2019/01/08 22:06:01 ERROR : Attempt 1/3 failed with 0 errors and: googleapi: Error 403: Insufficient Permission, insufficientPermissions
2019/01/08 22:06:01 ERROR : Attempt 2/3 failed with 0 errors and: googleapi: Error 403: Insufficient Permission, insufficientPermissions
2019/01/08 22:06:01 ERROR : Attempt 3/3 failed with 0 errors and: googleapi: Error 403: Insufficient Permission, insufficientPermissions
2019/01/08 22:06:01 Failed to cleanup: googleapi: Error 403: Insufficient Permission, insufficientPermissions
$ rclone about drive:
Used:    4
Trashed: 0
Other:   1.394G
$ rclone delete drive:file
$ rclone about drive:
Used:    4
Trashed: 4
Other:   1.394G
$ rclone ls drive:
$ rclone ls drive: --drive-trashed-only
        4 file
$ rclone delete drive:file --drive-trashed-only --drive-use-trash=false
$ rclone about drive:
Used:    0
Trashed: 0
Other:   1.394G

#8

Cleanup just calls https://developers.google.com/drive/api/v3/reference/files/emptyTrash so I guess that is out of scope for an app scoped key.

Whereas this

Will iterate all files from the root deleting trashed files.

Note that if you don’t want trashed files you can put --drive-use-trash=false or use_trash = false in the config.


#9

Right. I was looking at https://developers.google.com/drive/api/v3/about-auth and it says nothing about trash in the scope descriptions. Good to know that this is not a bug. :slight_smile:

Nah, I like being able to restore files. :slight_smile:

Anyway, with this config (and https://www.googleapis.com/auth/drive.appfolder,https://www.googleapis.com/auth/drive as API scopes):

[drive]
type = drive
scope = drive,drive.appfolder
service_account_file = <path to json-file>
root_folder_id = appDataFolder
impersonate = <email>

cleanup now works – which is to say, it works like the Empty trash button on the web interface in that it removes both app-specific and non-app-specific trash. I actually think this behaviour is a bit dangerous, but, of course, out of your hands. (The Empty trash button on the web interface may remove things that are not actually shown in the Trash folder on the web interface! At least with rclone you can fiddle with the root_folder_id and trashed_only options and SEE both types of trash.)

One last thing…

If I create a folder, say test7, via the web interface, and use this config:

[drive]
type = drive
scope = drive,drive.appfolder
service_account_file = <path to json-file>
impersonate = <email>

rclone shows the folder as both trash and non-trash! Surely this must be a bug in rclone?

$ rclone lsd drive: --drive-trashed-only=false
          -1 2019-01-09 22:07:24        -1 test7
$ rclone lsd drive: --drive-trashed-only=true
          -1 2019-01-09 22:07:24        -1 test7

#10

:smile:

That is good!

Rclone shows all folders in trashed-only mode otherwise you wouldn’t be able to see the files underneath them. So you can have a non trashed folder with trashed files in.

That is the reason why…

I’m not particularly happy with the --drive-trashed-only interface but I couldn’t think of anything better!


#11

Well, it confused me, so I’m not particularly happy with it either. :smile:

Maybe --drive-trashed-only=true should only consider a non-trashed folder if it has one or more trashed descendants (files or folders). This will probably make operations more expensive (I guess you need to do an initial depth-first traversal of the filesystem tree and mark relevant nodes), but at least --drive-trashed-only=true will work as advertised – each (leaf) file or folder processed will actually be “trashed-only”. :wink:


#12

A file system traversal will make it super expensive to run as directory traversals are really slow in drive.

If you could build a query using the v3 API query language it could be made fast, but I don’t think it is possible to express relationships of more than one level…