Incorrect results with My Drive shared folders and service accounts

Help and Support
1

I am getting an odd result using rclone with service accounts in My Drive on a G Suite account. Using version 1.49.0-013 now, but had exactly the same results with all recent releases and beta versions. [EDIT: Installed 1.49.1 and same result.]

A folder called Media in remote gdrive is shared with an email for a service account using 1.json. I have double checked that the json contents include the specific email that Media has been shared to.

Command 1 shows the correct count of all files and a correct size for the folder Media.

If I run Command 2 below with --drive-shared-with-me and --drive-service-account-file flags it shows a size and file count which only includes some of the files in Media.

If I create an alias remote using exactly the same --drive-shared-with-me and --drive-service-account-file flags and values and run Command 3 then rclone shows the correct/full size and file count.

In principle Commands 2 and 3 should show exactly the same result. But they do not.

I have repeated this process with other accounts, folders and jsons and get a very similar (inconsistent) result, where flags in the command line give me partial access to files in a folder, whereas creating an alias with the same flags gives full access.

I have repeated the experiment with rclone lsd, lsf as well as with rclone copy and sync. In each case the result is the same. Command line flags seem to not give the same result as using the same flags in an alias remote.

All of the above occurs only when using My Drive folders. When using the same configuration and commands on Team Drives everything works perfectly.

I repeated the commands with -vv and -vvv but there are no errors at all.

Thank you as always. rclone continues to be astonishingly useful in all other respects.

Command 1 gives correct result:

root@gce:# rclone size gdrive:Media
Total objects: 51343
Total size: 34.467 TBytes (37897044011916 Bytes)

Command 2 gives incorrect result using shared and SA flags:

root@gce:# rclone size gdrive:Media --drive-shared-with-me=true --drive-service-account-file=/sa/1.json
Total objects: 691
Total size: 235.971 GBytes (253371929943 Bytes)

Command 3 gives correct result using exact same shared and SA flags:

root@gce:# rclone size alias2:Media
Total objects: 51343
Total size: 34.467 TBytes (37897044011916 Bytes)

root@gce:# rclone config show alias2

[alias2]
type = alias
remote = gdrive:
shared_with_me = true
service_account_file = /sa/1.json

2

--drive-shared-with-me shows only the files shared with you, not your files, so you'd expect the numbers to be different.

How did you do that? did you put the shared_with_me = true in the config file? What does rclone config show remote show? (remove the token if you post it).

3

The Media folder is shared with the service account, so rclone size is being pointed at the same folder. The first time directly, the second time via the SA. Same files, same size.

My understanding is that service accounts are viewed as 'separate' accounts [Is this wrong?]. So files shared with an SA are not visible unless you use shared_with_me. If I run rclone lsd or size on the folder that has been shared but omit --drive-shared-with-me it fails with '2019/09/03 18:52:36 Failed to size: directory not found' [ As it should].

The My Drive remote, gdrive, has no files shared with it. In the web ui 'Shared with me' is empty. Whereas the files shown if I lsf with Command 3 in my first post do show files inside the Media folder.

I created the alias using rclone config update alias2 type alias remote gdrive: service_account_file /sa/1.json shared_with_me true

This is the config of the gdrive: remote
[gdrive]
type = drive
client_id = clientid1234.apps.googleusercontent.com
client_secret = secret
scope = drive
root_folder_id =
team_drive =
token = {"access_token":"token"}

4

Ah I see part of the puzzle... the alias definition has shared_with_me = true but that will not work - that needs to be on a drive backend. So alias2 is ignoring the shared_with_me.

My understanding is that service accounts can be seen as separate accounts, however they are most useful in conjunction with --drive-impersonate to make it appear they are a particular user.

5

I was afraid that might be the case. I don't suppose there is a master list of which flags will work correctly overlaid in an alias remote and which ones will not?

Is the same true of service_account_file? That is, is it ignored when placed in an alias remote?

--drive-impersonate does work, yes. But in this context it is identical to simply using the main account :slight_smile:

May I ask you: If I set service accounts and shared with me on a remote backend (the original one, not an alias) using export RCLONE_CONFIG_REMOTE_ , will that override the settings in the rclone.conf file?

For example, if the config has no service account and does not use shared-with-me, will setting it using the export command take effect?

Thank you for the explanation re alias flags not overriding the back end. Kind regards

6

Only the flags here will work in an alias remote: https://rclone.org/alias/#standard-options

Yes it will. Environment vars override config file and in turn command line flags override those.

1 Like
7

Thank you. Looking at the link you provided the only flag for an alias show in Standard Options is --alias-remote.

Does that mean that an alias config like the one below would ignore everything after type = alias and remote = gdrive: and defer to the flags in the gdrive remote?

[alias_gdrive]
type = alias
remote = gdrive:
shared_with_me = true
service_account_file = /sa/1.json
server_size_across_configs = true
use_trash = false

Where the original remote has this config

[gdrive]
type = drive
scope = drive:
client_id = clientid
client_secret = secret
shared_with_me = false
server_size_across_configs = false
use_trash = true
root_folder_id =
token = {token}

[[ EDIT: I suppose it's not possible to change the way this works, where alias flags would override the original flags where different? That would expand the utility of an alias remote :slight_smile: . ]]

8

Yes it will.

To do this we'd need to put the options to override into an alias variable...

So we could make a new alias variable called override into which you'd put a comma separated list of parameters to override in the destination remote so in your case like this

[alias_gdrive]
type = alias
remote = gdrive:
override = shared_with_me = true, service_account_file = /sa/1.json, server_size_across_configs = true, use_trash = false

What do you think? Would that be useful enough?

If so please make a new issue on github and I'll implement it!

9

That would indeed. Thank you :+1:
Feature request submitted https://github.com/rclone/rclone/issues/3509

I think we could assemble to full set of flags first with commas and variable expansion. So yes, I think it would work!

With your suggested variable would it look like this?

rclone config update alias2 type alias remote gdrive: override "service_account_file $a , shared_with_me $b , use_trash $c"

Rather than what I had used to make the original alias remote, this:

rclone config update alias2 type alias remote gdrive: service_account_file $a shared_with_me $b use_trash $c

1 Like
10

I think that should do it!

1 Like
11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.