How can I create multiple independent rclone configs when connecting to Google Drive?

What is the problem you are having with rclone?

We are running a project that facilitates connection for different users to different drives for various actions. We have a problem with large files that needs to be updated/sync etc. on Google Drive. When the process time exceeds 1h, the previous token we used for this task expires and needs to be refreshed. As I read other threads, I understood that rclone refreshes automatically the tokens. The problem is that we cannot have a global rconf for each individual users from where rclone can take the refresh token and refresh it because multiple users can trigger actions at the same time and data race can happen over rconf.

Credentials for Google Auth are created on the fly in our backend logic. (Basically, the user is inserting in UI, client_id, client_secret and client_token). Then the users hits something like "trigger" and then the command listed bellow is created and triggered in a subprocess.

OS: Linux - Ubuntu 20.04

Any thoughts how can I achieve this ?

Run the command 'rclone version' and share the full output of the command.

Version: rclone-v1.56.2-linux-amd64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv --drive-token {access_token: some_token, token_type: "Bearer", "refresh_token": some_refresh_token, "expiry": "2022-02-10T23:05:36.476357+00:00" }  --drive-team-drive "some_drive_team_hash"  copy :drive:ExecOnline/5. Managed Services/b.Productio/GDriveConnector /tmp/tmpgx1tp1sv --filter "+ To_Path/**/ *" --filter "- From_Some_path/ **/*.*" --filter - From_SomePath/* --filter "- *.DS_Store" --filter "-*" --max-age "0001-01-01T00:00:00+00:00"

A log from the command with the -vv flag

2022/02/28 18:00:47 DEBUG : rclone: Version "v1.53.1" starting with parameters 
["rclone" "-vv" "--drive-token" "{\"access_token\": 
\"some_token\", \"token_type\": \"Bearer\", \"refresh_token\": 
\"some_refresh_token", \"expiry\": \"2022-02-
28T18:00:56.088516+00:00\"}" "--drive-team-drive" "some_hash" "copy" 
"/tmp/tmpca6v76_4" ":drive:ExecOnline/5. Managed Services/b. 
Production/GDriveConnector"] 
2022/02/28 18:00:47 DEBUG : Creating backend with remote "/tmp/tmpca6v76_4" 
2022/02/28 18:00:47 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - 
using defaults 
2022/02/28 18:00:47 DEBUG : Creating backend with remote ":drive:ExecOnline/5. 
Managed Services/b. Production/GDriveConnector" 
2022/02/28 18:00:47 DEBUG : :drive: Failed to read token out of config file: config file 
not found 
2022/02/28 18:00:47 DEBUG : :drive: Token refresh failed try 1/5: oauth2: cannot 
fetch token: 401 Unauthorized 
Response: { 
  "error": "unauthorized_client", 
  "error_description": "Unauthorized"
}

Best path probably is a service account instead of using a normal user account.

Google drive (rclone.org)

1 Like

hello and welcome to the forum

can use as many config files as needed, using --config to point to the config file.

creates the remotes on the fly. many ways to do that
--- use connection strings, to create the remote on the fly
--- use rclone config create
--- use environment variables.

1 Like

Thank you for welcoming and reply !

  1. If it is to opt for rclone config create google_drive_client_id --non-interactive type=drive client_id={{client_id}}..., will it create a config file to a default path, that has in it client_id=1234, token=some_token etc ... ?

Is it a way also to specifiy to which path rclone config create should create the config file ?

  1. If I opt for creating conf on disk, it is enough to create a myconf.conf file to root of the project (let's say), and that would be enough by giving the path to --config flag ?

  2. Connection strings sounds very interesting. How can I use them more precisely in my case?

PS: It is also for the first time I am using rclone, so some questions might sound a little silly.

That's true, but this does not depend solely on myself. I guess we should stick for normal user account for a while. But it should be taken into consideration. Thanks a bunch !

  1. i do not use gdrive, so not familar with the exact syntax.
    tho i am sure this has been discussed a number of times in the forum.

--config=/path/to/rclone.conf

  1. take a read of the docs, try a few tests.

fwiw, when scripting with python, i create on the fly remotes using environment variables.
and feed that to subprocess.run.
and in those cases, no config file is used.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.