OneDrive and GoogleDrive - Refreshing Tokens

Help and Support
1

What is the problem you are having with rclone?

I am creating an access token json using my own python script. This is for both OneDrive and Google Drive.
Both the token jsons come back just fine using the Oauth procedure.
They both include refresh tokens.
I use rclone on a remote machine.
I paste the json responses with no extra spaces (except for the ones within value strings) as one single line, as the token value in rclone.conf.
I confirmed that I have the correct client id and client secret in the rclone config I use on the remote server.
Any command, such as copy, sync, lsd, etc work perfectly fine, until the token expires and a refresh token should get issued.
I have confirmed using Postman, that even after many hours or days, I can use the refresh token that is still in rclone.conf to get a new token and refresh token manually. Once I paste this into the same rclone.conf, rclone continues working fine until the new token expires.

What I noticed:

  • When using the built in automated config, the GDrive token does NOT have a refresh token but does have an "expiry" timestamp. When getting my own token via my script, I do not have "expiry" timestamp, only an "expires_in" seconds integer, e.g. 3600. Should I add an "expiry" timestamp to my json?
  • Rclone does not seem to need the "expiry" timestamp for OneDrive and refreshes the token fine.
  • I need to make sure that I do not have any unnecessary spaces in my json, and that it is formatted as one line for any of this to work.
  • Rclone seems to use http://localhost:53682/ at times and http://127.0.0.1:53682/ at other times. This inconsistency requires the registered app to accept both redirect URIs. (Also, Microsoft does not seem to like http with the loopback ip, it enforces https)
  • When "reconnecting" the remote via automated rclone config, rclone always sends me to reauthenticate the app. Should it not try and refresh automatically?

May be important:

  • I have a broader scope for my GDrive token than what is required for rclone to work (see config)
  • I am specifying a lot of additional config flags in my remote config than what is bare necessity, but I don't see how this could impact this behaviour.
  • I was specifying my own token urls in the config, but this did not seem to change the behaviour.

Suggestions:

  • Ideally I would like to see exactly what is sent and where upon rclone trying to refresh. This would help immensely with troubleshooting, especially since I can use the refresh token from rclone.conf and refresh the token myself just fine
  • When rclone fails to refresh the token and I am doing a simple "rclone lsd remote:" There is no relevant output in the console, not even with -vv. This is mildly annoying, as I always need to look in the logs to verify. A message would be nice, especially since this is critical failure.
  • I am fine with writing a watchdog type script that would refresh the tokens manually (tried that and it works), however upon a long job, rclone does not seem to automatically load the updated config, so my job fails after one hour regardless. Live config update would be already a great help.

Thanks in advance for any assistance. I highly appreciate it.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.1
- os/version: debian 11.6 (64 bit)
- os/kernel: 5.10.164.1-1.cm1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rlcone lsd MYGdrive:

The rclone config contents with secrets removed.

[MYGdrive]
type = drive
token = {"access_token":"REDACTED","expires_in":3599,"refresh_token":"REDACTED","scope":"https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/drive.labels https://www.googleapis.com/auth/drive.photos.readonly https://www.googleapis.com/auth/drive.labels.readonly https://www.googleapis.com/auth/admin.directory.user.readonly https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/admin.directory.user https://www.googleapis.com/auth/drive.metadata https://www.googleapis.com/auth/drive.admin.labels openid https://www.googleapis.com/auth/drive.file https://www.googleapis.com/auth/devstorage.write_only https://www.googleapis.com/auth/drive.appdata https://www.googleapis.com/auth/drive.install https://www.googleapis.com/auth/docs https://www.googleapis.com/auth/drive.readonly https://www.googleapis.com/auth/drive.metadata.readonly","token_type":"Bearer","id_token":"REDACTED"}
checksum = True
client_id = REDACTED
client_secret = REDACTED
fast_list = True
chunk_size = 256M
upload_cutoff = 250M
scope = drive
server_side_across_configs = True
no_versions = False
encoding = InvalidUtf8
skip_shortcuts = False
skip_dangling_shortcuts = False
export_formats = docx,xlsx,pptx,svg
allow_import_name_change = True
service_account_credentials = 
team_drive = 
auth_owner_only = False
use_trash = True
copy_shortcut_content = True
skip_gdocs = False
skip_checksum_gphotos = False
shared_with_me = False
trashed_only = False
starred-only = False
use_created_date = False
use_shared_date = False
acknowledge_abuse = False
keep_revision_forever = False
stop_on_upload_limit = False
stop_on_download_limit = False
hash_types = md5

A log from the command with the -vv flag

2023/04/27 09:43:05 Failed to create file system for "MYGdrive:": couldn't find root directory ID: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials
2

This is what the oauth library is expecting, yes.

I expect this is the problem you are having.

The providers are somewhat picky about redirect urls. Some accept http://localhost/ and some like http://127.0.0.1/ - that is why rclone uses different ones. It is possible that the provider's policy has change since the rclone oauth was implemented though!

rclone config reconnect doesn't try a refresh - it assumes that has failed.

Use -vv --dump bodies to see the HTTP transactions

You should get an auth error?

My question would be - why are you writing a python script to do stuff rclone already does? You can script rclones config setup very easily.

3

You should get an auth error?

You're right, just confirmed on my local machine. It does so for my local machine, but not for the remote one. This may have something to do with how I configure logging. No worries.

My question would be - why are you writing a python script to do stuff rclone already does? You can script rclones config ?setup very easily.

Ah, yes :slight_smile: . Well, I am using rclone for a bigger project for which I need broader scopes. I was hoping to make use of the same token for rclone and my additional functionality that I need and even make use of rclone automatically refreshing the token so that I can always grab it from the same config. It would be tedious and a security vulnerability if I had to maintain and secure more than one token per remote. That is why I am using my own script to authenticate the repo.

rclone config reconnect doesn't try a refresh - it assumes that has failed.

Right, understood, thank you. One question though, why does the prompt ask me if I already have a token and if I want to refresh? I think I just got confuse by the wording. My apologies.

Already have a token - refresh?
y) Yes (default)
n) No
y/n>

The providers are somewhat picky about redirect urls. Some accept http://localhost/ and some like http://127.0.0.1/ - that is why rclone uses different ones. It is possible that the provider's policy has change since the rclone oauth was implemented though!

Understood. I assume the inconsistency is not across a single repo type, but rather between different types, which is fine. Thank you.

Use -vv --dump bodies to see the HTTP transactions

Thank you! This gave me the following (notice, I am now checking on my local machine).

jansandorski@MacBook-Pro-Jan mygreatcloud-remote-auth % rclone lsd MYgdrive: -vv --dump bodies
2023/04/27 13:51:10 DEBUG : rclone: Version "v1.62.1" starting with parameters ["rclone" "lsd" "MYgdrive:" "-vv" "--dump" "bodies"]
2023/04/27 13:51:10 DEBUG : Creating backend with remote "MYgdrive:"
2023/04/27 13:51:10 DEBUG : Using config file from "/Users/jansandorski/.config/rclone/rclone.conf"
2023/04/27 13:51:10 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/04/27 13:51:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/04/27 13:51:10 DEBUG : HTTP REQUEST (req 0x14000a8e700)
2023/04/27 13:51:10 DEBUG : GET /drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.62.1
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.20.2 gdcl/0.112.0
Accept-Encoding: gzip

2023/04/27 13:51:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/04/27 13:51:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/04/27 13:51:10 DEBUG : HTTP RESPONSE (req 0x14000a8e700)
2023/04/27 13:51:10 DEBUG : HTTP/1.1 401 Unauthorized
Transfer-Encoding: chunked
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Thu, 27 Apr 2023 11:51:10 GMT
Server: ESF
Www-Authenticate: Bearer realm="https://accounts.google.com/", error="invalid_token"
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

1fb
{
  "error": {
    "code": 401,
    "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "errors": [
      {
        "message": "Invalid Credentials",
        "domain": "global",
        "reason": "authError",
        "location": "Authorization",
        "locationType": "header"
      }
    ],
    "status": "UNAUTHENTICATED"
  }
}

0

2023/04/27 13:51:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/04/27 13:51:10 Failed to create file system for "MYgdrive:": couldn't find root directory ID: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials

Config for MYgdrive:

jansandorski@MacBook-Pro-Jan mygreatcloud-remote-auth % cat /Users/jansandorski/.config/rclone/rclone.conf

[MYgdrive]
type = drive
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED
scope = drive
token = {"access_token":"REDACTED","expires_in":3599,"refresh_token":"REDACTED","scope":"https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/drive.labels https://www.googleapis.com/auth/drive.photos.readonly https://www.googleapis.com/auth/drive.labels.readonly https://www.googleapis.com/auth/admin.directory.user.readonly https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/admin.directory.user https://www.googleapis.com/auth/drive.metadata https://www.googleapis.com/auth/drive.admin.labels openid https://www.googleapis.com/auth/drive.file https://www.googleapis.com/auth/devstorage.write_only https://www.googleapis.com/auth/drive.appdata https://www.googleapis.com/auth/drive.install https://www.googleapis.com/auth/docs https://www.googleapis.com/auth/drive.readonly https://www.googleapis.com/auth/drive.metadata.readonly","token_type":"Bearer","id_token":"REDACTED"} 
team_drive =

I grabbed my refresh token straight from the above config file and used postman to get a new token like this:

import requests

url = "https://oauth2.googleapis.com/token"

payload='grant_type=refresh_token&refresh_token=REDACTED&client_id=REDACTED.apps.googleusercontent.com&client_secret=REDACTED'
headers = {
  'Content-Type': 'application/x-www-form-urlencoded'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

(this is python, but I got the response straight in Postman. same outcome though)
I then pasted the new json response that I got into rclone.conf, and now it works fine again.
Here is my new json:

{ "access_token": "REDACTED", "expires_in": 3599, "scope": "https://www.googleapis.com/auth/drive.metadata.readonly https://www.googleapis.com/auth/drive.photos.readonly https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/admin.directory.user https://www.googleapis.com/auth/drive.labels.readonly https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/drive.labels https://www.googleapis.com/auth/drive.metadata https://www.googleapis.com/auth/admin.directory.user.readonly openid https://www.googleapis.com/auth/drive.file https://www.googleapis.com/auth/devstorage.write_only https://www.googleapis.com/auth/drive.appdata https://www.googleapis.com/auth/drive.install https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/docs https://www.googleapis.com/auth/drive.readonly https://www.googleapis.com/auth/drive.admin.labels", "token_type": "Bearer", "id_token": "REDACTED" }

Notice, how it no longer has a new refresh token, which is normal.

Questions:
How does rclone refresh the token? Does it update the json string? Does it store the new access token somewhere else? If it updates the json string in the rclone.conf file, where does it store the original refresh token? Does the "access_token" become the "refresh_token"? What values does it need in the json string to be able to refresh the token fine?

I appreciate greatly your help. What you wrote already gave me better understanding. I hope the above makes sense and you can push me even further :). Thanks

4

Ok. I did a bit more digging. Please confirm if I got this right.

  1. Rclone requires three values to be in the json string
  • the "access_token"
  • the "refresh_token"
  • the "expiry" value in the following format:
    "expiry":"2023-04-26T12:02:52.984891+02:00"

You mentioned this in your previous response. I'm sorry for having missed this.

I added the "expiry" value in the GDrive json and made sure there is a refresh token as well. This seems to now allow rclone to refresh the token itself.

I will confirm and let it expire a couple of times, but it seems to have solved the problem.

Once again, thank you for your patience and assistance.

5

Yes that is a little confusing given that there are two types of token, the bearer token and the refresh token! Oauth is very complicated :slight_smile:

Rclone uses the refresh token to get a new access token.

Yes it updates the config file and this may include a new bearer token and a new refresh token. (Some providers the refresh token only works once).

It stores whatever refresh token it got back in the config file.

No. Rclone gets a new access and refresh token when the expiry is past.

This should be enough

{
  "access_token": "XXX",
  "token_type": "Bearer",
  "refresh_token": "XXX",
  "expiry": "2023-04-27T11:23:01.095252814+01:00"
}

I hope that helps!

6

Yes, I was able to confirm that for the tokens to get refreshed by rclone, they need the mentioned values. I was missing the "expiry" value as it is not part of the token response by default, it needs to be calculated and programmatically added if I want to alter rclone.conf by some custom script.

Thanks. This can be closed now.

1 Like
7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.