Sharepoint Entra ID SingleTenant Client ID and Secret - No Username- Token Expires in 1 hour

[Mikero (Mike)]

What is the problem you are having with rclone?

I want to be able to refresh my token before the hour time limit expires without kill and relaunching the rclone process.

We dont want to use a username for login. We want to just use the client ID and Secret.

I can create the token manually using powershell code below and then i can update the config file and it will run for an hour.

Since this issue isn't resolved yet (see articles at bottom of this code), is there a way i can send an update command to Rclone after the config is updated so i can update the conifg file with the new token until the issue below is resolved?

i currently can run a script that updates the config with new token every 55 minutes but then I have to klll the rclone process and relaunch.

url = "https://login.microsoftonline.com/my company tenant id/oauth2/v2.0/token"
$headers = @{
    "Content-Type" = "application/x-www-form-urlencoded"
}
$body = @{
    client_id = "my company provided ID"
    client_secret = "my company provided secret"
    grant_type = "client_credentials"
    scope = "https://graph.microsoft.com/.default"
}

# Send the request and store the response
$response = Invoke-RestMethod -Uri $url -Method POST -Headers $headers -Body $body

# Extract the token from the response
$token = $response.access_token

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

• os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
• os/kernel: 10.0.17763.5458 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.21.6
• go/linking: static
• go/tags: cmount

Yes

Which cloud storage system are you using? (eg Google Drive)

Sharepoint

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount remote: x:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[remote]
type = onedrive
client_id = Provided id
client_secret = provided secret
no_versions = true
link_scope = organization
av_override = true
auth_url = https://login.microsoftonline.com/2My company tennat id/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/My company tennat id/oauth2/v2.0/token
token = XXX
drive_id = XXX
drive_type = documentLibrary
hash_type = sha256
### Double check the config for sensitive info before posting publicly

Im having the issue described in these closed articles here:

Sharepoint (Onedrive) single tenant with a shared secret

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them. What is the problem you are having with rclone? I registered a single tenant application in the Azure portal. Now I am able to fetch the access key without entering my personal credential and by using client_id with this command. This is called as Access token request with a shared secret by Microsaf…

and here:

RCLONE without username and password for MS Sharepoint remote

Rclone prints a URL when it opens the browser for auth. You can copy and paste this into an incognito window if you need different auth than you have in your main browser window.

I realize this issue isn't resolved or at least i dont see it resolved in the github trail but i was wondering if there is temporary workaround or hack that I can implement in the mean time to update the token.

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

@Mikero

is there a way i can send an update command to Rclone after the config is updated so i can update the conifg file with the new token

what about rclone config update

[Mikero (Mike)]

didn't work,

maybe i ran the wrong update command, i tried

rclone config update remote env_auth=true config_refresh_token=false --non-interactive

i also tried rclone about remote: after updating and that doesn't seem to work either.

[asdffdsa (jojothehumanmonkey)]

"Each time that happens rclone will print or DEBUG a message saying how to affect the value taken."
so use -vv for debug output

and need to figure out which field to change?

perhaps something like
rclone config update remote config_refresh_token=false token=xxx -vv

[Mikero (Mike)]

I think you've got me in the right direction but i must be doing something wrong:
rclone config update remote config_refresh_token=false token=$token -vv

rclone : 2024/03/06 20:47:29 DEBUG : rclone: Version "v1.65.2" starting with parameters ["C:\rclone\rclone.exe" "config" "update" "remote" "config_refresh_token=false" "token={eyJ0eXAiOiJKV1QiLCJub25jZSI6IktVUDUycUd0Qjc4VTVREST OF TOKEN}" "`vv"]

2024/03/06 20:50:14 DEBUG : remote: config out: out=, err=failed to configure OneDrive: invalid character 'e' looking for beginning of object key string
Error: failed to configure OneDrive: invalid character 'e' looking for beginning of object key string
Usage:
rclone config update name [key value]+ [flags]

the invalid character seems to be first the letter of token because if i change the token to this:

{"token_type":"Bearer","expires_in":3599,"ext_expires_in":3599,"access_token":"eyJ0eXAiOiJKV1QiLCJub25jZSI6IktVUDUycUStart of tken}" "vv"]

I get:

2024/03/06 20:53:36 DEBUG : remote: config out: out=&{State:choose_type Option: OAuth: Error: Result:}, err=
2024/03/06 20:53:36 DEBUG : remote: config in: state="choose_type", result=""
2024/03/06 20:53:36 DEBUG : remote: config out: out=, err=failed to configure OneDrive: invalid character 't' looking for beginning of object key string
Error: failed to configure OneDrive: invalid character 't' looking for beginning of object key string
Usage:

[asdffdsa (jojothehumanmonkey)]

try to post the full debug logs, not just snippets.

@Mikero

I think you've got me in the right direction but i must be doing something wrong:
rclone config update remote config_refresh_token=false token=$token -vv

really, not sure,
might need convert the token into base64 encoded JSON blob

for my onedrive account, this is what token field looks like
token = {"access_token":"","token_type":"Bearer","refresh_token":"","expiry":""}

[Mikero (Mike)]

maybe it's the order of the names.. but that's weird because I'm able to successfully use the JSON Blob when running config with user interaction and it works fine but it doesn't work when I do it this way :
Without a variable

 rclone config update remote config_refresh_token=false token="{"token_type":"Bearer","expires_in":3599,"access_token":"XXX_MYTOKEN","refresh_token":null}"   -vv
2024/03/07 17:37:18 DEBUG : rclone: Version "v1.65.2" starting with parameters ["C:\\rclone\\rclone.exe" "config" "update" "remote" "config_refresh_token=false" "token={token_type:Bearer,expires_in:3599,access_token:XXX_MyToken,refresh_token:null}" "-vv"]
2024/03/07 17:37:18 DEBUG : Using config file from "C:\\Users\\mrolivares-mda\\AppData\\Roaming\\rclone\\rclone.conf"
2024/03/07 17:37:18 DEBUG : Saving config "token" in section "remote" of the config file
2024/03/07 17:37:18 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/03/07 17:37:18 DEBUG : remote: config in: state="", result=""
2024/03/07 17:37:18 DEBUG : remote: config out: out=&{State:*oauth,choose_type,, Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/03/07 17:37:18 DEBUG : remote: config in: state="*oauth,choose_type,,", result=""
2024/03/07 17:37:18 DEBUG : Override value found, choosing value "false" for state "*oauth-confirm,choose_type,,"
2024/03/07 17:37:18 DEBUG : remote: config out: out=&{State:*oauth-confirm,choose_type,, Option:<nil> OAuth:<nil> Error: Result:false}, err=<nil>
2024/03/07 17:37:18 DEBUG : remote: config in: state="*oauth-confirm,choose_type,,", result="false"
2024/03/07 17:37:18 DEBUG : remote: config out: out=&{State:*oauth-done,choose_type,, Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/03/07 17:37:18 DEBUG : remote: config in: state="*oauth-done,choose_type,,", result=""
2024/03/07 17:37:18 DEBUG : remote: config out: out=&{State:choose_type Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/03/07 17:37:18 DEBUG : remote: config in: state="choose_type", result=""
2024/03/07 17:37:18 DEBUG : remote: config out: out=<nil>, err=failed to configure OneDrive: invalid character 't' looking for beginning of object key string
Error: failed to configure OneDrive: invalid character 't' looking for beginning of object key string
Usage:
  rclone config update name [key value]+ [flags]

Flags:
      --all               Ask the full set of config questions
      --continue          Continue the configuration process with an answer
  -h, --help              help for update
      --no-obscure        Force any passwords not to be obscured

and with one:

rclone config update remote config_refresh_token=false token=$jsonBlob -vv
2024/03/07 17:37:27 DEBUG : rclone: Version "v1.65.2" starting with parameters ["C:\\rclone\\rclone.exe" "config" "update" "remote" "config_refresh_token=false" "token={token_type:Bearer,expires_in:3599,access_token:XXX_mytoken,refresh_token:null}" "-vv"]
2024/03/07 17:37:27 DEBUG : Using config file from "C:\\Users\\mrolivares-mda\\AppData\\Roaming\\rclone\\rclone.conf"
2024/03/07 17:37:27 DEBUG : Saving config "token" in section "remote" of the config file
2024/03/07 17:37:27 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/03/07 17:37:27 DEBUG : remote: config in: state="", result=""
2024/03/07 17:37:27 DEBUG : remote: config out: out=&{State:*oauth,choose_type,, Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/03/07 17:37:27 DEBUG : remote: config in: state="*oauth,choose_type,,", result=""
2024/03/07 17:37:27 DEBUG : Override value found, choosing value "false" for state "*oauth-confirm,choose_type,,"
2024/03/07 17:37:27 DEBUG : remote: config out: out=&{State:*oauth-confirm,choose_type,, Option:<nil> OAuth:<nil> Error: Result:false}, err=<nil>
2024/03/07 17:37:27 DEBUG : remote: config in: state="*oauth-confirm,choose_type,,", result="false"
2024/03/07 17:37:27 DEBUG : remote: config out: out=&{State:*oauth-done,choose_type,, Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/03/07 17:37:27 DEBUG : remote: config in: state="*oauth-done,choose_type,,", result=""
2024/03/07 17:37:27 DEBUG : remote: config out: out=&{State:choose_type Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/03/07 17:37:27 DEBUG : remote: config in: state="choose_type", result=""
2024/03/07 17:37:27 DEBUG : remote: config out: out=<nil>, err=failed to configure OneDrive: invalid character 't' looking for beginning of object key string
Error: failed to configure OneDrive: invalid character 't' looking for beginning of object key string
Usage:
  rclone config update name [key value]+ [flags]

Flags:
      --all               Ask the full set of config questions
      --continue          Continue the configuration process with an answer
  -h, --help              help for update
      --no-obscure        Force any passwords not to be obscured
      --non-interactive   Don't interact with user and return questions
      --obscure           Force any passwords to be obscured
      --result string     Result - use with --continue
      --state string      State - use with --continue

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2024/03/07 17:37:27 Fatal error: failed to configure OneDrive: invalid character 't' looking for beginning of object key string

[asdffdsa (jojothehumanmonkey)]

at this point, the only thing i can suggest is to post the exact tokens and blobs, just change a single character in the middie. do not delete or add, just change.

maybe with that extra info, someone else more experience with tokens can comment.

[Mikero (Mike)]

when i run

rclone config update remote config_refresh_token false token $jsontoken

it strips the " parenthesis out of my token. I can copy and past the token manually and it does the same thing. I can tell because when i go into the config it strips them out.

When i copy and paste the token the interactive format it takes it just fine and keeps the double quotes. How can i get it to keep the double quotes when passing the token through the update command.

I tried from a command prompt and powershell with the same outcome. I tried escaping the "" and still nothing.

[Mikero (Mike)]

I figured out a work around to my problem. not sure how well this will work but this is what I'm doing:

I created a service using NSSM that launches a .bat file that launches my script. I changed the service settings to allow up to 60 seconds for a clean exit and to restart application on exit.

my script updates the rclone config file with the new token
then runs

Start-Process -FilePath "C:\rclone\rclone.exe" -ArgumentList "sync", "\\My\Network\Path", "remote:Sharepoint_Storage, "--ignore-checksum", "--ignore-size", "--backup-dir=remote:RcloneBackupDir", "--config=C:\rclone\rclone.conf", "--max-duration=58m" -NoNewWindow -Wait
exit

When the application exits after 58 minutes it exits the script and causes the service to restart, regenerate the token and resume the sync process.

Initial test seemed to work weI'll se how it goes.

[jansan (Jan Sandorski)]

@asdffdsa

rclone config update

Hi Mike,

I am very much interested in rclone allowing for client_credentials type OAuth flow. I read the issues on Github and found at least 3 other topics on the forum that all promised some updates, but to no avail. Maybe I myself might get interested in developing this, who knows.

In the meantime, I got interested in your interim solution and have a few questions.

1. I assume on the fly token update would not work as rclone loads the config once in the beginning and then disregard changes, right? I was hoping to update the token using a custom script, like you do, but not interrupt a sync process. Hoping for rclone to just retry the config load upon first failed attempt.

2. If you are essentially restarting rclone every 58 minutes, how do you handle recovery? My data scanning may easily exceed an hour and may therefore never reach some parts of the data. Any ideas? I have some, but they involve a ton of custom work.

Let me know your thoughts.

[Mikero (Mike)]

1. I couldn't find a way for RCLONE to refresh the configuration without stopping and restarting. I recently ran into the issue where it just keeps scanning the same files in the hour it's running and doesn't make it through the whole collection. Thus it's not copying the last 10% of the data. I had to bring my transfers down to 1 because i was getting throttled but i'm gong to try increasing the checkers to see if that speeds up the file scan and it can make it through the all the files. Not sure if this will cause throttling but I plan to test it this week. If you find a way to refresh the config without relaunching Rclone please let me know.

2. Everything i pretty much said in 1. I'm going to try and increase checkers and see if that can scan the entire data set without throttling in 58 minutes.

@ncw By any chance have y'all had a chance to look at this?

Oh and the Rclone Config Update command doesn't work to refresh the config once it's running at least not for me.. It would be great if i could just recreate the config and refresh rclone without stopping it. My script recreates the file without issue. I just can't get it to refresh without a start and stop. This would also be a fix for me if we could just "refresh" instead of a stop and start.

[Mikero (Mike)]

i increased the checkers and it cleared everything.. i did get throttled once but it cleared all the files