Issue with rclone

Help and Support
1

Hello,

We initiated an inquiry on behalf of CyberPanel regarding the integration of rclone with our backup system. However, we have encountered issues with Google Drive backups. A corresponding issue has been opened with Rustic, and further details can be found here: Problem using Google drive via rclone · Issue #1119 · rustic-rs/rustic · GitHub.

Following instructions provided by the Rustic team, we conducted thorough testing and confirmed that their configuration is functioning properly. It appears that the issue stems from rclone. Additional details can be found below.

[root@dockertesting CyberCP]# python plogical/test.py
{'files': [{'id': '1IV1Ve1ZL2omlU14hNWZnur11B9cMKH_L', 'name': 'backup-n8n.wpmautic.net-02.11.2024_00-00-11.tar.gz'}, {'id': '1m_GJAzVsOFUrPYa4scV-pezzgcgO4cwR', 'name': '02.11.2024_00-00-10'}, {'id': '1ZP5JD6F9ZngvvrivRAuwBvRzVyF4i0_M', 'name': 'backup-n8n.wpmautic.net-02.08.2024_10-26-17.tar.gz'}, {'id': '1lNrJzzvqOyy37HH7yTpy0r5m2021xOsN', 'name': 'backup-wpmautic.net-02.08.2024_10-26-07.tar.gz'}, {'id': '13c4Y7nUfY2RUpIyqk9V3449rMCBOFGVY', 'name': '02.08.2024_10-26-06'}, {'id': '1JTkofG4p0fsvk0rcB5aC1bV14m_6FkBX', 'name': 'backup-n8n.wpmautic.net-02.08.2024_05-43-42.tar.gz'}, {'id': '1TlVfjK71DBBdL6ulc5fvGNwYpIlHdcmG', 'name': 'backup-wpmautic.net-02.08.2024_05-43-32.tar.gz'}, {'id': '1YBPSmSPvEnfnXrehK2FGTGUhB0oeMg1t', 'name': '02.08.2024_05-43-31'}, {'id': '168mbyO-q_StOleYHHDGEqS_FAn0ojb7e', 'name': 'usmakolkaritest-95.217.248.69'}, {'id': '1kYe_AHKhA-K9KfZs6ONIkq521IHsVL90', 'name': 'backup-cyberpanel.fun-06.15.2020_10-39-50.tar.gz'}]}

(CyberCP) [root@dockertesting CyberCP]# sudo -u wpmau6540 rclone ls unasirdrive:~
2024/04/18 11:59:26 Failed to create file system for "unasirdrive:~": couldn't find root directory ID: Get "https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true": couldn't fetch token: unauthorized_client: if you're using your own client id/secret, make sure they're properly set up following the docs
(CyberCP) [root@dockertesting CyberCP]#

2

welcome to the forum,

so far, not seeing an issue with rclone.

when you posted, you were asked to answer a template of questions ?
please answer all the questions ???

3

I will chip in to add more information. The first command

[root@dockertesting CyberCP]# python plogical/test.py

We've use to verify that google token is working properly using the python code.

Our rlcone config file

[unasirdrive]
type = drive
scope = drive
token = {"access_token":"","token_type":"Bearer","refresh_token":"","expiry":"2024-04-08T21:53:00.123456789Z"}
team_drive =

I've taken refresh and access token out of this for security. It used to work fine just few days ago but now with same config it gives error both while using with rustic and directly with rclone as well

error is as below

(CyberCP) [root@dockertesting CyberCP]# sudo -u wpmau6540 rclone ls unasirdrive:~
2024/04/18 11:59:26 Failed to create file system for "unasirdrive:~": couldn't find root directory ID: Get "https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true": couldn't fetch token: unauthorized_client: if you're using your own client id/secret, make sure they're properly set up following the docs
4

please understand, we are just volunteers, cannot see into your machine.
so, please answer all the questions without having to keep asking over again, ok?

perhaps, just start over from scratch, re-create the gdrive remote using rclone config

5

We've set up the remote multiple times and I think pretty much cleared everything (note: we manually set up the remote, but we checked with rclone cli too). Here it is again

Note: The config present in rclone file is verified by checking via python code (i mean the access and refresh token works)

The full command you’re attempting to use.

sudo -u wpmau6540 rclone ls unasirdrive:~

A logfile of rclone’s output with personal information removed. If it is large, you can use services like pastebin.com . It’s usually helpful to increase the logging with -v or -vv depending.

2024/04/18 13:10:26 Failed to create file system for "unasirdrive:~": couldn't find root directory ID: Get "https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true": couldn't fetch token: unauthorized_client: if you're using your own client id/secret, make sure they're properly set up following the docs

The rclone config you’re using. If you don’t know where to find it, check here. Before posting ensure you’ve removed any confidential information like credentials.

[unasirdrive]
type = drive
scope = drive
token = {"access_token":"","token_type":"Bearer","refresh_token":"","expiry":"2024-04-08T21:53:00.123456789Z"}
team_drive =

What version of rclone you’re using. It’s also helpful to try rclone with the latest beta if you’re using a stable release to understand if your issue was recently fixed.

rclone v1.66.0
- os/version: almalinux 8.9 (64 bit)
- os/kernel: 4.18.0-305.el8.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none
6

test rclone on the root of gdrive, not a subfolder.
rclone lsd unasirdrive: -vv

post a debug log, not just a one-line snippet.

7 
[root@dockertesting ~]# sudo -u wpmau6540 rclone lsd unasirdrive: -vv
2024/04/18 13:23:08 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "lsd" "unasirdrive:" "-vv"]
2024/04/18 13:23:08 DEBUG : Creating backend with remote "unasirdrive:"
2024/04/18 13:23:08 DEBUG : Using config file from "/home/wpmautic.net/.config/rclone/rclone.conf"
2024/04/18 13:23:08 DEBUG : unasirdrive: Loaded invalid token from config file - ignoring
2024/04/18 13:23:08 DEBUG : unasirdrive: got fatal oauth error: oauth2: "unauthorized_client" "Unauthorized"
2024/04/18 13:23:08 Failed to create file system for "unasirdrive:": couldn't find root directory ID: Get "https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true": couldn't fetch token: unauthorized_client: if you're using your own client id/secret, make sure they're properly set up following the docs
[root@dockertesting ~]#

Interesting that the same token works using the python code.

8

as a test, might run the command as root user, without sudo -u wpmau6540.

9 
(CyberCP) [root@dockertesting CyberCP]# rclone lsd unasirdrive: -vv --config=/home/wpmautic.net/.config/rclone/rclone.conf
2024/04/18 13:29:04 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "lsd" "unasirdrive:" "-vv" "--config=/home/wpmautic.net/.config/rclone/rclone.conf"]
2024/04/18 13:29:04 DEBUG : Creating backend with remote "unasirdrive:"
2024/04/18 13:29:04 DEBUG : Using config file from "/home/wpmautic.net/.config/rclone/rclone.conf"
2024/04/18 13:29:04 DEBUG : unasirdrive: Loaded invalid token from config file - ignoring
2024/04/18 13:29:04 DEBUG : unasirdrive: got fatal oauth error: oauth2: "unauthorized_client" "Unauthorized"
2024/04/18 13:29:04 Failed to create file system for "unasirdrive:": couldn't find root directory ID: Get "https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true": couldn't fetch token: unauthorized_client: if you're using your own client id/secret, make sure they're properly set up following the docs
(CyberCP) [root@dockertesting CyberCP]#

Just to make sure i use exactly same token/refresh token in the python code and it can list files.

10

how are you creating the rclone remote, using rclone config or what?

what python code?

11

We manually create config using python, but tried via rclone cli too. The python code i used to test token is like this

from google.oauth2.credentials import Credentials
from google.auth.transport.requests import Request
import google.oauth2.credentials
from googleapiclient.discovery import build

def is_token_valid(token):
    credentials = google.oauth2.credentials.Credentials(token['access_token'],
                                                        token['refresh_token'],
                                                        "https://oauth2.googleapis.com/token", None, None,
                                                        "['https://www.googleapis.com/auth/drive.file']")

    drive = build('drive', 'v3', credentials=credentials)
    result = drive.files().list(pageSize=10, fields="files(id, name)").execute()
    print(result)
12

i do something similar with S3 remotes and boto3. create a temporary session token and feed it to rclone.
in your case, there must be some subtle difference, perhaps with encoding.

have you tried it the other way?
create the remote using rclone config and feed that token to the python script.

13

I will check and revert and I appreciate your help with this, thank you.

14

fwiw, my backup script does not use a rclone config file at all.
it passes everything to rclone as environment variables using subprocess.run

15

Just to update it seems to be issue with client_id not being set up.

If I enter my own client_id and client_secret it works.

16

Yes, we need to explicitly set client_id and client_secret maybe because the default client_id of rclone reached its limit.

Leaving this info here, so it could be helpful with anyone in same situation.