How does client ID work, exactly? Can it time out?

What is the problem you are having with rclone?

My problem is actually a question. Yesterday I decided to resume my push of data to google drive from console in stead of the window manager and that worked initially well. I had discovered --progress and it is a life saver. After a few hours I started getting errors and it was only creating errors from that point on. The errors I got referred to the OAuth2 authorization. I have it coupled to my own client ID. So I ended the process and studied the log and it appears to have been a sudden disruption in authorization after hours of uploading. So I restarted the window manager (gnome / wayland) and reconnected my process of rclone to my personal client ID using rclone config. After that I could resume without problem. I have not had a problem since.

So my question is: does the disruption of OAuth2 have something to do with running from console in stead of gnome? Or does that happen periodically any way?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here

rclone is designed to run from console, including oauth2 and updating tokens.

beyond that, hard to give an exact answer, as you did not answer any of the questions.

I expected it to be a general problem or just connected to the console. As it is not I will answer all the questions. Thanks for the reply. c(EDIT it will take a while as I am working from a different machine now.)

No, probaby not.

It shouldn't happen - rclone should refresh the token before it expires.

If the time and timezone on your computer isn't accurate this could cause the problem. It might also just be one of those things!

I cannot seem to edit my post, so I add the unanswered queries here.

Run the command 'rclone version' and share the full output of the command.

gbonnema  ~  rclone version  1
rclone 1.60.1

  • os/version: fedora 37 (64 bit)
  • os/kernel: 6.1.18-200.fc37.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: dynamic
  • go/tags: none
    gbonnema  ~ 

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I was running a script as follows:


#!/usr/bin/env bash
# Push data changes to google

read -p "pushing data to google, are you sure you want to overwrite google? (y/n): " answer

if [ "y" = "$answer" ]
then
	read -p "are all applications closed? (y/n): " closed
	if [ "y" = "$closed" ]
	then
		dtstring="$(date +%F)_$(date +%T)"
		tmpfile=$(mktemp --suffix=.sync "$dtstring"_pull_"$HOSTNAME"_XXXXXX)
		echo sync to google is about to start
		rclone sync /ddisk-home ggl:ddisk-home --progress --log-file $tmpfile -v
		logdir="/ddisk-home/rclone/logs"
		echo "cp $tmpfile $logdir"
		cp $tmpfile $logdir
		echo "created $logdir"
		rclone sync /ddisk-home/rclone ggl:ddisk-home/rclone --progress -v
	fi
fi

echo push data to google finished
exit

The rclone config contents with secrets removed.

[ggl]
type = drive
client_id = <hidden>
client_secret = <hidden>
scope = drive
token = {"access_token":"<hidden>"}
team_drive = 

[emi]
type = drive
client_id = <hidden>
client_secret = <hidden>
scope = drive.file
token = {"access_token":"<hidden>"}
team_drive = 

A log from the command with the -vv flag

Unfortunately I don't have -vv logs, just the logs I got from regular logging, as it was an incident. Hopefully not repeatable.
I print the start of the file, and then the piece where it starts to go wrong (and continues to go wrong from that point on).


2023/03/16 08:15:03 NOTICE: gbonnema/video_local_fins_music: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/video: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/talen: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/books: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/20-woonruimte: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/Music: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/yle: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/.steam.pid: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/.multirust: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/.steampath: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:03 NOTICE: gbonnema/onderhoud: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:06 INFO  : rclone/logs/2023-03-14_16:58:40_pull_Dg8125.sync: Copied (new)
2023/03/16 08:15:06 INFO  : rclone/logs/2023-03-15_06:49:15_pull_yizukT.sync: Copied (new)
2023/03/16 08:15:08 NOTICE: gbonnema/.kde/cache-maxima: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:11 NOTICE: gbonnema/.kde/share/icons/hicolor/128x128/apps/chrome-cnciopoikihiagdjbjpnocolokfelagl-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:11 NOTICE: gbonnema/.kde/share/icons/hicolor/128x128/apps/chrome-nmmhkkegccagdldgiimedpiccmgmieda-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:11 NOTICE: gbonnema/.kde/share/icons/hicolor/128x128/apps/chrome-pjkljhegncpnkpknbcohdijeoejaedia-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:11 NOTICE: gbonnema/.kde/share/icons/hicolor/128x128/apps/chrome-apdfllckaahabafndbhieahigkjlhalf-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:11 NOTICE: gbonnema/.mozilla/firefox/wh85dheo.default-1612442844822/lock: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:12 NOTICE: gbonnema/.kde/share/icons/hicolor/16x16/apps/chrome-cnciopoikihiagdjbjpnocolokfelagl-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:12 NOTICE: gbonnema/.kde/share/icons/hicolor/16x16/apps/chrome-nmmhkkegccagdldgiimedpiccmgmieda-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:12 NOTICE: gbonnema/.kde/share/icons/hicolor/48x48/apps/chrome-cnciopoikihiagdjbjpnocolokfelagl-Default.png: Can't follow symlink without -L/--copy-links
2023/03/16 08:15:16 INFO  : gbonnema/.config/nautilus/desktop-metadata: Copied (new)
2023/03/16 08:15:18 INFO  : gbonnema/.npm/_cacache/index-v5/3b/1c/80eab9a72a034242c8a2dda410ca7d5be85f232a851643ae678a76b52292: Copied (new)
2023/03/16 08:15:18 INFO  : gbonnema/.config/nautilus/search-metadata: Copied (new)
......
2023/03/16 14:39:47 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-03-13/ovpn_udp/nl300.nordvpn.com.udp.ovpn: Copied (new)
2023/03/16 14:39:47 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-01-30/ovpn_udp/us2023.nordvpn.com.udp.ovpn: Copied (new)
2023/03/16 14:39:47 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-04-19/ovpn_tcp/nl93.nordvpn.com.tcp.ovpn: Copied (new)
2023/03/16 14:39:47 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-02-15/ovpn_tcp/us1823.nordvpn.com.tcp.ovpn: Copied (new)
2023/03/16 14:39:48 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-02-22/ovpn_tcp/us1427.nordvpn.com.tcp.ovpn: Copied (new)
2023/03/16 14:39:48 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-02-22/ovpn_udp/uk405.nordvpn.com.udp.ovpn: Copied (new)
2023/03/16 14:39:49 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-02-15/ovpn_udp/us1568.nordvpn.com.udp.ovpn: Copied (new)
2023/03/16 14:39:49 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-03-13/ovpn_tcp/uk331.nordvpn.com.tcp.ovpn: Copied (new)
2023/03/16 14:39:50 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-03-13/ovpn_udp/nl301.nordvpn.com.udp.ovpn: Copied (new)
2023/03/16 14:39:50 ERROR : gbonnema/Downloads/nordvpn/vpn-files/2019-02-22/ovpn_tcp/us1428.nordvpn.com.tcp.ovpn: Failed to copy: couldn't list directory: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials
2023/03/16 14:39:50 INFO  : gbonnema/Downloads/nordvpn/vpn-files/2019-01-30/ovpn_udp/us2024.nordvpn.com.udp.ovpn: Copied (new)
2023/03/16 14:39:50 ERROR : gbonnema/Downloads/nordvpn/vpn-files/2019-02-22/ovpn_udp/uk406.nordvpn.com.udp.ovpn: Failed to copy: couldn't list directory: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials
2023/03/16 14:39:50 ERROR : gbonnema/Downloads/nordvpn/vpn-files/2019-02-15/ovpn_udp/us1569.nordvpn.com.udp.ovpn: Failed to copy: couldn't list directory: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials
...... 
errors until the end (I interrupted the process)

Well this shouldn't happen, but it obviously did! I guess the question is, does it keep happening or was it a once off?

Well ...... (crossing my fingers, knock on wood, jump through hoops and hope to die), I have been uploading almost continuously now and it has not happened yet. So, maybe it was a one-off. Either that, or it is connected to running from a console (i.e. not from any window manager). If it changes I will let you know. Thanks all for the replies!

1 Like

Well, not so fast. Today, it happened again from my laptop instead of from my PC. Although the error occurred on my laptop, the PC was also effected, so I refreshed the client ID and secret and connected them anew after which the error was gone on the PC. I had to do the same on the laptop to be able to use rclone. I will copy a few lines from the log, which are comparable to the last log. Remark though that this error occurs when I run jobs over multiple days. After aproximately 2 days the error occurs. I am not fully sure this is the identifying circumstance, but it is a common characteristic between the 2 occurrences.

2023/03/24 07:38:05 INFO : projects/c/gcc/gcc/testsuite/gcc.dg/gimplefe-error-6.c: Copied (new)
2023/03/24 07:38:05 INFO : Documents/00-GuusBonnema/10-alle-werk-zaken/ex-werkgevers/10-capgemini/00-history-PA-PT/PA/projects/SPE-probe/.metadata/.plugins/org.eclipse.core.runtime/.settings/org.eclipse.debug.ui.prefs: Copied (new)
2023/03/24 07:38:05 INFO : Documents/00-GuusBonnema/10-alle-werk-zaken/ex-werkgevers/10-capgemini/Outlook-mbox/SENT/000-SENT/000-SENT/701.msg: Copied (new)
2023/03/24 07:38:06 ERROR : projects/c/gcc/gcc/testsuite/gcc.c-torture/compile/pr26213.c: Failed to copy: failed to open source object: open file failed: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See ’Google साइन-इन’ को अपने वेब ऐप्लिकेशन में इंटिग्रेट करना  |  Authentication  |  Google Developers.
More details:
Reason: authError, Message: Invalid Credentials
2023/03/24 07:38:06 ERROR : Documents/ibackup/backup-paris/01-permanent-backup/home/abonnema/.mozilla/abonnema/5hvowxx2.slt/Cache/288AD0B5d01: Failed to copy: failed to open source object: open file failed: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See ’Google साइन-इन’ को अपने वेब ऐप्लिकेशन में इंटिग्रेट करना  |  Authentication  |  Google Developers.
More details:

I really hope this gives enough information.

That sounds like something to do with your account. Did it stop working at the same time on your PC and laptop? I presume they aren't sharing a config file. Did your client ID get banned or something?

About the banning, I don´t know how to see that. I do know I could refresh without problems. Both laptop and PC have their own config, no share. Is there an option in my (google) account that I should be checking?

EDIT; I am now continuing the download without problems. The account did not get any communication from google, and I presume banning would not go unnoticed, so I do not think it was banned (also I have no problems after, which is the same as last time). Is it possible that the OAuth2 expired? Remember in both cases it happened after several days of uploading and downloading.

After you do the inital rclone config or rclone config reconnect rclone gets two tokens from google, an access token and a refresh token.

The access token expires every hour and rclone refreshes it with the refresh token. That has clearly been working since your transfers have been longer than 1 hour.

It is possible the refresh token expired and this is a symptom of client_ids which haven't been put into production I think.

From the drive docs:

Keeping the application in "Testing" will work as well, but the limitation is that any grants will expire after a week, which can be annoying to refresh constantly. If, for whatever reason, a short grant time is not a problem, then keeping the application in testing mode would also be sufficient.

yes, I did not put the app into production. If that is necessary I will, but at the moment I do not oversee the consequences of putting into production. I believe google has to approve it? If that is the cause, which seems plausible, then I will put it into production. I will check the docs for the consequence of putting into production (I assume this has to be google documentation, not rclone documentation?). Thanks for the response. I am starting to understand the symptoms.

i always put my gdrive client ids into production, never had an issue in many years.
never once, had to have goole approve it, never a delay in being able to use the client id.
and that is commonly done by other forum members.

you can create a new client id, put that into production and test.

I probably just have a lack of knowledge and a lack of experience with putting client ID's in production.

My first assumption was thinking that I was putting my project in production and from the formulation of @asdffdsa I gather you just put the ID in production.

The second assumption was that it looks like everyone can use my client ID and that freaked me out.

For these reasons I chose for testing.

Ok, so the first assumption really is a misunderstanding, it is not the project, but the client ID one puts into production. How about the second assumption: is that a misunderstanding too?

Thanks for helping me so far, I will probably get this sorted by the time I get my client ID into production. Still, I would like to understand the second part of my misunderstandings .....

well, it is your client id, only you have access to it.

Yeah, it is mine, the question is whether others can find it. Anyway, I will put it into production once my download is complete. I assume the "putting into production" process will disrupt the ongoing download?

Thanks, will point to the answer that saved the day, so others can enjoy its knowledge too. Speaking of knowledge, should this tidbit be in rclone's man page too? Maybe it is already, it is pretty extensive, I could have missed it.

Thanks all!

I just put the app into published status, but the verification status indicates "verification is not necessary". I wonder whether verification is necessary to prevent the kind of problem I had here. Any advice?

P.S. Google calls this "publish app" which is why I thought the app was being published.
P.S1. I have a screenprint of the current status, but I don't think I can include it here. Let me know if I can.

verification is not required, i have never done it, most users do not.

now, use the published app client id for a few days.

Thanks, I just published the same old client ID I had created before and up to now it just works. Tonight I will do a sync with my basic data sets and see what happens. Should take about 4 hours.

Thanks for your assistence!

1 Like

I have done it several times with the rclone client IDs. It's quite a painful process so avoid if you can!

1 Like