Rclone authorize, for google drive, when the headless rclone is the ONLY rclone

We (rsync.net) are trying to write a comprehensive and detailed "export gmail" workflow.

We're very close to finished and all of the "hard stuff" (google takeout, OAuth, IDs, scoping, etc.) is done.

The problem is the very last step in configuring the remote. All of this work is invoking 'rclone' over SSH, like this:

ssh user@rsync.net rclone config ...

... so it is, indeed, a headless use-case. Therefore our answer to "Use auto config?" is "N(o)".

... but then we are given the 'rclone authorize "drive" "sdkjfksdjskdj .....' and there is no place to run that. There are no other install locations or other clients - the only client is the headless one.

I understand that the thinking here is that the end user will just run that 'rclone authorize' command on their laptop or whatever but in this use-case that does not exist.

The only rclone binary available is the headless one.

So, if headless rclone is all a user has access to, how can that output from "Use auto config?" be reformatted, or transformed or ... (?) ... in order to complete that step ?

Thank you.

I don't think it can as Google deprecated any of the old stuff and you have to use the OAuth2 flow now which requires a browser prompt so you can't be headless. The only way around that would be configuring a service account I'd imagine.

This gives details on that part:

Out-Of-Band (OOB) flow Migration Guide | Google Identity | Google Developers

Thanks for your response.

I wonder if we can go a bit deeper here - if only for future readers / archives ...

I believe the method I am using is, indeed, OAuth - and as I mentioned, I get to the point where rclone provides a challenge/response URL:

rclone authorize "drive" "eyJsdfsdfsdfsdfsdzc0NjI

... why can't I cut and paste some form of that challenge URL into any old browser, like my laptop, and then paste back a response to rclone ?

...

Looking beyond this, I use a lot of service APIs - like twilio, or sendgrid, or AWS for instance, and it consists of a very simple workflow where I copy a generated API key from (twilio.com, for instance) into my script and that's that ... why is it that google drive (a consumer, non-technical, end-user product) does not have a simple API mechanism ?

What we're doing here with creating projects and service domains and security scopes ... all of this makes sense, but not for a trivial use-case like this.

What is going on here with google drive ? I am basically spinning up an enterprise level software development workflow, with publishing to millions of users, just to connect to my own gdrive ...

If you were to run rclone authorize, it needs to log you in via the browser to get a token back:

rclone authorize drive
2022/08/29 16:19:43 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=4qsY4KzG1kxJXR1L2X71mg
2022/08/29 16:19:43 NOTICE: Log in and authorize rclone for access
2022/08/29 16:19:43 NOTICE: Waiting for code...

So it's working based on the pre-approved redirect URI that's there and you must sign in and finally you'll get a code back. You can't skip those steps to my knowledge as any old way to do that was since deprecated.

Many apps I use force a browser window like Spark or something like that.

If you were rolling out enterprise wide, you'd probably use a service account as one option or you'd have to package up the process and the user must auth per Google's requirements. I'll use Spark as an example if I deploy that to folks, they still have to follow the oAuth process from Google to login which is a bit annoying but that's Google.

I'm not sure offhand how rclone would address Google's requirements without doing what it does, but I'm not a developer either so I might be misunderstanding or missing something unfortunately.

Again, thank you.

Let me digress a bit - I remember there is (was ?) an option to use the rclone group authentication, but the downside is that it is shared globally and so you are throttled and have contention issues with other global users ...

But could we (rsync.net) create our own identity, like the rclone project did, and share that with just our users ?

Would that compress this workup/setup process greatly ?

To my understanding no as it's not a client ID/secret thing, it's the user auth connecting to your account.

You'd have to use the same process if you'd use the default client ID/secret in rclone already.

The client ID/secret is more for the API use rather than the oAuth process.

The service account goes around that as you don't need to auth at all as it just uses the service account. Other than using a service account, I'm not aware of any other way to bypass the oAuth requirements from Google.

did you update Rclone to the actually version?
I got the same problem before and after a update all is running well.

I have no idea what you mean and how it relates to this topic.

Hmmm ...

It sounds like setting up a service account is a more appropriate workflow for something like this.

But I wonder, if someone exports their personal gmail to google takeout, can you use an (unrelated) service account with rclone to download that takeout file ?

EDIT: we're interested in paying someone to help us develop this workflow ... if you're interested, just email us ... the eventual HOWTO would be generalized to ALL rclone users and would be public for the entire community ...

Perhaps some other Google users can chime in as I personally migrated away from Google to Dropbox so I don't use it / have it anymore.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.