Download a file from Google Drive that requires authentication

ivandeex · May 18, 2021, 3:35pm

Curl uses netscape cookie jar format, this guy too:

ivandeex · May 18, 2021, 3:41pm

Offtopic

The core difference between a cookie jar file and a headers text file is mutability. Whenever a program updates cookies, it updates the jar file. That's why the flag is --cookie-jar and not --cookies-from.

By the way, rclone.conf is a mixed format which contains both user-controlled settings and programmatically updated tokens.

Probably we could add a new boolean flag (Mutable or Volatile or Token) to config.Option ?

Having an optional --tokens-file /path/to/file.ext (or --mutable-state or --rclone-jar or whatever) would close a number of github requests, namely:

ncw · May 19, 2021, 11:14am

Yes that would help a lot. Maybe --config-rw and --config? Rclone could store mutable config and tokens in ~/.cache/rclone easily too... I like the simplicity of the config file for most users though - I'd see that as being specialized. The cookie jar could easily go in .cache too.

ivandeex · May 19, 2021, 1:23pm

Good idea, let me elaborate.

Rclone will by default (without any additional flags) create in ~/.cache/rclone (or in a location set by --cache-dir) two new "dynamic" files, named for example tokens (in the INI format like rclone.conf but only containing changed tokens in the same sections as rclone.conf) and cookies (in the netscape cookie jar format).
tokens will have the same INI format as rclone.conf with sections and values, but keep only tokens. Sections without tokens will be skipped/ignored.
Tokens in "static" ~/.config/rclone/rclone.conf (or wherever config search finds it) are left as is. Rclone just gives precedence to tokens in the "dynamic" file, if found.
rclone config and rclone config authorize remote: will still write token to "static" file but if it finds the same token in the same section in the "dynamic" file, it will additionally ask user whether she agrees to prune it (it's an interactive command, in the end).
Custom locations can be set with --config-rw=PATH/FILE and --cookie-jar=PATH/FILE.
Probably we need a special flag value ``, --cookie-jar=none if someone wants to have legacy cookie-related and token-related behavior, i.e. tokens written to rclone.conf, cookies written nowhere.
Rclone will know what options should go to tokens by looking at the boolean field config.Option.Mutable (or whatever name you like). If a non-mutable option value is found in token, rclone will ignore it, probably issuing a warning.
Safety. Whenever rclone updates a "dynamic" file, e.g. obtains new token, it saved previous state (if any) as tokens~ (with tilde and probably a timestamp).
Users wishing to use custom cookies will overwrite ~/.cache/rclone/cookies or run rclone as rclone ... --cookie-jar /path/to/cookies/saved/from/curl saving a backup copy in advance (because rclone can overwrite this file, should be mentioned in the docs).

WARNING!
I made a few edits to the text.
The last edit was at 13:43 UTC.
Sorry!

ncw · May 19, 2021, 1:39pm

I agree 100% with cookies. Tokens I think needs a flag - or maybe rclone could check for the existence of the tokens file?

I think the format should be exactly rclone.conf

Note that there are other things other than tokens which get stored, eg whether an sftp backend supports md5sum or a whole pile of stuff for some backends!

Works for me

This needs some thought doesn't it... Your proposal sounds good but maybe trickier to implement.

I don't see a problem storing cookies, but a way to disable them might be a good idea.

I was assuming that any writes that backends do after rclone config would go into the extra config file?

Good idea! We should probably do this for the config file too.

I don't want to break the copy the config file and you've got all your config simplicity of rclone by default.

ivandeex · May 19, 2021, 1:44pm

WARNING!
I made a few edits to the text.
The last edit was at 13:43 UTC.
Please re-read my comment. Sorry!

ncw · May 19, 2021, 1:59pm

It looks like I replied to a very late copy of the message so I think my reply stands above!

ivandeex · May 19, 2021, 2:11pm

Do you think this deserves a PR?
Or maybe we can start from a github issue, discuss open items, and then proceed to PR?
Or split this into 2 issues/PRs, one for tokens, another for cookies?

ivandeex · May 19, 2021, 2:15pm

These settings are not mutable.
rclone only reads them from rclone.conf
Why keep a copy?

To summarize:

You propose a full file override
I propose a "diff"-like INI with only changed values (and additionally allowed to be changed by the Option flag, i.e. known in advance)

My proposal is trickier, but implementable like follows:

for reading: introduce a new config value source with new priority
for writing: don't know yet, will think

ncw · May 19, 2021, 5:18pm

At some point!

I think #3655 will do for the config file. I think we need a new issue for the cookies. The cookies one should be quite straight forward compared to #3655

They get set by the backend though, not as part of rclone config

github.com

rclone/rclone/blob/71f75a1d95a0154a6d77e725e097ed733a1e2db9/backend/sftp/sftp.go#L1165-L1168

    
      
          	if changed {
          		f.m.Set("md5sum_command", f.opt.Md5sumCommand)
          		f.m.Set("sha1sum_command", f.opt.Sha1sumCommand)
          	}

No I propose a diff also. It needs to be not just for token, it needs to be for anything which the backend sets.

I think that if we introduce a new config source for reading that would be OK.

For writing we always write to the diff file, unless we are running rclone config (or some other config action triggered via the API). We can set a flag in the context to show we are running a config action.

By default we don't have a diff file unless --immutable-config (or some other flag name yet to be determined) is set. We can have a build tag for nixOS users

An idea: it might be best to let the configfile abstraction handle this?

ivandeex · May 19, 2021, 5:57pm

Ah! Later I also came to mean this "anything backend wants to change and persist".

I additionally wanted to have backend mark all such options with Option.Mutable.
This should allow a concrete config engine behind the configfile interface know in advance that such a setting is special. Usages of this may vary: warn user if a non-mutable setting is found in the diff, throw error if backend tries to Put a non-mutable setting, and so on.

ncw · May 21, 2021, 8:13am

Good idea... The only downside might be the occasions where the backend "upgrades" the config which sometimes involves reading and setting variables that are normally considered non-mutable. What do you think should happen here?

ivandeex · May 21, 2021, 8:41am

Per-backend internal flags FullConfigOwner or fs.RegisterBackend argument?
(names tentative as I'm being commuting with my cell in hands, expect some typo fixes soon btw)

What should be done for unkown options in rw-config, needs analysis of such backends. Can you give an example?

ncw · May 21, 2021, 9:08am

I think if we stick to rclone config and anything descended from it writes in original config file, otherwise everything writes to diff file then I think it will cover all the cases. For a backend "upgrade" we can tell the user to run rclone config reconnect

Unknown options need to be settable - they get set in the backends in the config phase sometimes.

git grep m.Set -- backend/ gives the list of potential offenders that aren't oauth token saving.

system · May 24, 2021, 9:09am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.