Unable to use on-fly backend with configuration string in shell

Mickael_Camier · September 21, 2021, 5:59pm

What is the problem you are having with rclone?

Calling any rclone command with on-fly backend with configuration string containing a token fail

What is your rclone version (output from `rclone version`)

% rclone version
rclone v1.56.0

os/version: darwin 11.5.2 (64 bit)
os/kernel: 20.6.0 (x86_64)
os/type: darwin
os/arch: amd64
go/version: go1.16.6
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

google, dropbox

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone about :dropbox,token='{"access_token"\:"long_access_token","token_type"\:"bearer","refresh_token"\:"long_refresh_token","expiry"\:"2021-09-21T19:50:47.165097+02:00"}':

The rclone config contents with secrets removed.

i need no persistant config file to be used

A log from the command with the `-vv` flag

2021/09/21 19:56:46 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "about" "-vv" ":dropbox,token={\"access_token\"\\:\"long_access_token\",\"token_type\"\\:\"bearer\",\"refresh_token\"\\:\"long_refresh_token\",\"expiry\"\\:\"2021-09-21T19:50:47.165097+02:00\"}:"]
2021/09/21 19:56:46 DEBUG : Creating backend with remote ":dropbox,token={\"access_token\"\\:\"long_access_token\",\"token_type\"\\:\"bearer\",\"refresh_token\"\\:\"long_refresh_token\",\"expiry\"\\:\"2021-09-21T19:50:47.165097+02:00\"}:"
2021/09/21 19:56:46 DEBUG : :dropbox: detected overridden config - adding "{lPMdu}" suffix to name
2021/09/21 19:56:46 DEBUG : Using config file from "/Users/mcamier/.config/rclone/rclone.conf"
2021/09/21 19:56:46 Failed to create file system for ":dropbox,token={\"access_token\"\\:\"long_access_token\",\"token_type\"\\:\"bearer\",\"refresh_token\"\\:\"long_refresh_token\",\"expiry\"\\:\"2021-09-21T19:50:47.165097+02:00\"}:": failed to configure dropbox: invalid character '\\' after object key

Animosity022 · September 21, 2021, 7:12pm

I think @albertony can chime in as I feel I'm close but missing something.

This link talks about using double quotes for items you want to enclose that have quotes in them:

github.com/rclone/rclone

Connection strings for on the fly backend configuration

opened 05:07PM - 03 Feb 21 UTC

closed 03:20PM - 11 May 21 UTC

ncw

enhancement refactoring

## Current system The current syntax for a `remote` is [described in the docs…](https://rclone.org/docs/#syntax-of-remote-paths), but in summary it is 1. `/path/to/dir` - This refers to the local file system. 2. `remote:path/to/dir` - This refers to a `remote:` as defined in the config file 3. `:backend:path/to/dir` - Create a remote on the fly for `backend` This system has two main problems: 1) Currently when using option 3 if you want to provide configuration for the backend created, then you need to supply command line options, eg `--http-url`, or the corresponding environment variable `RCLONE_HTTP_URL`. This works fine for command line use, but does not work well over the API as there is nowhere to pass in the additional options. 2) When using option 2 or 3 if there are two remotes named on the command line then the options (eg `--http-url` or `RCLONE_HTTP_URL`) apply to **both** remotes. This often causes confusion for users when they are for example trying to set `--drive-shared-with-me` in a server side copy on the source only. This does not work as the user expected rclone copy --drive-shared-with-me drive:source drive:dest ## Proposed replacement To fix these problems it is proposed to turn the current syntax into more of a connection string. So the syntax would become something like (note exact syntax is discussed later) 1. `/path/to/dir` - This refers to the local file system. 2. `remote:path/to/dir` - This refers to a remote: as defined in the config file 3. `:backend:path/to/dir` - Create a remote on the fly for backend 4. `remote,option1=value1,option2=value2:path/to/dir` - As option 2 but with additional config (example syntax) 5. `:backend,option1=value1,option2=value2:path/to/dir` - As option 3 but with additional config (example syntax) This would turn the current syntax into more of a connection string. This would fix the API problem above as wherever an `fs` parameter was called for one of these connection strings could be used instead `remote,option1=value1,option2=value2:` `:backend,option1=value1,option2=value2:`. It would also fix the command line problem, the example above would become rclone copy drive,shared_with_me=true:source drive:dest ### Options The options in the connection string (`option1=value`) would be as expressed in the config file. So to set the parameter `--drive-shared-with-me` you would use `shared_with_me=true`. The docs already explain each value and how to set it in the config file. ### Connection string syntax Above is an example syntax for the connection string. The option string needs to be parse unambiguously into backend or remote name, options and a path. Currently the parser is quite simple - it just looks for the first colon `:`. The connection string also needs to be easy for users to use (not require too much escaping in the shell) as they will create them by hand. The connection string needs to be able to express every possible connection string. These goals are in tension, so some thought will be required! For use over the API, the best solution might be to allow the `fs` parameter to be a JSON blob as well as a string. A JSON blob maps very easily into all the parameters necessary. The character `:` is often used in URLs, and character `=` is often used in base64 data so it would be nice to be able to express these in a simple fashion without escaping, eg `:http,url=https://rclone.org/,no_head=true:` The option names come from a limited character set of `[a-z0-9_]+` Some example syntaxes, easy for users to type: 1. `:http,url=https\://rclone.org/,no_head=true:` - backquote characters - Traditional but backquotes are annoying to add on unix CLIs as they need to be escaped - Neither `,` nor `:` need escaping in unix shells 2. `:http,url="https://rclone.org/",no_head=true:` - make it so arguments can optionally be quoted - easy to parse but again annoying on unix CLIs as you'd need to put the double quotes in single quotes. - Can't remember how you escape quotes on Windows CLI - it is possible - Maybe allow single quotes too which would make life easier for Windows users 3. `:http{url=https://rclone.org/}{no_head=true}:` 4. `:http[url=https://rclone.org/][no_head=true]:` - use pair of delimiters - means you can use the '{' '}' delimiters in the argument as long as they match in pairs - both `{}` and `[]` have interpretations in the unix shell so this means any parameter use would need to be quoted 5. Database style connection string - each parameter must end in `;` - `:http;url=https://rclone.org/;no_head=true;:` - `;` is a good character to use since it is rare in rclone configs (only in an azureblob connection string in my config!) - `;` needs escaping in the unix shell so this means any parameter use would need to be quoted - Will be familiar to DB users - Perhaps error prone to have a `;:` at the end instead of just `:` If a setting is a boolean value, then no `=` sign would be the same as `true` so you could write `drive,shared_with_me:` instead of `drive,shared_with_me=true:` Over the API sending JSON would probably be easiest, so allow the `fs` parameter to be a JSON blob: - `{"type":"http","url":"https://rclone.org/","no_head":"true"}` - `{"_parent":"remote","url":"https://rclone.org/","no_head":"true"}` - This needs to be able to override an existing backend also, perhaps have a `_parent` parameter for that? - `{"type":"http","url":"https://rclone.org/","no_head":"true","_root=":"path/to/root"}` - The `_root` parameter if present would specify the root that the backend is using - perhaps have a `_name` parameter to name the remote. #### Prior art Connection strings for databases seem to be like - `Data Source=c:\mydb.db;Version=3;Pooling=True;Max Pool Size=100;` Using a trailing `;` stops `:` in the values being a problem. From: https://docs.microsoft.com/en-us/dotnet/api/system.data.oracleclient.oracleconnection.connectionstring?view=netframework-4.8

I was testing a bit and I can't quite get the token part to work properly myself (I have a Mac as well and was testing there).

I feel I'm missing something silly as you are on a path but you are escaping the quotes with the backslash and it should doubled up on quotes per that note.

I'll goof around a bit more once I get time as well.

albertony · September 21, 2021, 8:43pm

Shell quoting, and json token string..
Ran a quick test myself, but were not able to get it right.

I think you should not escape the colons?

rclone about :dropbox,token='{"access_token":"long_access_token",

Instead of:

rclone about :dropbox,token='{"access_token"\:"long_access_token",

Animosity022 · September 21, 2021, 8:55pm

That's what I was thinking as well as I tried that as it stops with a JSON error on the token so something else must be messing it up as I get:

drive: failed when making oauth client: failed to create oauth client: unexpected end of JSON input

I am using Google Drive as I don't have a dropbox account but concept should be the same.

Mickael_Camier · September 21, 2021, 9:09pm

I tried many ways of escaping or not colons and curly brackets, or using quotes and doubles quotes but always end up with errors

ivandeex · September 21, 2021, 9:35pm

Shell-quoting of rclone-quoted connection strings might be as confusing as Nolan's Inception.

Let me propose a strategy that looks easier on beholder's eyes, IMHO...

Let's write down the json token as it is:

{"access_token":"long_access_token","token_type":"bearer","refresh_token":"long_refresh_token","expiry":"2021-09-21T19:50:47.165097+02:00"}

We want to put it in single or double quotes because its json contains commas so json commas stay inside json instead of being treated as commas between connection string parts. Let's save single quotes for shell quoting. We'll put the token inside double-quotes at this stage. The connection string rules let us do it IFF we.....

...Double the double-quotes "" (quadruple?) in the token (btw., this CSV-style "" looks arguably better than shell-style \" IMO:

{""access_token"":""long_access_token"",""token_type"":""bearer"",""refresh_token"":""long_refresh_token"",""expiry"":""2021-09-21T19:50:47.165097+02:00""}

Surround quadruply-quotified token from step 2 in a "pair of double quotes" to build up the connection string (note the connect string comma between dropbox and token):

:dropbox,token="{""access_token"":""long_access_token"",""token_type"":""bearer"",""refresh_token"":""long_refresh_token"",""expiry"":""2021-09-21T19:50:47.165097+02:00""}":

Time to write down the shell command line by surrounding the remote argument in 'single quotes' we saved above:

rclone about ':dropbox,token="{""access_token"":""long_access_token"",""token_type"":""bearer"",""refresh_token"":""long_refresh_token"",""expiry"":""2021-09-21T19:50:47.165097+02:00""}":'

Shell will pass everything between single quotes (i.e. the remote string from step 3) unchanged to rclone who will in turn split the connect string by outside commas and unwrap/undouble CSV quotes in the token part.

It's just one of many strategies. Use if you like.

ivandeex · September 21, 2021, 9:49pm

I have an idea to slightly extend the connect string parser in rclone to accept backticks as a 3rd type of quotes to keep single quotes for shell and still use double quotes in json tokens without backslashing or doubling, like

rclone about ':dropbox,token=`{"access_token":"long_access_token","token_type":"bearer","refresh_token":"long_refresh_token","expiry":"2021-09-21T19:50:47.165097+02:00"}`:'

This looks better and should work fine in any unix shell and in cmd on windows. Powershell won't benefit as it uses backtick as an escape symbol (but I guess we can double the backtick in powershell).

@ncw @albertony
Do u like? Just an idea for the future...

Mickael_Camier · September 22, 2021, 7:11am

This solution works fine, thanks for your help

albertony · September 22, 2021, 7:20am

Nice explanation!

Regarding backticks: I think it is a good idea to have an alternative to the double- or escaped double quotes. Backticks looks good, but its a bit unfortunate its not compatible with PowerShell.

Another thing is that the outer single quotes will not work in windows cmd, and you need to escape double quotes - or triple (sextuple?) them - to prevent them from being "eaten". Unfortunately I don't have a good setup for testing this, but it seems the following are examples that work:

rclone about ":dropbox,token=\"{\"\"access_token\"\":\"\"long_access_token\"\",\"\"token_type\"\":\"\"bearer\"\",\"\"refresh_token\"\":\"\"long_refresh_token\"\",\"\"expiry\"\":\"\"2021-09-21T19:50:47.165097+02:00\"\"}\":"
rclone about ":dropbox,token='{\"access_token\":\"long_access_token\",\"token_type\":\"bearer\",\"refresh_token\":\"long_refresh_token\",\"expiry\":\"2021-09-21T19:50:47.165097+02:00\"}':"
rclone about :dropbox,token='{\"access_token\":\"long_access_token\",\"token_type\":\"bearer\",\"refresh_token\":\"long_refresh_token\",\"expiry\":\"2021-09-21T19:50:47.165097+02:00\"}':
rclone about :dropbox,token='{"""access_token""":"""long_access_token""","""token_type""":"""bearer""","""refresh_token""":"""long_refresh_token""","""expiry""":"""2021-09-21T19:50:47.165097+02:00"""}':

Now, even with backticks I think you would need the escaping/tripling, to get the double quotes that json requires?

rclone about :dropbox,token=`{\"access_token\":\"long_access_token\",\"token_type\":\"bearer\",\"refresh_token\":\"long_refresh_token\",\"expiry\":\"2021-09-21T19:50:47.165097+02:00\"}`:
rclone about :dropbox,token=`{"""access_token""":"""long_access_token""","""token_type""":"""bearer""","""refresh_token""":"""long_refresh_token""","""expiry""":"""2021-09-21T19:50:47.165097+02:00"""}`:

ncw · September 23, 2021, 1:41pm

Hmm, yes, isn't quoting fun!

I put the " and the ' in for precisely this reason - namely sometimes you need more than one level of quote. I was imagining Windows people would mostly use " for the outer level and ' for the inner level and Unix people the reverse, but of course there is JSON and its " use to mess up that scheme!

Adding ` is an interesting idea for one more level.... Not 100% backwards compatible though. To make it backwards compatible it would need to be something which currently gives a syntax error. Note also that using ` inside " in Unix is a recipe for fun times!

[ Aside: I always enjoyed perl's solution to this - you could make up your own quotes with qw so qw|potato| if the char you chose came in pairs then you used the other one to close so qw(potato). ]

I think, on balance, we should leave it as-is since the solution you came up with @ivandeex works fine and doubling up quotes will be familiar to Windows CLI users at least. (It wasn't until I wrote the parser for the config strings that I realised why doubling the quotes to escape them is good - it is really easy to parse!).

ivandeex · September 23, 2021, 4:07pm

I meant to use both ` and " inside ' which is fine on Unix

or python's idea of triple quotes: ''' or """

well ok...

system · September 26, 2021, 4:07pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.