Unable to use on-fly backend with configuration string in shell

What is the problem you are having with rclone?

Calling any rclone command with on-fly backend with configuration string containing a token fail

What is your rclone version (output from rclone version)

% rclone version
rclone v1.56.0

  • os/version: darwin 11.5.2 (64 bit)
  • os/kernel: 20.6.0 (x86_64)
  • os/type: darwin
  • os/arch: amd64
  • go/version: go1.16.6
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

google, dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone about :dropbox,token='{"access_token"\:"long_access_token","token_type"\:"bearer","refresh_token"\:"long_refresh_token","expiry"\:"2021-09-21T19:50:47.165097+02:00"}':

The rclone config contents with secrets removed.

i need no persistant config file to be used

A log from the command with the -vv flag

2021/09/21 19:56:46 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "about" "-vv" ":dropbox,token={\"access_token\"\\:\"long_access_token\",\"token_type\"\\:\"bearer\",\"refresh_token\"\\:\"long_refresh_token\",\"expiry\"\\:\"2021-09-21T19:50:47.165097+02:00\"}:"]
2021/09/21 19:56:46 DEBUG : Creating backend with remote ":dropbox,token={\"access_token\"\\:\"long_access_token\",\"token_type\"\\:\"bearer\",\"refresh_token\"\\:\"long_refresh_token\",\"expiry\"\\:\"2021-09-21T19:50:47.165097+02:00\"}:"
2021/09/21 19:56:46 DEBUG : :dropbox: detected overridden config - adding "{lPMdu}" suffix to name
2021/09/21 19:56:46 DEBUG : Using config file from "/Users/mcamier/.config/rclone/rclone.conf"
2021/09/21 19:56:46 Failed to create file system for ":dropbox,token={\"access_token\"\\:\"long_access_token\",\"token_type\"\\:\"bearer\",\"refresh_token\"\\:\"long_refresh_token\",\"expiry\"\\:\"2021-09-21T19:50:47.165097+02:00\"}:": failed to configure dropbox: invalid character '\\' after object key
1 Like

I think @albertony can chime in as I feel I'm close but missing something.

This link talks about using double quotes for items you want to enclose that have quotes in them:

I was testing a bit and I can't quite get the token part to work properly myself (I have a Mac as well and was testing there).

I feel I'm missing something silly as you are on a path but you are escaping the quotes with the backslash and it should doubled up on quotes per that note.

I'll goof around a bit more once I get time as well.

Shell quoting, and json token string.. :weary:
Ran a quick test myself, but were not able to get it right.

I think you should not escape the colons?

rclone about :dropbox,token='{"access_token":"long_access_token",

Instead of:

rclone about :dropbox,token='{"access_token"\:"long_access_token",

That's what I was thinking as well as I tried that as it stops with a JSON error on the token so something else must be messing it up as I get:

drive: failed when making oauth client: failed to create oauth client: unexpected end of JSON input

I am using Google Drive as I don't have a dropbox account but concept should be the same.

I tried many ways of escaping or not colons and curly brackets, or using quotes and doubles quotes but always end up with errors :frowning:

Shell-quoting of rclone-quoted connection strings might be as confusing as Nolan's Inception.

Let me propose a strategy that looks easier on beholder's eyes, IMHO...

  1. Let's write down the json token as it is:

We want to put it in single or double quotes because its json contains commas so json commas stay inside json instead of being treated as commas between connection string parts. Let's save single quotes for shell quoting. We'll put the token inside double-quotes at this stage. The connection string rules let us do it IFF we.....

  1. ...Double the double-quotes "" (quadruple?) in the token (btw., this CSV-style "" looks arguably better than shell-style \" IMO:
  1. Surround quadruply-quotified token from step 2 in a "pair of double quotes" to build up the connection string (note the connect string comma between dropbox and token):
  1. Time to write down the shell command line by surrounding the remote argument in 'single quotes' we saved above:
rclone about ':dropbox,token="{""access_token"":""long_access_token"",""token_type"":""bearer"",""refresh_token"":""long_refresh_token"",""expiry"":""2021-09-21T19:50:47.165097+02:00""}":'

Shell will pass everything between single quotes (i.e. the remote string from step 3) unchanged to rclone who will in turn split the connect string by outside commas and unwrap/undouble CSV quotes in the token part.

It's just one of many strategies. Use if you like.

I have an idea to slightly extend the connect string parser in rclone to accept backticks as a 3rd type of quotes to keep single quotes for shell and still use double quotes in json tokens without backslashing or doubling, like

rclone about ':dropbox,token=`{"access_token":"long_access_token","token_type":"bearer","refresh_token":"long_refresh_token","expiry":"2021-09-21T19:50:47.165097+02:00"}`:'

This looks better and should work fine in any unix shell and in cmd on windows. Powershell won't benefit as it uses backtick as an escape symbol (but I guess we can double the backtick in powershell).

@ncw @albertony
Do u like? Just an idea for the future...

This solution works fine, thanks for your help :hugs:

Nice explanation! :+1:

Regarding backticks: I think it is a good idea to have an alternative to the double- or escaped double quotes. Backticks looks good, but its a bit unfortunate its not compatible with PowerShell.

Another thing is that the outer single quotes will not work in windows cmd, and you need to escape double quotes - or triple (sextuple?) them - to prevent them from being "eaten". Unfortunately I don't have a good setup for testing this, but it seems the following are examples that work:

rclone about ":dropbox,token=\"{\"\"access_token\"\":\"\"long_access_token\"\",\"\"token_type\"\":\"\"bearer\"\",\"\"refresh_token\"\":\"\"long_refresh_token\"\",\"\"expiry\"\":\"\"2021-09-21T19:50:47.165097+02:00\"\"}\":"
rclone about ":dropbox,token='{\"access_token\":\"long_access_token\",\"token_type\":\"bearer\",\"refresh_token\":\"long_refresh_token\",\"expiry\":\"2021-09-21T19:50:47.165097+02:00\"}':"
rclone about :dropbox,token='{\"access_token\":\"long_access_token\",\"token_type\":\"bearer\",\"refresh_token\":\"long_refresh_token\",\"expiry\":\"2021-09-21T19:50:47.165097+02:00\"}':
rclone about :dropbox,token='{"""access_token""":"""long_access_token""","""token_type""":"""bearer""","""refresh_token""":"""long_refresh_token""","""expiry""":"""2021-09-21T19:50:47.165097+02:00"""}':

Now, even with backticks I think you would need the escaping/tripling, to get the double quotes that json requires?

rclone about :dropbox,token=`{\"access_token\":\"long_access_token\",\"token_type\":\"bearer\",\"refresh_token\":\"long_refresh_token\",\"expiry\":\"2021-09-21T19:50:47.165097+02:00\"}`:
rclone about :dropbox,token=`{"""access_token""":"""long_access_token""","""token_type""":"""bearer""","""refresh_token""":"""long_refresh_token""","""expiry""":"""2021-09-21T19:50:47.165097+02:00"""}`:

Hmm, yes, isn't quoting fun!

I put the " and the ' in for precisely this reason - namely sometimes you need more than one level of quote. I was imagining Windows people would mostly use " for the outer level and ' for the inner level and Unix people the reverse, but of course there is JSON and its " use to mess up that scheme!

Adding ` is an interesting idea for one more level.... Not 100% backwards compatible though. To make it backwards compatible it would need to be something which currently gives a syntax error. Note also that using ` inside " in Unix is a recipe for fun times!

[ Aside: I always enjoyed perl's solution to this - you could make up your own quotes with qw so qw|potato| if the char you chose came in pairs then you used the other one to close so qw(potato). ]

I think, on balance, we should leave it as-is since the solution you came up with @ivandeex works fine and doubling up quotes will be familiar to Windows CLI users at least. (It wasn't until I wrote the parser for the config strings that I realised why doubling the quotes to escape them is good - it is really easy to parse!).

I meant to use both ` and " inside ' which is fine on Unix

or python's idea of triple quotes: ''' or """

well ok...

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.