What is the correct syntax for an on-the-fly union remote?

dullage · March 8, 2022, 1:22pm

What is the problem you are having with rclone?

I can't work out how to use an "on-the-fly" union remote.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.22000.493 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

N/A, Local Filesystem

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Using the instructions here and here it appears that I can use an on-the-fly union remote with the following command:

rclone ls ":union,upstreams=C:\a C:\b:"

However, I am getting the following response:

2022/03/08 13:15:59 Failed to create file system for ":union,upstreams=C:\\a C:\\b:": union can't point to a single upstream - check the value of the upstreams setting

There must be a mistake in the syntax I'm using but I can't figure out what that mistake is.

The rclone config contents with secrets removed.

N/A - Using an on-the-fly config.

A log from the command with the `-vv` flag

2022/03/08 13:16:26 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\Users\\Adamd\\.local\\bin\\rclone.exe" "ls" ":union,upstreams=C:\\a C:\\b:" "-vv"]
2022/03/08 13:16:26 DEBUG : Creating backend with remote ":union,upstreams=C:\\a C:\\b:"
2022/03/08 13:16:26 DEBUG : :union: detected overridden config - adding "{YJPtf}" suffix to name
2022/03/08 13:16:26 DEBUG : Using config file from "C:\\Users\\Adamd\\.config\\rclone\\rclone.conf"
2022/03/08 13:16:26 Failed to create file system for ":union,upstreams=C:\\a C:\\b:": union can't point to a single upstream - check the value of the upstreams setting

asdffdsa · March 8, 2022, 1:56pm

hello and welcome to the forum,

connection strings can be very confusing

If the parameter has a : or , then it must be placed in quotes " or '

so try rclone ls ":union,upstreams='c:\a c:\b':"

dullage · March 8, 2022, 4:28pm

That's the ticket! Thanks for your help. Now...next question...what to do when there are spaces in the paths. Things I've tried:

rclone ls ":union,upstreams='c:\a 1 c:\b 2':"
2022/03/08 16:21:55 ERROR : : error listing: directory not found

rclone ls ":union,upstreams='""c:\a 1"" ""c:\b 2""':"
Usage:
  rclone ls remote:path [flags]

rclone ls ":union,upstreams=""'c:\a 1' 'c:\b 2'"":"
2022/03/08 16:23:34 Failed to create file system for ":union,upstreams='c:\\a 1' 'c:\\b 2':": expecting `:` or `,` or another quote after a quote

ncw · March 8, 2022, 5:11pm

You will have to put it in "quotes". These in turn will need quoting to protect them from the Windows shell, so something like

rclone ls ":union,upstreams='""c:\a 1"" ""c:\b 2""':"

(untested!)

asdffdsa · March 8, 2022, 5:15pm

hi,
i tried that and it does not work on windows and so far, i cannot figure out how to make this work

rclone ls ":union,upstreams='""c:\a 1"" ""c:\b 2""':" -vv 
Usage:
  rclone ls remote:path [flags]

Flags:
  -h, --help   help for ls

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command ls needs 1 arguments maximum: you provided 4 non flag arguments: [":union,upstreams='\"c:\\a" "1" "c:\\b" "2': -vv"]

albertony · March 8, 2022, 5:24pm

Fun with quotes....

I think this this works (from cmd):

rclone ls ":union,upstreams='"""c:\a 1""" """c:\b 2"""':" -vv

ncw · March 8, 2022, 5:25pm

I thought you doubled up " in double quoted strings to get a single quote? I suppose that probably depends on whether you are using CMD or Powershell. I'm pretty sure that syntax should work for CMD.

There is a comprehensive stack overflow about this here scripting - Escaping Double Quotes in Batch Script - Stack Overflow which seems to indicate that it is up to the called program to parse the "". The result here

[":union,upstreams='\"c:\\a" "1" "c:\\b" "2': -vv"]

Seems to indicate that

":union,upstreams='""c:\a 1"" ""c:\b 2""':"

was parsed as these 4 strings which seems strange...

:union,upstreams='"c:\\a
1
c:\\b
2': -vv

I wonder if it is the go runtime parsing this? In which case it would be worth trying this, which would be Go's way of quoting it.

rclone ls ":union,upstreams='\"c:\a 1\" \"c:\b 2\"':" -vv

Under unix this quoting is handled by the shell and programs never see it.

asdffdsa · March 8, 2022, 5:26pm

rclone ls ":union,upstreams='"""c:\a 1""" """c:\b 2"""':" -vv

that works from both command line and .cmd batch.

asdffdsa · March 8, 2022, 6:07pm

that works command line and .cmd batch script

>rclone ls ":union,upstreams='\"c:\a 1\" \"c:\b 2\"':" -vv
2022/03/08 13:06:35 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" ":union,upstreams='\"c:\\a 1\" \"c:\\b 2\"':" "-vv"]
2022/03/08 13:06:35 DEBUG : Creating backend with remote ":union,upstreams='\"c:\\a 1\" \"c:\\b 2\"':"
2022/03/08 13:06:35 DEBUG : :union: detected overridden config - adding "{YuX2B}" suffix to name
2022/03/08 13:06:35 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2022/03/08 13:06:35 DEBUG : Creating backend with remote "c:/b 2"
2022/03/08 13:06:35 DEBUG : Creating backend with remote "c:/a 1"
2022/03/08 13:06:35 DEBUG : fs cache: renaming cache item "c:/b 2" to be canonical "//?/c:/b 2"
2022/03/08 13:06:35 DEBUG : fs cache: switching user supplied name "c:/b 2" for canonical name "//?/c:/b 2"
2022/03/08 13:06:35 DEBUG : fs cache: renaming cache item "c:/a 1" to be canonical "//?/c:/a 1"
2022/03/08 13:06:35 DEBUG : fs cache: switching user supplied name "c:/a 1" for canonical name "//?/c:/a 1"
2022/03/08 13:06:35 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
2022/03/08 13:06:35 DEBUG : fs cache: renaming cache item ":union,upstreams='\"c:\\a 1\" \"c:\\b 2\"':" to be canonical ":union{YuX2B}:"
        1 a1.txt
        1 b2.txt

dullage · March 8, 2022, 7:50pm

Unfortunately this doesn't work for me in Powerhsell. I get this:

rclone ls ":union,upstreams='"""c:\a 1""" """c:\b 2"""':" -vv
2022/03/08 19:06:45 DEBUG : Using config file from "C:\\Users\\adam\\.config\\rclone\\rclone.conf"
2022/03/08 19:06:45 DEBUG : rclone: Version "v1.55.1" starting with parameters ["C:\\Users\\adam\\.local\\bin\\rclone.exe" "ls" ":union,upstreams=' c:\\a 1" "c:\\b 2':" "-vv"]
Usage:
  rclone ls remote:path [flags]

The same story with this one in Powershell as well:

rclone ls ":union,upstreams='\"c:\a 1\" \"c:\b 2\"':" -vv
2022/03/08 19:45:50 DEBUG : Using config file from "C:\\Users\\adam\\.config\\rclone\\rclone.conf"
2022/03/08 19:45:50 DEBUG : rclone: Version "v1.55.1" starting with parameters ["C:\\Users\\adam\\.local\\bin\\rclone.exe" "ls" ":union,upstreams='\\" "c:\\a" "1\\ \\c:\\b" "2\\':" "-vv"]
Usage:
  rclone ls remote:path [flags]

Both however work in Command Prompt which is fine for what I need . Thanks for all the help.

albertony · March 8, 2022, 8:31pm

I suspected that.

This variant seem to work from PowerShell:

rclone ls ':union,upstreams=''\"c:\a 1\" \"c:\b 2\"'':' -vv

dullage · March 8, 2022, 9:13pm

Awesome, works for me too (in Powershell). Now I have the full suite of incantations!

system · March 11, 2022, 9:14pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.