Connection string with union backend and a lot of quotes

Help and Support
1

What is the problem you are having with rclone?

I'm unable to use the connection string syntax for the union back end, when one of the upstreams is an Azure Blob Service. Probably due to quotes...

What is your rclone version (output from rclone version)

rclone v1.56.2

  • os/version: Microsoft Windows 10 Enterprise 2009 (64 bit)
  • os/kernel: 10.0.19043.1348 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.16.8
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Azure Blob Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$env:RCLONE_UNION_UPSTREAMS = '"D:\Datasets" ":azureblob,sas_url=''https://xxxxx.blob.core.windows.net/datasets?sv=sig'':datasets"'; rclone mount :union: * --network-mode --attr-timeout 99h --cache-dir D:\Temp --vfs-cache-mode full --vfs-cache-max-age 99h --read-only

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

2021/11/18 23:05:12 DEBUG : Setting default for union-upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:05:12 DEBUG : rclone: Version "v1.56.2" starting with parameters ["D:\\Dev\\rclone.exe" "mount" ":union:" "*" "--network-mode" "--attr-timeout" "99h" "--cache-dir" "D:\\Temp" "--vfs-cache-mode" "full" "--vfs-cache-max-age" "99h" "--read-only" "-vv"]
2021/11/18 23:05:12 DEBUG : Creating backend with remote ":union:"
2021/11/18 23:05:12 DEBUG : Setting union_upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:05:12 DEBUG : :union: detected overridden config - adding "{8gKLE}" suffix to name
2021/11/18 23:05:12 DEBUG : Setting union_upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:05:12 DEBUG : Using config file from "C:\\Users\\styagev\\AppData\\Roaming\\rclone\\rclone.conf"
2021/11/18 23:05:12 DEBUG : Creating backend with remote "D:/Datasets"
2021/11/18 23:05:12 DEBUG : Creating backend with remote ":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets"
2021/11/18 23:05:12 DEBUG : :azureblob: detected overridden config - adding "{QHTuA}" suffix to name
2021/11/18 23:05:12 DEBUG : fs cache: renaming cache item "D:/Datasets" to be canonical "//?/D:/Datasets"
2021/11/18 23:05:12 DEBUG : fs cache: switching user supplied name "D:/Datasets" for canonical name "//?/D:/Datasets"
2021/11/18 23:05:12 DEBUG : fs cache: renaming cache item ":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets" to be canonical ":azureblob{QHTuA}:datasets"
2021/11/18 23:05:12 DEBUG : Creating backend with remote ":azureblob,sas_url='https:/xxxxx.blob.core.windows.net/datasets?sv=sig':datasets"
2021/11/18 23:05:12 DEBUG : :azureblob: detected overridden config - adding "{B_upQ}" suffix to name
2021/11/18 23:05:12 Failed to create file system for ":union:": Container name in SAS URL and container provided in command do not match

However, the problem isn't with the blob storage connection string, since this works:

> rclone mount ":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets" * --network-mode --attr-timeout 99h --cache-dir D:\Temp --vfs-cache-mode full --vfs-cache-max-age 99h --read-only -vv

2021/11/18 23:19:53 DEBUG : Setting default for union-upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:19:53 DEBUG : rclone: Version "v1.56.2" starting with parameters ["D:\\Dev\\rclone.exe" "mount" ":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets" "*" "--network-mode" "--attr-timeout" "99h" "--cache-dir" "D:\\Temp" "--vfs-cache-mode" "full" "--vfs-cache-max-age" "99h" 
"--read-only" "-vv"]
2021/11/18 23:19:53 DEBUG : Creating backend with remote ":azureblob,sas_url='https://xxxxx.blob.core.windows.net/datasets?sv=sig':datasets"
2021/11/18 23:19:53 DEBUG : :azureblob: detected overridden config - adding "{QHTuA}" suffix to name
2021/11/18 23:19:53 DEBUG : Using config file from "C:\\Users\\styagev\\AppData\\Roaming\\rclone\\rclone.conf"
2021/11/18 23:19:53 DEBUG : fs cache: renaming cache item ":azureblob,sas_url='2021/11/18 23:19:53 DEBUG : Setting default for union-upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://princetonexp.blob.core.windows.net/datasets?sv=2020-08-04&st=2021-11-18T19%3A52%3A21Z&se=2021-11-19T19%3A52%3A21Z&sr=c&sp=racwl&sig=5k5eIPNhO3Zo1P12bNAqXpwD5KhonhbHFsuawIZqLWk%3D':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:19:53 DEBUG : rclone: Version "v1.56.2" starting with parameters ["D:\\Dev\\rclone.exe" "mount" ":azureblob,sas_url='https://princetonexp.blob.core.windows.net/datasets?sv=2020-08-04&st=2021-11-18T19%3A52%3A21Z&se=2021-11-19T19%3A52%3A21Z&sr=c&sp=racwl&sig=5k5eIPNhO3Zo1P12bNAqXpwD5KhonhbHFsuawIZqLWk%3D':datasets" "*" "--network-mode" "--attr-timeout" "99h" "--cache-dir" "D:\\Temp" "--vfs-cache-mode" "full" "--vfs-cache-max-age" "99h" 
"--read-only" "-vv"]
2021/11/18 23:19:53 DEBUG : Creating backend with remote ":azureblob,sas_url='https://princetonexp.blob.core.windows.net/datasets?sv=2020-08-04&st=2021-11-18T19%3A52%3A21Z&se=2021-11-19T19%3A52%3A21Z&sr=c&sp=racwl&sig=5k5eIPNhO3Zo1P12bNAqXpwD5KhonhbHFsuawIZqLWk%3D':datasets"
2021/11/18 23:19:53 DEBUG : :azureblob: detected overridden config - adding "{QHTuA}" suffix to name
2021/11/18 23:19:53 DEBUG : Using config file from "C:\\Users\\styagev\\AppData\\Roaming\\rclone\\rclone.conf"
2021/11/18 23:19:53 DEBUG : fs cache: renaming cache item ":azureblob,sas_url='https://princetonexp.blob.core.windows.net/datasets?sv=2020-08-04&st=2021-11-18T19%3A52%3A21Z&se=2021-11-19T19%3A52%3A21Z&sr=c&sp=racwl&sig=5k5eIPNhO3Zo1P12bNAqXpwD5KhonhbHFsuawIZqLWk%3D':datasets" to be canonical ":azureblob{QHTuA}:datasets"':datasets" to be canonical ":azureblob{QHTuA}:datasets"
2021/11/18 23:19:53 INFO  : Azure container datasets: poll-interval is not supported by this remote
...

I'm suspecting that duplicate creation of the azureblob remote. Why is that happening? Also note the second (duplicate) remote it attempts to create is strangely missing one forward slash after https: in the sas_url parameter (this is not a paste error)

Also - using account name and key instead of SAS URL also works,

2

Can you try on 1.57?

As there is a fix in there for:

Union
Fix rename not working with union of local disk and bucket based remote (Nick Craig-Wood)
3

Same.

(flsync) PS D:\Projects\flsync\flsync\python> $env:RCLONE_UNION_UPSTREAMS = '"D:\Datasets" ":azureblob,sas_url=''https://xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig'':datasets"'; rclone mount :union: * --network-mode --attr-timeout 99h --cache-dir D:\Temp --vfs-cache-mode full --vfs-cache-max-age 99h --read-only -vv

2021/11/18 23:39:14 DEBUG : Setting default for union-upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:39:14 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\Dev\\rclone.exe" "mount" ":union:" "*" "--network-mode" "--attr-timeout" "99h" "--cache-dir" "D:\\Temp" "--vfs-cache-mode" "full" "--vfs-cache-max-age" "99h" "--read-only" "-vv"]
2021/11/18 23:39:14 DEBUG : Creating backend with remote ":union:"
2021/11/18 23:39:14 DEBUG : Setting union_upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:39:14 DEBUG : :union: detected overridden config - adding "{8gKLE}" suffix to name
2021/11/18 23:39:14 DEBUG : Setting union_upstreams="\"D:\\Datasets\" \":azureblob,sas_url='https://xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig':datasets\"" from environment variable RCLONE_UNION_UPSTREAMS
2021/11/18 23:39:14 DEBUG : Using config file from "C:\\Users\\styagev\\AppData\\Roaming\\rclone\\rclone.conf"
2021/11/18 23:39:14 DEBUG : Creating backend with remote "D:/Datasets"
2021/11/18 23:39:14 DEBUG : Creating backend with remote ":azureblob,sas_url='https://xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig':datasets"
2021/11/18 23:39:14 DEBUG : :azureblob: detected overridden config - adding "{QHTuA}" suffix to name
2021/11/18 23:39:14 DEBUG : fs cache: renaming cache item "D:/Datasets" to be canonical "//?/D:/Datasets"
2021/11/18 23:39:14 DEBUG : fs cache: switching user supplied name "D:/Datasets" for canonical name "//?/D:/Datasets"
2021/11/18 23:39:14 DEBUG : fs cache: renaming cache item ":azureblob,sas_url='https://xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig':datasets" to be canonical ":azureblob{QHTuA}:datasets"
2021/11/18 23:39:14 DEBUG : Creating backend with remote ":azureblob,sas_url='https:/xxxxxxx.blob.core.windows.net/datasets?sv=sigsigsig':datasets"
2021/11/18 23:39:14 DEBUG : :azureblob: detected overridden config - adding "{B_upQ}" suffix to name
2021/11/18 23:39:14 Failed to create file system for ":union:": Container name in SAS URL and container provided in command do not match

(flsync) PS D:\Projects\flsync\flsync\python> rclone version
rclone v1.57.0
- os/version: Microsoft Windows 10 Enterprise 2009 (64 bit)
- os/kernel: 10.0.19043.1348 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount
4

Ok, didn't think it would matter but wanted to just cross it off the list.

1 Like
5

Everyone loves quoting problems!

Hmm, that missing / must be the problem.

That must mean that the path has been normalized, changing the // into /.

And why is it created twice???

Well I found the answer to both of those questions in the source code. I've attempted a fix here - can you have a go with it?

v1.58.0-beta.5900.422cba29e.fix-union-path on branch fix-union-path (uploaded in 15-30 mins)

1 Like
6

Yes! Works! Thanks @ncw!

7

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.58.

If we make a 1.57.1 I'll put it in that.

1 Like
8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.