Connecting to Azure emulator

Infarch · June 19, 2019, 10:59am

Hi All!

I'm trying to connect rclone to local Azure blob emulator. Unfortunately, without any success. Tried to use the following config:

name: blob
account: devstoreaccount1
key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
extended config/endpoint: http://127.0.0.1:10000/devstoreaccount1/

The result is:

E:\tools\rclone-v1.48.0-windows-amd64>rclone lsd blob
2019/06/19 13:46:30 ERROR : : error listing: directory not found
2019/06/19 13:46:30 Failed to lsd with 2 errors: last error was: directory not found

Does anyone know how to fix the error?

Thank you in advance!

ncw · June 19, 2019, 12:29pm

Note that needs a : on the end, so rclone lsd blob: otherwise you are listing a local directory.

Infarch · June 19, 2019, 1:04pm

Thank you for the suggestion! Still does not work but I see much information in console:

E:\tools\rclone-v1.48.0-windows-amd64>rclone lsd blob:
 2019/06/19 15:51:18 ERROR : : error listing: -> github.com/ncw/rclone/backend/azureblob.httpClientFactory.func1.1, C:/gopath/src/github.com/ncw/rclone/backend/azureblob/azureblob.go:287
HTTP request failed
 
Get https://devstoreaccount1.http://127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001: dial tcp: lookup devstoreaccount1.http: no such host
2019/06/19 15:51:18 Failed to lsd with 2 errors: last error was: -> github.com/ncw/rclone/backend/azureblob.httpClientFactory.func1.1, C:/gopath/src/github.com/ncw/rclone/backend/azureblob/azureblob.go:287
HTTP request failed

Get https://devstoreaccount1.http://127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001: dial tcp: lookup devstoreaccount1.http: no such host

The requested URL is wrong: it must be http://127.0.0.1:10000/devstoreaccount1/... instead. Whether it is possible that rclone does not work with the emulator? I cannot belive in that.

ncw · June 20, 2019, 7:05am

Can you run with -vv --dump bodies and post the output please?

Also where did you get the emulator?

Infarch · June 20, 2019, 8:03am

Some description of the emulator could be found here:

The output:

e:\tools\rclone-v1.48.0-windows-amd64>rclone lsd blob: -vv --dump bodies
2019/06/20 10:56:22 DEBUG : rclone: Version "v1.48.0" starting with parameters ["rclone" "lsd" "blob:" "-vv" "--dump" "bodies"]
2019/06/20 10:56:22 DEBUG : Using config file from "C:\\Users\\sku\\.config\\rclone\\rclone.conf"
2019/06/20 10:56:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:56:22 DEBUG : HTTP REQUEST (req 0xc0003da200)
2019/06/20 10:56:22 DEBUG : GET //127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001 HTTP/1.1
Host: devstoreaccount1.http:
User-Agent: rclone/v1.48.0
Authorization: XXXX
X-Ms-Client-Request-Id: e833d22e-72de-4957-6d39-24e7dc1b2fb5
X-Ms-Version: 2018-11-09
x-ms-date: Thu, 20 Jun 2019 07:56:22 GMT
Accept-Encoding: gzip

2019/06/20 10:56:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:56:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:56:22 DEBUG : HTTP RESPONSE (req 0xc0003da200)
2019/06/20 10:56:22 DEBUG : Error: dial tcp: lookup devstoreaccount1.http: no such host
2019/06/20 10:56:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:56:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:56:25 DEBUG : HTTP REQUEST (req 0xc0000f6300)
2019/06/20 10:56:25 DEBUG : GET //127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001 HTTP/1.1
Host: devstoreaccount1.http:
User-Agent: rclone/v1.48.0
Authorization: XXXX
X-Ms-Client-Request-Id: e833d22e-72de-4957-6d39-24e7dc1b2fb5
X-Ms-Version: 2018-11-09
x-ms-date: Thu, 20 Jun 2019 07:56:25 GMT
Accept-Encoding: gzip

2019/06/20 10:56:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:56:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:56:25 DEBUG : HTTP RESPONSE (req 0xc0000f6300)
2019/06/20 10:56:25 DEBUG : Error: dial tcp: lookup devstoreaccount1.http: no such host
2019/06/20 10:56:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:56:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:56:38 DEBUG : HTTP REQUEST (req 0xc000308200)
2019/06/20 10:56:38 DEBUG : GET //127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001 HTTP/1.1
Host: devstoreaccount1.http:
User-Agent: rclone/v1.48.0
Authorization: XXXX
X-Ms-Client-Request-Id: e833d22e-72de-4957-6d39-24e7dc1b2fb5
X-Ms-Version: 2018-11-09
x-ms-date: Thu, 20 Jun 2019 07:56:38 GMT
Accept-Encoding: gzip
2019/06/20 10:56:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:56:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:56:38 DEBUG : HTTP RESPONSE (req 0xc000308200)
2019/06/20 10:56:38 DEBUG : Error: dial tcp: lookup devstoreaccount1.http: no such host
2019/06/20 10:56:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:57:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:57:03 DEBUG : HTTP REQUEST (req 0xc000177700)
2019/06/20 10:57:03 DEBUG : GET //127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001 HTTP/1.1
Host: devstoreaccount1.http:
User-Agent: rclone/v1.48.0
Authorization: XXXX
X-Ms-Client-Request-Id: e833d22e-72de-4957-6d39-24e7dc1b2fb5
X-Ms-Version: 2018-11-09
x-ms-date: Thu, 20 Jun 2019 07:57:03 GMT
Accept-Encoding: gzip
2019/06/20 10:57:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/06/20 10:57:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:57:03 DEBUG : HTTP RESPONSE (req 0xc000177700)
2019/06/20 10:57:03 DEBUG : Error: dial tcp: lookup devstoreaccount1.http: no such host
2019/06/20 10:57:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/06/20 10:57:03 ERROR : : error listing: -> github.com/ncw/rclone/backend/azureblob.httpClientFactory.func1.1, C:/gopath/src/github.com/ncw/rclone/backend/azureblob/azureblob.go:287
HTTP request failed

Get https://devstoreaccount1.http://127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001: dial tcp: lookup devstoreaccount1.http: no such host
2019/06/20 10:57:03 Failed to lsd with 2 errors: last error was: -> github.com/ncw/rclone/backend/azureblob.httpClientFactory.func1.1, C:/gopath/src/github.com/ncw/rclone/backend/azureblob/azureblob.go:287
HTTP request failed

Get https://devstoreaccount1.http://127.0.0.1:10000/devstoreaccount1/?comp=list&maxresults=5000&timeout=31536001: dial tcp: lookup devstoreaccount1.http: no such host

ncw · June 20, 2019, 9:06am

There is definitely something wrong here with the endpoint.

Can you paste your config too please? rclone config show blob - XXX out the secrets.

Infarch · June 20, 2019, 10:53am

Sure it is wrong
But I don't know other ways for connecting to emulator. I had a hope that endpoint represents the entire address. But I was wrong.

[blob]
type = azureblob
account = devstoreaccount1
key = Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
endpoint = http://127.0.0.1:10000/devstoreaccount1/

And there is nothing secret, the emulator uses the same credentials on all machines.

ncw · June 20, 2019, 12:29pm

I had a look through the azureblob code in rclone and I think

endpoint should be a host
there isn't a way of configuring an endpoint URL

So to make this work I think we need to fix rclone.

Can you please make a new issue on github and I'll ping the azureblob maintainer about it - thanks!

Infarch · June 20, 2019, 1:35pm

So, I did: https://github.com/ncw/rclone/issues/3281

system · September 18, 2019, 1:44pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.