Runtime error in "rclone serve restic" with "azureblob" backend when initializing backup repository

Suspected Bug
1

What is the problem you are having with rclone?

I am trying to use rclone serve restic with a "azureblob" backend. When I am trying to initialize a new backup-repository "foo", using restic -r rest:http://localhost:8080/foo, I am getting the following error output from rclone:

2022/10/18 12:57:22 NOTICE: Azure root: Serving restic REST API on http://localhost:8080/
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0x9d2fc8]

goroutine 73 [running]:
github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock({{{{0x0, 0x0}, {0x0, 0x0}, 0x0, {0x0, 0x0}, {0xc00004e500, 0x4e}, {0x0, ...}, ...}, ...}}, ...)
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zz_generated_block_blob.go:509 +0x448
github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock({{{{{...}, {...}}}}, {{{{...}, {...}, 0x0, {...}, {...}, {...}, _, {...}, ...}, ...}}}, ...)
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/url_block_blob.go:97 +0x1df
github.com/Azure/azure-storage-blob-go/azblob.(*copier).write(0xc000357c00, {{0xc000b80000, 0x400000, 0x400000}, {0xc0005c0360, 0x58}, 0x1da})
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/chunkwriting.go:165 +0x263
github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk.func1()
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/chunkwriting.go:136 +0xc7
github.com/rclone/rclone/backend/azureblob.(*poolWrapper).Run.func1()
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1623 +0x26
created by github.com/rclone/rclone/backend/azureblob.(*poolWrapper).Run
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1622 +0x8a

The configuration of the "azureblob" backend seems to work, as copying files to/from the Blob store works using rclone copy.

I suspect that this problem is specific to the "azureblob" backend - it does not occur when I am using a "local" backend instead.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

  • os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
  • os/kernel: 10.0.19044.2130 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.18.6
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Microsoft Azure Blob Storage (i.e. type "azureblob")

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --log-file rclone-serve-restic.log serve restic machineupgrade:

The rclone config contents with secrets removed.

{
    "local": {
        "nounc": "true",
        "type": "local"
    },
    "machineupgrade": {
        "sas_url": "xxx",
        "type": "azureblob"
    }
}

A log from the command with the -vv flag

2022/10/18 13:19:39 DEBUG : rclone: Version "v1.59.2" starting with parameters ["C:\\Users\\ThomasPonweiser\\AppData\\Local\\Microsoft\\WindowsApps\\rclone.exe" "-vv" "--log-file" "rclone-serve-restic-local-verbose.log" "serve" "restic" "local:"]
2022/10/18 13:19:39 DEBUG : Creating backend with remote "local:"
2022/10/18 13:19:39 DEBUG : Using config file from "C:\\Users\\ThomasPonweiser\\AppData\\Roaming\\rclone\\rclone.conf"
2022/10/18 13:19:39 DEBUG : fs cache: renaming cache item "local:" to be canonical "C:/projects/edp/tmp/restic-test"
2022/10/18 13:19:39 NOTICE: Local file system at C:/projects/edp/tmp/restic-test: Serving restic REST API on http://localhost:8080/
2022/10/18 13:20:05 DEBUG : Local file system at C:/projects/edp/tmp/restic-test: HEAD /test-01/config
2022/10/18 13:20:24 DEBUG : Local file system at C:/projects/edp/tmp/restic-test: HEAD /test-02/config
2022/10/18 13:20:24 DEBUG : test-02/config: HEAD request error: object not found
2022/10/18 13:20:24 DEBUG : Local file system at C:/projects/edp/tmp/restic-test: POST /test-02/
2022/10/18 13:20:24 INFO  : test-02: Creating repository
2022/10/18 13:20:24 DEBUG : Local file system at C:/projects/edp/tmp/restic-test: HEAD /test-02/config
2022/10/18 13:20:24 DEBUG : test-02/config: HEAD request error: object not found
2022/10/18 13:20:26 DEBUG : Local file system at C:/projects/edp/tmp/restic-test: POST /test-02/keys/17ab91338f1e9d81bf3859868dad586527d28cbefa54a5e4be3ce4aa55906e12
2022/10/18 13:20:26 DEBUG : Local file system at C:/projects/edp/tmp/restic-test: POST /test-02/config
2022/10/18 13:20:43 DEBUG : rclone: Version "v1.59.2" starting with parameters ["C:\\Users\\ThomasPonweiser\\AppData\\Local\\Microsoft\\WindowsApps\\rclone.exe" "-vv" "--log-file" "rclone-serve-restic-local-verbose.log" "serve" "restic" "machineupgrade:"]
2022/10/18 13:20:43 DEBUG : Creating backend with remote "machineupgrade:"
2022/10/18 13:20:43 DEBUG : Using config file from "C:\\Users\\ThomasPonweiser\\AppData\\Roaming\\rclone\\rclone.conf"
2022/10/18 13:20:43 NOTICE: Azure root: Serving restic REST API on http://localhost:8080/
2022/10/18 13:20:52 DEBUG : Azure root: HEAD /test-01/config
2022/10/18 13:20:52 DEBUG : test-01/config: HEAD request error: object not found
2022/10/18 13:20:52 DEBUG : Azure root: POST /test-01/
2022/10/18 13:20:52 INFO  : test-01: Creating repository
2022/10/18 13:20:52 DEBUG : Azure root: HEAD /test-01/config
2022/10/18 13:20:52 DEBUG : test-01/config: HEAD request error: object not found
2022/10/18 13:20:55 DEBUG : Azure root: POST /test-01/keys/b573540ff2d784058e99c8376071e0de271e18249b052584c630448cc2cc399c
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0x9d2fc8]

goroutine 87 [running]:
github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock({{{{0x0, 0x0}, {0x0, 0x0}, 0x0, {0x0, 0x0}, {0xc0005ae190, 0x4e}, {0x0, ...}, ...}, ...}}, ...)
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zz_generated_block_blob.go:509 +0x448
github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock({{{{{...}, {...}}}}, {{{{...}, {...}, 0x0, {...}, {...}, {...}, _, {...}, ...}, ...}}}, ...)
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/url_block_blob.go:97 +0x1df
github.com/Azure/azure-storage-blob-go/azblob.(*copier).write(0xc00021b6c0, {{0xc000ad6000, 0x400000, 0x400000}, {0xc00011ec60, 0x58}, 0x1da})
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/chunkwriting.go:165 +0x263
github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk.func1()
	github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/chunkwriting.go:136 +0xc7
github.com/rclone/rclone/backend/azureblob.(*poolWrapper).Run.func1()
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1623 +0x26
created by github.com/rclone/rclone/backend/azureblob.(*poolWrapper).Run
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1622 +0x8a
2

Hi Ponweiser,

Good description and analysis, you have already tried many of the right things.

I don't think there are many with restic knowledge answering in this forum, so chances of help/fix may be better if you can reproduce with something more common. Preferably using a simple OS copy/put without restic in the loop.

Perhaps you can reproduce using rclone serve sftp , rclone serve webdav or rclone mount?

3

I have a vague memory that this is something to do with the SAS URL. Copying the file to the root will do that using a SAS URL.

You need to include the bucket name, so

rclone --log-file rclone-serve-restic.log serve restic machineupgrade:nameofbucketgoeshere

Rclone should warn about this. Its really a bug in the azure SDK though! See Sync: Memory violation on Raspberry Pi 4b · Issue #5820 · rclone/rclone · GitHub

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.