Azure Blob storage with VFS cache: "concurrent map writes" exception

What is the problem you are having with rclone?

After some time of running container, docker volume plugin (rclone serve docker) when using VFS cache throws exception fatal error: concurrent map writes. After this content of volume is not available anymore, so I have to recreate volume (and container which depends on it).

Run the command 'rclone version' and share the full output of the command.

$ sudo runc --root /run/docker/runtime-runc/plugins.moby exec $(docker plugin list --no-trunc | awk '/rclone/{print$1}') rclone version
 rclone v1.64.0
- os/version: alpine 3.18.3 (64 bit)
- os/kernel: 6.2.0-1011-azure (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

azureblob + crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

docker plugin set rclone args="--log-level DEBUG --log-file /var/log/rclone.log --allow-other --buffer-size 32Mi --dir-cache-time 1d --fast-list --links --poll-interval 1h --read-only --vfs-cache-mode full --vfs-cache-max-age 1w --vfs-cache-max-size 0 --vfs-cache-min-free-space 10G --vfs-cache-poll-interval 1m --vfs-write-back 1m --vfs-fast-fingerprint --vfs-read-ahead 64Mi"
      - rclone_azure_crypt_music:/media/music:ro
    driver: rclone
      remote: "azure-crypt:Music"

The rclone config contents with secrets removed.

type = azureblob
account = *****
key = *****
directory_markers = true

type = crypt
remote = azure:crypt
password = *****
password2 = *****
filename_encoding = base64

A log from the command with the -vv flag

Logs from plugin

sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $(docker plugin list --no-trunc | awk '/rclone/{print$1}') cat /var/log/rclone.log
fatal error: concurrent map writes

goroutine 396926 [running]:*Object).setMetadata(0xc00b947c70, 0xc006821e60)
        /go/src/ +0x178*Object).decodeMetaDataFromDownloadResponse(0xc00b947c70, 0xc00bd0b600)
        /go/src/ +0x1a9*Object).Open(0xc00b947c70, {0x2760318?, 0x397f1c0}, {0xc00be37be0, 0x2, 0x2})
        /go/src/ +0x485*Object).Open.func1({0x2760318, 0x397f1c0}, 0x42a42c0, 0x10010?)
        /go/src/ +0x174*decrypter).RangeSeek(0xc00be39320, {0x2760318, 0x397f1c0}, 0x42ac000, 0x30?, 0x8000000)
        /go/src/ +0x376*Cipher).newDecrypterSeek(0x0?, {0x2760318, 0x397f1c0}, 0xc0065e9380, 0x42ac000, 0x8000000)
        /go/src/ +0x19d*Cipher).DecryptDataSeek(0xc0027a1b78?, {0x2760318?, 0x397f1c0?}, 0x4105c5?, 0x1db5fe0?, 0x20?)
        /go/src/ +0x1d*Object).Open(0xc0027a1b78, {0x2760318, 0x397f1c0}, {0xc00bddb9e0?, 0x2, 0x2?})
        /go/src/ +0x30b*ChunkedReader).openRange(0xc00c696690)
        /go/src/ +0x4b2*ChunkedReader).Read(0xc00c696690, {0xc017964000?, 0xc007df0601?, 0xc00c314f10?})
        /go/src/ +0x315{0x273f360, 0xc00c696690}, {0xc017964000, 0x1000, 0x100000})
        /go/src/ +0x63*buffer).read(0xc0065e8f90, {0x273f360?, 0xc00c696690?})
        /go/src/ +0x3a*AsyncReader).init.func1()
        /go/src/ +0x153
created by*AsyncReader).init in goroutine 396911
        /go/src/ +0x1c5

Full log: rclone logs -

Mounted volume not accesible anymore:

docker exec -it plex ls -lah /media/music
ls: cannot access '/media/music': Transport endpoint is not connected


Plugin without configured VFS cache arguments works fine.
I found similar, fixed in 1.51 bug, but in obsolete 'cache' provider.

Maybe the same change should be implemented in VFS code?

This crash is coming from the azure blob backend rather than the VFS. It might be something the VFS is doing to make it worse though.

Its relatively easy to fix though - please give this a go

v1.65.0-beta.7358.ccd0b70b7.fix-azureblob-metadata on branch fix-azureblob-metadata (uploaded in 15-30 mins)

1 Like

Thanks for quick reply.
How to apply this to docker volume plugin?

Ok I think I got this - replaced bin in plugin container:

$ sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $(docker plugin list --no-trunc | awk '/rclone/{print$1}') rclone --version
rclone v1.65.0-beta.7358.ccd0b70b7.fix-azureblob-metadata
- os/version: alpine 3.18.3 (64 bit)
- os/kernel: 6.2.0-1011-azure (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none

Giving it a spin, I will let you know what're the results soon.

Edit 2:
So far no errors! :slight_smile:

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.65 - if we do a point release I'll put it in v1.64.1 also

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.