Possible dupe of dupe of #41686

Suspected Bug
1

What is the problem you are having with rclone?

I think I've got an intermittent dupe of #41686, but I don't seem to have permission to update to the latest beta so I can't test that potential fix yet. Thought it might be helpful to log here just in case. Hopefully I'll be able to reproduce more usefully (and sort an upgrade)...

An extra detail: the one seemingly successful transfer thus far has left 4 partial files of a total of 76 transferred (with original-filename as the file name to transfer prior to the .zip extension):

original-filename.zip.laloduq4.partial

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.2.0-1015-azure (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: snap

Which cloud storage system are you using? (eg Google Drive)

Azure Blob

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ rclone copy server-name:container-name/path local/path # Currently trying to replicate with 
$ rclone copy server-name:container-name/path local/path -vv --log-file bug.log # haven't reproduced with this yet, will keep trying

The rclone config contents with secrets removed.

[alto2txt]
type = azureblob
account = ###
key = ###
tenant = ###
client_id = ###

A log from the command with the -vv flag

Start of error (rest on pastebin):

fatal error: concurrent map writes
 
goroutine 802 [running]:
github.com/rclone/rclone/backend/azureblob.(*Object).setMetadata(0xc000364380, 0xc0009ce4e0)
        /build/rclone/parts/rclone/src/backend/azureblob/azureblob.go:1631 +0x178
github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromDownloadResponse(0xc000364380, 0xc000928e00)
        /build/rclone/parts/rclone/src/backend/azureblob/azureblob.go:1736 +0x1a9
github.com/rclone/rclone/backend/azureblob.(*Object).Open(0xc000364380, {0x2760938?, 0xc000ae2140}, {0xc000737890, 0x1, 0x1})
        /build/rclone/parts/rclone/src/backend/azureblob/azureblob.go:1935 +0x485
github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc000926380)
        /build/rclone/parts/rclone/src/fs/operations/reopen.go:116 +0x523
github.com/rclone/rclone/fs/operations.NewReOpen({0x2760938?, 0xc000ae2140}, {0x2772218?, 0xc000364380}, 0xa, {0xc000737880, 0x1, 0x1})
        /build/rclone/parts/rclone/src/fs/operations/reopen.go:49 +0x19d
github.com/rclone/rclone/fs/operations.Open({0x2760938, 0xc000ae2140}, {0x2772218, 0xc000364380}, {0xc000737880, 0x1, 0x1})
        /build/rclone/parts/rclone/src/fs/operations/reopen.go:71 +0x68
github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0xc000ae2190, {0x2760938, 0xc000ae2140}, 0x6, {0x2758490, 0xc000910050})
        /build/rclone/parts/rclone/src/fs/operations/multithread.go:84 +0x2f1
github.com/rclone/rclone/fs/operations.multiThreadCopy.func2()
        /build/rclone/parts/rclone/src/fs/operations/multithread.go:213 +0x2b
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 62
        /root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96

Full logs on pastebin

2

welcome to the forum,

not sure you need the beta, looks like the fix has already been merged into stable
https://rclone.org/changelog/#v1-64-1-2023-10-17

Azure Blob
    Fix "fatal error: concurrent map writes" (Nick Craig-Wood)
3

Ah man, I thought I sent this from a train yesterday, obviously didn't go through, my bad:

Thanks for the quick reply! Yeah I figured that fix sorted this but it seems I don't have permission to update to 1.64.1. Seeing if I can get round that via snap.

4

Think it's sorted now. At least can't reproduce after updating to edge, including no remaining partial components. Maybe worth setting v1.64.1 to stable for snap soon?

My method of updating (ultimately requiring sudo permission):

$ rclone --version
rclone v1.64.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.2.0-1015-azure (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: snap
$  sudo snap info rclone
name:      rclone
summary:   rsync for cloud storage
publisher: hideo aoyama (aoilinux)
store-url: https://snapcraft.io/rclone
contact:   https://github.com/boukendesho/rclone-snap
license:   unset
description: |
  "rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2,
  One Drive, Swift, Hubic, Cloudfiles, Google Cloud Storage, Yandex Files

  Due to the strict permissions control of Snap,`rclone mount` feature currently not
  working in snap. If you want to use mount feature, either install a precompiled
  binary or enable the relevant option when installing from source.

  Default config directory: /home/$USER/snap/rclone/current/.config/rclone.
commands:
  - rclone
snap-id:      IsLsJunB7e4PJIEtB0VEzfrSWBxgWYlR
tracking:     latest/stable
refresh-date: 20 days ago, at 15:41 UTC
channels:
  latest/stable:    1.64.0 2023-09-16 (477) 14MB -
  latest/candidate: ↑
  latest/beta:      ↑
  latest/edge:      1.64.2 2023-10-20 (483) 14MB -
installed:          1.64.0            (477) 14MB -
$ sudo snap refresh  --channel=latest/edge
rclone (edge) 1.64.2 from hideo aoyama (aoilinux) refreshed
5

good

seems to support latest stable source-tag: 'v1.64.2'
maybe the publisher, @boukendesho, can comment?

rclone is a portable executable, has no dependencies, does not need to be installed.
simply download and run.

6

Great thanks. Think that's enough to close on my end at least.

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.