Azure Container Errors

What is the problem you are having with rclone?

Rclone is producing an I/O error when trying to use an rclone mount directory as a target for an application backup (couchbase).

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

  • os/version: ubuntu 18.04 (64 bit)
  • os/kernel: 5.4.0-1086-azure (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Azure Blob Container

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Rclone mount:

/usr/bin/rclone mount azure-blob:couchbase-stg-backup-test \
/mnt/azure-blob \
--allow-other \
--attr-timeout 1000h  \
--buffer-size 64M  \
--dir-cache-time 10000h \
--log-level INFO \
--log-file /tmp/rclone.log \
--poll-interval 30s \
--use-mmap \
--cache-dir=/datadisk/rclone-cache/ \
--vfs-cache-mode writes \
--transfers 100 \
--timeout 1h \
--umask 002

I know the polling flag is useless here. I just forgot to remove it from the systemd service.

The actual command causing the error is a couchbase backup:

/opt/couchbase/bin/cbbackupmgr backup --cluster "http://localhost:8091" --archive /mnt/azure-blob/backups/     --repo 2022-08-25     --username *** --password *** --resume 

And the error produced by couchbase:

Error backing up cluster: input/output error

The rclone config contents with secrets removed.

[azure-blob]
type = azureblob
account = ***
key = ***

A log from the command with the -vv flag

2022/08/25 20:33:32 INFO  : Azure container couchbase-stg-backup-test: poll-interval is not supported by this remote
2022/08/25 20:33:32 INFO  : vfs cache: cleaned: objects 96 (was 96) in use 0, to upload 0, uploading 0, total size 2.686Mi (was 2.686Mi)
2022/08/25 20:33:42 INFO  : backups/lock.lk: vfs cache: queuing for upload in 5s
2022/08/25 20:33:42 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-0: vfs cache: queuing for upload in 5s
2022/08/25 20:33:42 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-0: vfs cache: renamed in cache to "backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/.collection_manifest"
2022/08/25 20:33:42 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/.collection_manifest: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-1: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-1: vfs cache: renamed in cache to "backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_321.fol"
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-2: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-2: vfs cache: renamed in cache to "backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_343.fol"
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-3: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-3: vfs cache: renamed in cache to "backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_543.fol"
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-4: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/data_343.rift.0: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 ERROR : IO error: can't open file - writer failed
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-4: vfs cache: renamed in cache to "backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_563.fol"
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-5: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/lock.lk: vfs cache: removed cache file as file deleted
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/index_321.sqlite.0: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/index_343.sqlite.0: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/logs/stats/disk/2022-08-25-backup-1661459622: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/logs/stats/net/2022-08-25-backup-1661459622: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/logs/stats/cpu/2022-08-25-backup-1661459622: vfs cache: queuing for upload in 5s
2022/08/25 20:33:43 INFO  : backups/logs/backup-0.log: vfs cache: queuing for upload in 5s
2022/08/25 20:33:47 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/.collection_manifest: Copied (new)
2022/08/25 20:33:47 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/.collection_manifest: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/logs/stats/disk/2022-08-25-backup-1661459622: Copied (new)
2022/08/25 20:33:48 INFO  : backups/logs/stats/disk/2022-08-25-backup-1661459622: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_321.fol: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_321.fol: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/logs/stats/net/2022-08-25-backup-1661459622: Copied (new)
2022/08/25 20:33:48 INFO  : backups/logs/stats/net/2022-08-25-backup-1661459622: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/data_343.rift.0: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/data_343.rift.0: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/index_321.sqlite.0: Copied (new)
2022/08/25 20:33:48 INFO  : backups/logs/stats/cpu/2022-08-25-backup-1661459622: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/index_321.sqlite.0: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/logs/stats/cpu/2022-08-25-backup-1661459622: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/logs/backup-0.log: Copied (replaced existing)
2022/08/25 20:33:48 INFO  : backups/logs/backup-0.log: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_543.fol: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_563.fol: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_543.fol: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-5: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_563.fol: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/temp-0-5: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_343.fol: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/failoverlog_343.fol: vfs cache: upload succeeded try #1
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/index_343.sqlite.0: Copied (new)
2022/08/25 20:33:48 INFO  : backups/2022-08-25/2022-08-25T20_06_46.73886858Z/global-ad989971c12b05a4b8b015ae28e152b5/data/index_343.sqlite.0: vfs cache: upload succeeded try #1

DEGUG log is huge.
rclone.log (1.1 MB)

I am getting these errors whenever I try to rm -rf a directory/file in the mount as well. But ironically, mv operations work just fine (which if I remember correctly, rclone copies and then deletes, since azure blob doesn't support server side move)

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.