Need help with rclone mount config for dovecot email directory regarding duplicate dovecot transaction logs

Help and Support
1

What is the problem you are having with rclone?

I'm using rclone to mount a Cloudflare R2 bucket as the mail directory for my dovecot emailserver. However, I am running into errors from dovecot regarding "duplicate transaction logs".

I have read through the dovecot docs and it seems this issue is related to using virtual filesystems like NFS and fuse and dovecot's feature of "locking" files to prevent multiple sources from writing to a file at the same time.

I am able to sync email to the rclone mounted mail directory but when I access the mail via IMAP I get the following error logs. I am thinking this has something to do with my rclone mount config. Can you please review my config and let me know if there is something I can adjust? To be clear, there are no errors from rclone.

rclone version 
rclone v1.61.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-1027-oracle (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Cloudflare R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount coscrypt-mx: /opt/mx/maildata \
--config="/opt/rclone/rclone.conf" --cache-dir="/opt/rclone/cache" \
--log-level=DEBUG --log-file="/opt/rclone/mount.log" \
--uid=5000 --gid=5000 --umask=022 \
--allow-other --allow-non-empty --use-server-modtime \
--attr-timeout=10s \
--dir-cache-time=24h \
--vfs-read-chunk-size=0 \
--vfs-cache-max-size=5G \
--vfs-write-back=10m \
--vfs-fast-fingerprint \
--vfs-cache-mode=full \
--vfs-read-ahead=128M --buffer-size=32Mi --transfers=16 \
--s3-chunk-size=100M --s3-upload-cutoff=500M --s3-no-check-bucket

The rclone config contents with secrets removed.

rclone config 
[r2]
type = s3
provider = Cloudflare
access_key_id =
secret_access_key =
region = auto
endpoint = https://1234.r2.cloudflarestorage.com

[coscrypt-mx]
type = crypt
remote = r2:mx
password =
password2 =

dovecot error logs

dovecot: imap(admin@archive.com)<1273><KX4JZqL6tpZoJG5n>: Error: Mailbox INBOX: Transaction log /var/mail/archive.com/admin/dovecot.index.log: duplicate transaction log sequence (2)

A log from the command with the -vv flag

rclone logs 
2023/05/01 10:11:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item archive.com/admin/.INBOX/dovecot.index.cache not removed, freed 0 bytes

2023/05/01 10:11:49 INFO : vfs cache: cleaned: objects 134 (was 134) in use 134, to upload 168, uploading 0, total size 508.828Ki (was 508.828Ki)

2023/05/01 10:12:06 INFO : Signal received: interrupt

2023/05/01 10:12:06 DEBUG : vfs cache: cleaner exiting

2023/05/01 10:12:06 ERROR : /opt/mx/maildata: Unmounted rclone mount

2023/05/01 10:12:06 INFO : Exiting...
2

The rclone fuse mount doesn't support locking (as in flock or fcntl). Rclone could possibly implement it but it would be unreliable as cloud storage systems don't support locking themselves so rclone would have to emulate it.

I suspect this is at the root of the problem.

3

Thanks, Nick. That's what I suspected too. So Dovecot has a third option for locking called dotlock and with that enabled, I was able to create the Dovecot mail folders on the rclone-mounted bucket. With either flock or fcntl I wasn't able to create the mail folders.

Do you have any insight into why Dovecot would be seeing "duplicate transaction logs" with the fuse mount? Is there anything with the VFS cache settings I can try to adjust to resolve this?

Also, to help me understand better, do you know, to the linux kernel, how similar are NFS mounts versus fuse mounts? There's a lot of documentation about mail storage on NFS but not much on fuse mounts. Dovecot says they also support mail storage on GlusterFS, so I'm going to explore that more.

Could I try to implement in a fork of rclone the emulation of locking? Do you have any recommendations on how I could start with this?

Many thanks!

4

dotlock just creates files doesn't it which will be fine with rclone mount.

No - does it do that with dotlock? I would take a squint in the source code to see if that give you any ideas.

If there was a scale from 10 (behaves like a perfectly normal file system) to 1 (does weird stuff) then NFS is probably about 8, rclone mount with --vfs-cache-mode writes/full is about 7 and rclone mount without vfs cache mode is about 5.

I think GluserFS will act more like a proper file system, more at the NFS levels of proper so may be a better choice for this.

The first thing to do would be to see how to implement locking in a FUSE file system (I don't know the answer to that).

Then we'd have to work out how to implement that on the cloud storage backend. Creating files is all we can do so it would have to work like that.

I think you'd end up implementing something rather like dotlock in the end!

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.