Optimizing Docker-Compose setup with "rclone serve docker"

Hello! Before anything: RClone is amazing, thanks to everyone who contributed to it - it is absolutely amazing and I am super happy with it! <3

What is the problem you are having with rclone?

Not a problem; but rather a question.

Right now, I use rclone mount to mount a few of my backends to the file system, then point Docker Compose volumes to those mountpoints. Not the most optional, and it has caused issues sometimes too. So, I would like to optimize this setup by setting up configurations that properly use the Docker plugin via rclone serve docker.

  1. Assume I have a working SystemD unit that automatically starts the plugin, do the VFS options apply to all mounted backends? (Which I would absolutely like!)
  2. What is a basic example to properly mount a backend as a Docker volume? Specifically with a subfolder (i.e. wasabi-crypt:/mastodon)
  3. Can I serve multiple instances of the plugin with different options? More specifically, I would like to set a hard size limit for the mounts; mainly to avoid paying more because a process decides to be greedy. (Mastodon somehow ammassed 300GB of cache data... that was a rude awakening when I realized that.)

Run the command 'rclone version' and share the full output of the command.

# rclone --version
rclone v1.60.1
- os/version: openwrt 22.03.2 (64 bit)
- os/kernel: 5.10.110 (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.19.3
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Wasabi, Mega, SMB, SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# Something like this at the start:
rclone serve docker --vfs-cache-max-size=2G --vfs-cache-mode=full --log-file=/var/log/rclone.log ...
# Maybe something like this?
services:
  a_thing:
    # ...
    volumes:
      - type: rclone
        from: wasabi-crypt:/mastodon/system
        target: /mastodon/public/system

The rclone config contents with secrets removed.

[wasabi]
type = s3
provider = wasabi
env_auth = false
access_key_id = <redacted>
secret_access_key = <redacted>
endpoint = s3.eu-central-1.wasabisys.com
acl = private

[wasabi-crypt]
type = crypt
remote = wasabi:rclone_mnt
filename_encryption = standard
directory_name_encryption = true
password = <redacted>
password2 = <redacted>

[mega]
type = mega
user = <redacted>
pass = <redacted>

[mega-compress]
type = compress
remote = mega
level = 9

[mega-crypt-compress]
type = crypt
remote = mega:
password = <redacted>
password2 = <redacted>
no_data_encryption = false

A log from the command with the -vv flag

Have none :slight_smile: Basically just trying to learn more and optimizing things. I've run into a few permission issues to the point where I just want to move on from mount-stacking ^^; On the OpenWrt router - which I want to trial this on, as it is not my production system - I actually use the fstab method (symlink rclone to /sbin/mount.rclone and add backend mount_point opts to /etc/fstab) but this has it's limits - sometimes, when the connection stagnates and Rclone loses connections, the mounts can get stuck, disconnect and thus put me into the situation where I have to do a lot of restarting by hand. I'd like to avoid that. Hence, why I want to take more advantage of rclone serve.

Again, thanks to everyone who worked on RClone, it is just such a phantastic tool and has been a great tool in so many situations!

Kind regards,
Ingwie

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.