Best approach for rclone Docker volume in combination with Docker Swarm

What is the problem you are having with rclone?

I am unsure what the best solution is to use rclone Docker volume plugin in combination with docker swarm. We have a Docker swarm composed of 3 nodes, and one of our containers needs to be stateful, so in other words a volume has to be mounted and the files of this volume need to be accessible on the other 2 nodes if Swarm decides to deploy the service on another node.

Run the command 'rclone version' and share the full output of the command.

Since we are using rclone docker volume plugin, I can't just type in rclone version. Instead I will paste the output of 'docker inspect rclone':

[
    {
        "Config": {
            "Args": {
                "Description": "",
                "Name": "args",
                "Settable": [
                    "value"
                ],
                "Value": []
            },
            "Description": "Rclone volume plugin for Docker",
            "DockerVersion": "24.0.7",
            "Documentation": "https://rclone.org/docker",
            "Entrypoint": [
                "rclone",
                "serve",
                "docker"
            ],
            "Env": [
                {
                    "Description": "",
                    "Name": "RCLONE_VERBOSE",
                    "Settable": [
                        "value"
                    ],
                    "Value": "0"
                },
                {
                    "Description": "",
                    "Name": "RCLONE_CONFIG",
                    "Settable": null,
                    "Value": "/data/config/rclone.conf"
                },
                {
                    "Description": "",
                    "Name": "RCLONE_CACHE_DIR",
                    "Settable": null,
                    "Value": "/data/cache"
                },
                {
                    "Description": "",
                    "Name": "RCLONE_BASE_DIR",
                    "Settable": null,
                    "Value": "/mnt"
                },
                {
                    "Description": "",
                    "Name": "HTTP_PROXY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "HTTPS_PROXY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "NO_PROXY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                }
            ],
            "Interface": {
                "Socket": "rclone.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": false,
                "Capabilities": [
                    "CAP_SYS_ADMIN"
                ],
                "Devices": [
                    {
                        "Description": "",
                        "Name": "",
                        "Path": "/dev/fuse",
                        "Settable": null
                    }
                ]
            },
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/data/config",
                    "Name": "config",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/config",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/data/cache",
                    "Name": "cache",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/cache",
                    "Type": "bind"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/mnt",
            "User": {},
            "WorkDir": "/data",
            "rootfs": {
                "diff_ids": [
                    "sha256:b3ccc0a5dd52d04f17888509a427a0150442b689630b9d85d39b822b43db9415"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "244b01eb11550228dd58eb8a6c98ba40d726116491f3c284b30ad29af2e88b68",
        "Name": "rclone:latest",
        "PluginReference": "docker.io/rclone/docker-volume-rclone:amd64",
        "Settings": {
            "Args": [],
            "Devices": [
                {
                    "Description": "",
                    "Name": "",
                    "Path": "/dev/fuse",
                    "Settable": null
                }
            ],
            "Env": [
                "RCLONE_VERBOSE=0",
                "RCLONE_CONFIG=/data/config/rclone.conf",
                "RCLONE_CACHE_DIR=/data/cache",
                "RCLONE_BASE_DIR=/mnt",
                "HTTP_PROXY=",
                "HTTPS_PROXY=",
                "NO_PROXY="
            ],
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/data/config",
                    "Name": "config",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/config",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/data/cache",
                    "Name": "cache",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/cache",
                    "Type": "bind"
                }
            ]
        }
    }
]

Which cloud storage system are you using? (eg Google Drive)

S3 (Ceph)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

As above, I will show instead the Docker compose volume section of the affected stack:

  rclone-api-data:
    name: ${APPLICATION}-dev-rclone-api-data
    driver: rclone
    driver_opts:
      allow_other: 'true'
      vfs_cache_mode: full
      vfs_read_ahead: 10G
      no-modtime: 'true'
      type: s3
      s3_provider: Ceph
      s3_access_key_id: REDACTED
      s3_secret_access_key: REDACTED
      s3_region: default
      s3_endpoint: https://eu2.contabostorage.com/
      path: application-dev/api

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

The folder '/var/lib/docker-plugins/rclone/config' is empty, but I used the following command to install rclone Docker volume plugin

docker plugin install rclone/docker-volume-rclone:amd64 args="-v" --alias rclone --grant-all-permissions

The goal I am trying to achieve

The best case would be to have all the files stored on S3 be available on the nodes too. I tried to achieve this with 'vfs_read_ahead: 10G', but I am unsure if this is a suitable solution or if there is another, better way to achieve this. Is there a way to make rclone download all files from S3 on container startup, cache them on the node and on every file change apply the same change on S3?