Rclone mount in Docker Compose service with VFS cache - Sonarr unable to read files while they're in cache

I have two mounts running in docker containers, set up as compose services.
All my services are running under their own user and a shared group. For example, Rclone is running as rclone:dukemedia (13003:13000), Sonarr is running as sonarr:dukemedia (13001:13000), etc.
The two mounts are a data mount that stores shared data like usenet downloads and Sonarr/Radarr media:

remote-data:
    container_name: remote-data
    image: rclone/rclone:latest
    user: 13003:${SHARED_GID}
    privileged: true
    environment:
      - PUID=13003
      - PGID=${SHARED_GID}
      - UMASK=002
    volumes:
      - ${DIR_LOCAL_CONF}/rclone:/config/rclone
      - ${DIR_LOCAL_CONF}/rclone-cache:/cache
      - ${DIR_RCLONE_DATA}:/data:shared
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
      - /etc/fuse.conf:/etc/fuse.conf:ro
    devices:
      - /dev/fuse
    security_opt:
      - apparmor:unconfined
    command:
      [
        'mount', 'crypt:data', '/data/mount',
        '--vfs-cache-mode', 'full',
        '--vfs-write-back', '1h',
        '--umask', '002',
        '--uid', '13003',
        '--gid', '${SHARED_GID}',
        '--dir-perms', '0775',
        '--file-perms', '664',
        '--cache-dir', '/cache',
        '--allow-other',
        '-vvv'
      ]
    healthcheck:
      test: grep -q ready "/data/mount/healthcheck.txt"
      interval: 10s
      timeout: 10s
      retries: 1
      start_period: 30s

... and a config mount that stores config files for most other apps:

remote-config:
    container_name: remote-config
    image: rclone/rclone:latest
    user: 13003:${SHARED_GID}
    privileged: true
    environment:
      - PUID=13003
      - PGID=${SHARED_GID}
      - UMASK=002
    volumes:
      - ${DIR_LOCAL_CONF}/rclone:/config/rclone
      - ${DIR_LOCAL_CONF}/rclone-cache:/cache
      - ${DIR_RCLONE_CONFIG}:/data:shared
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
      - /etc/fuse.conf:/etc/fuse.conf:ro
    devices:
      - /dev/fuse
    security_opt:
      - apparmor:unconfined
    command:
      [
        'mount', 'crypt:config', '/data/mount',
        '--vfs-cache-mode', 'full',
        '--vfs-write-back', '10m',
        '--umask', '002',
        '--uid', '13003',
        '--gid', '${SHARED_GID}',
        '--dir-perms', '0775',
        '--file-perms', '664',
        '--cache-dir', '/cache',
        '--allow-other',
        '-vvv'
      ]
    healthcheck:
      test: grep -q ready "/data/mount/healthcheck.txt"
      interval: 10s
      timeout: 10s
      retries: 1
      start_period: 30s

Both services are running the latest tag and running rclone --version inside of them at the time of writing gives me:

rclone v1.61.1
- os/version: alpine 3.17.0 (64 bit)
- os/kernel: 5.15.0-56-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Both containers are mounting the same crypt remote, which covers a Google Drive backend (Workspace Enterprise)
They are binding to the same cache directory on the host, I don't know if this could be a problem, but I assume not since each mount seems to get its own directory under that.

Both containers are using this Rclone config:

[gdrive]
type = drive
client_id = <REDACTED>
client_secret = <REDACTED>
scope = drive
token = <REDACTED>
team_drive = 

[crypt]
type = crypt
remote = gdrive:rclone
password = <REDACTED>
password2 = <REDACTED>

Now, for the actual issue: Sonarr/Radarr sends something to Sab. Sab downloads it. Sonarr/Radarr sees that it's done. No issues so far. But now Sonarr/Radarr will use Mediainfo to try to detect samples (and probably some other things) and this is where things go wrong. Sonarr/Radarr continually tries to do this and spam the events tab/debug log with the following: Hastebin: Send and Save Text or Code Snippets for Free | Toptal®
I originally assumed that this was an issue with the Sonarr/Radarr containers, but I noticed that literally within seconds of the cache writeback kicking in, Sonarr/Radarr is able able to do what it needs to do, and imports the files without any issues so it seems it's an issue with the VFS cache or at least with the way Sonarr/Radarr is able to interact with it.

At the same time as this happening, I get the following in the rclone log (container output with -vvv on the mount command): Hastebin: Send and Save Text or Code Snippets for Free | Toptal®

That's a semi confusing setup to step through.

I use dockers for sonarr/radarr without issue but I don't use rclone in a docker.

I also run my containers all as the same user.

It looks like something permissionish related or more the docker setup between rclone/sonarr.

Maybe test removing the rclone from the docker and see if things work and try to replicate the issue with less moving parts.

Thanks for the reply and happy holidays :slight_smile:

Some history for context: I was originally trying to use the Rclone Docker Volume plugin to create a shared volume for my containers, and it worked except for the same issue I'm having now. I assumed it had something to do with the volume plugin and since I couldn't find many resources on using it, I decided to just run rclone on the host machine. I set that up, automounting with systemd, but now I ran into an issue where systemd was not able to interact very cleanly with my docker containers and I would sometimes have my docker services start before the mounts were ready, resulting in them writing to the directories I was attempting to use as my mountpoint. After fiddling with this for a while, I thought maybe a better setup would be running rclone in a normal service container, not via the volume plugin, and then use a basic file-based healthcheck to establish service dependencies. So, here we are. I can of course go back to running rclone on the host, but I'd rather try to troubleshoot this. Thanks for the suggestion though.

I don't use automount as I use a service file for each of my rclone mounts.

I use an override for docker that requires each mount to be running for docker to function and docker does not start until after rclones are mounted.

felix@gemini:/etc/systemd/system$ cat docker.service.d/override.conf
[Unit]
After=rclone-movies.service rclone-tv.service
Requires=rclone-movies.service rclone-tv.service

Overrides allow you to not change package service files and add your own changes to them without breaking things.

I reboot/stop/etc all the time and if rclone was to stop, all my docker services would stop as it works quite well for me/my use case.

That's interesting, do you mind showing me what your rclone services look like? It didn't even occur to me to completely block Docker from running before the mounts. My issue was that my rclone services would run and I assume systemd would see them as OK before the FUSE mounts themselves were actually ready. I would love to have this all running in Docker, but I would love even more to have all this just working. I did not know about this override functionality.

Edit: never mind, I found your homescripts repo when looking for documentation on systemd overrides. Thanks for the help, I'll try to replicate your method.

1 Like

Just wanna thank you again, the override solved the issue I had with systemd before, and so there's no real reason for me to be running rclone as a docker compose service. I would still love to get that to work, but oh well, this works perfectly fine.

I’m not familiar enough with docket volumes to debug that. You try to ask perhaps on their forums or Reddit does well too.

I just migrated my stuff to containers and rclone being one binary and one config file seemed to be overkill for docker for me.