Docker containers which depend on Rclone Docker Volume Plugin fail to start after server reboot

[lohmatij (Andrey Valentsov)]

What is the problem you are having with rclone?

Docker Volume Plugin is working really good, except the cases when I'm restarting my server. After every restart I have to log in to Portainer and restart a stack (docker compose file) to load all containers which depend on volumes provided by Rclone Docker Plugin. Looks like Docker is trying to start affected containers before Rclone volume is available, I'm not sure how to work around this issue.

Docker Plugin is running as Systems service, configured per instructions is this guide:

Docker Volume Plugin

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

• os/version: ubuntu 22.04 (64 bit)
• os/kernel: 6.5.0-1025-oracle (aarch64)
• os/type: linux
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.21.1
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Mail.Ru Cloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Docker compose file I'm using:

version: '3.5'
      
services: 

  volume_waiter:
    image: busybox
    container_name: volume_waiter
    restart: on-failure
    volumes:
      - downloads:/data
    command: sh -c 'until [ "$(ls -A /data)" ]; do echo "waiting for volume"; sleep 2; done; echo "volume available"; sleep infinity'
    stdin_open: true
    tty: true
    healthcheck:
      test: ["CMD-SHELL", "ls -A /data | grep -q ."]
      interval: 3s
      timeout: 1s
      retries: 20

networks:
  intranet:
    external: true

volumes:
  downloads:
    name: downloads
    driver: rclone
    driver_opts:
      remote: 'mailru_2g:downloads'
      vfs-cache-mode: full
      allow_other: 'true'
      vfs-write-back: 1h
      vfs-cache-max-size: 80G
#      vfs-cache-min-free-space: 5G     
      vfs-cache-max-age: 10w
      vfs-cache-poll-interval: 5m
      gid: 1000
      uid: 1000

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[mailru]
type = mailru
user = XXX
pass = XXX
speedup_enable = true
token = XXX
b0ea8053b927d7241824913737363830","expiry" = "2023-08-16T20:54:35.295881-07:00"}

[mailru_2g]
type = chunker
remote = mailru:Chunker
chunk_size = 2Gi
hash_type = md5
volume_name = Mail.ru > 2GB

A log from the command that you were trying to run with the -vv flag

I checked why container didn't start successfully and here is the info from docker:

State
Dead false
Error get downloads: no such volume
ExitCode 137
FinishedAt 2024-07-02T05:29:16.487441506Z
Health { FailingStreak: 0, Log: [object Object],[object Object],[object Object],[object Object],[object Object], Status: unhealthy }
OOMKilled false
Paused false
Pid 0
Restarting false
Running false
StartedAt 2024-07-02T05:27:30.260993306Z
Status exited

Based on the Docker error message Error get downloads: no such volume and the fact that the containers start normally later, I believe the problem occurs during initialization. It seems that rclone is not ready to serve the volume when Docker is already expecting it. How can I work around this issue?

[lohmatij (Andrey Valentsov)]

I also checked systemd logs but can't see anything suspicious. Full log between 2 reboots:

Starting Docker Volume Plugin for rclone...
Started Docker Volume Plugin for rclone.
INFO  : Serving systemd socket
INFO  : Chunked 'mailru_2g:downloads': poll-interval is not supported by this remote
INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
INFO  : Volume "downloads" unmounted externally
Stopping Docker Volume Plugin for rclone...
INFO  : Signal received: terminated
INFO  : Exiting...
docker-volume-rclone.service: Main process exited, code=exited, status=143/n/a
docker-volume-rclone.service: Failed with result 'exit-code'.
Stopped Docker Volume Plugin for rclone.

[lohmatij (Andrey Valentsov)]

I switched to the docker plugin and it's all good now.