Rclone docker plugin blocks host

banhans · June 16, 2024, 6:03am

I installed the rclone docker volume plugin and use it to mount an encrypted share hosted on an SFTP share. A container running plex then tries to index the data on that share. After a few minutes to hours, the host system becomes unresponsive (no ssh login possible, other services like dns also dead). I can shutdown via power button press and reboot.

It looks like the network stack(??) on the host breaks at some point?

Here is my setup:
Docker version 26.1.4 with rclone volume plugin.

Installed by:
docker plugin install rclone/docker-volume-rclone:amd64 args="-vv" --alias rclone --grant-all-permissions

docker-compose.yml:

  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Berlin
      - VERSION=docker
      - PLEX_CLAIM=xxx
    volumes:
      - './plex/config:/config'
      - 'donau-crypt:/data'
    ports:
      - '32400:32400/tcp'
    restart: unless-stopped

volumes:
  donau-crypt:
    driver: rclone
    driver_opts:
      remote: 'donau-crypt:'
      allow_other: 'true'
      vfs_cache_mode: full
      poll_interval: 0

rclone.conf:

[yyy-sub2-Hetzner]
type = sftp
host = yyy.your-storagebox.de
user = zzz
pass = xxx
shell_type = unix
md5sum_command = none
sha1sum_command = none

[donau-crypt]
type = crypt
remote = yyy-sub2-Hetzner:rclone-crypt
filename_encryption = obfuscate
directory_name_encryption = true
password = xxx

Logs from journalctl:

before the shutdown:

Jun 16 07:21:13 smarti dockerd[752]: time="2024-06-16T07:21:13+02:00" level=error msg="2024/06/16 05:21:13 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)" plugin=4a2fac8ad830490b9feb7abc509>
Jun 16 07:21:50 smarti dockerd[752]: time="2024-06-16T07:21:50.986655465+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:53851" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:538>
Jun 16 07:21:50 smarti dockerd[752]: time="2024-06-16T07:21:50.986766170+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:52342" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:523>
Jun 16 07:21:53 smarti dockerd[752]: time="2024-06-16T07:21:53.489692360+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:40478" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:404>
Jun 16 07:21:53 smarti dockerd[752]: time="2024-06-16T07:21:53.489809785+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:33546" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:335>
Jun 16 07:21:54 smarti dockerd[752]: time="2024-06-16T07:21:54.988755463+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:56736" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:567>
Jun 16 07:21:54 smarti dockerd[752]: time="2024-06-16T07:21:54.988872036+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:50156" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:501>
Jun 16 07:22:13 smarti dockerd[752]: time="2024-06-16T07:22:13+02:00" level=error msg="2024/06/16 05:22:13 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)" plugin=4a2fac8ad830490b9feb7abc509>
Jun 16 07:23:13 smarti dockerd[752]: time="2024-06-16T07:23:13+02:00" level=error msg="2024/06/16 05:23:13 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)" plugin=4a2fac8ad830490b9feb7abc509>
Jun 16 07:23:14 smarti dockerd[752]: time="2024-06-16T07:23:14.338334393+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:44875" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:448>
Jun 16 07:23:14 smarti dockerd[752]: time="2024-06-16T07:23:14.338450415+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:56581" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:565>
Jun 16 07:23:16 smarti dockerd[752]: time="2024-06-16T07:23:16.840520535+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:34486" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:344>
Jun 16 07:23:16 smarti dockerd[752]: time="2024-06-16T07:23:16.840635243+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:44736" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:447>
Jun 16 07:23:18 smarti dockerd[752]: time="2024-06-16T07:23:18.340916396+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:38457" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:384>
Jun 16 07:23:18 smarti dockerd[752]: time="2024-06-16T07:23:18.341019739+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:55185" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:551>
Jun 16 07:23:19 smarti systemd-logind[679]: Power key pressed short.
Jun 16 07:23:19 smarti systemd-logind[679]: Powering off...
Jun 16 07:23:19 smarti systemd-logind[679]: System is powering down.

this is what happens when it goes "dead":

Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item video/movies/en/Home (1989)/Home (1989) DivX-360p.nfo n>
Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 INFO  : vfs cache: cleaned: objects 819 (was 819) in use 0, to upload 0, uploading 0, total size 10.271Gi (was 10.271Gi)" plugin=4a2fac8ad>
Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 DEBUG : pacer: low level retry 1/1 (error couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout)" plugin=4a2fac8ad830490b9feb7abc>
Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 ERROR : /: Dir.Stat error: List: couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout" plugin=4a2fac8ad830490b9feb7abc509aa68184>
Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 ERROR : IO error: List: couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout" plugin=4a2fac8ad830490b9feb7abc509aa68184c162c7184>
Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 DEBUG : /: >Lookup: node=<nil>, err=List: couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout" plugin=4a2fac8ad830490b9feb7abc5>
Jun 15 23:42:13 smarti dockerd[752]: time="2024-06-15T23:42:13+02:00" level=error msg="2024/06/15 21:42:13 DEBUG : /: Lookup: name=\"video\"" plugin=4a2fac8ad830490b9feb7abc509aa68184c162c718479bd1a7e657540dca14d1
Jun 15 23:42:14 smarti dhclient[643]: DHCPDISCOVER on enp2s0f0 to 255.255.255.255 port 67 interval 7
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15+02:00" level=error msg="2024/06/15 21:42:15 DEBUG : pacer: low level retry 1/1 (error couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout)" plugin=4a2fac8ad830490b9feb7abc>
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15+02:00" level=error msg="2024/06/15 21:42:15 ERROR : /: Dir.Stat error: List: couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout" plugin=4a2fac8ad830490b9feb7abc509aa68184>
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15+02:00" level=error msg="2024/06/15 21:42:15 ERROR : IO error: List: couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout" plugin=4a2fac8ad830490b9feb7abc509aa68184c162c7184>
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15+02:00" level=error msg="2024/06/15 21:42:15 DEBUG : /: >Lookup: node=<nil>, err=List: couldn't connect SSH: dial tcp 167.235.176.251:22: i/o timeout" plugin=4a2fac8ad830490b9feb7abc5>
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15+02:00" level=error msg="2024/06/15 21:42:15 DEBUG : /: Lookup: name=\"video\"" plugin=4a2fac8ad830490b9feb7abc509aa68184c162c718479bd1a7e657540dca14d1
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15.949260482+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:44649" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:446>
Jun 15 23:42:15 smarti dockerd[752]: time="2024-06-15T23:42:15.949323522+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.0.3:59096" dns-server="udp:192.168.178.10:53" error="read udp 172.18.0.3:590>
Jun 15 23:42:17 smarti dockerd[752]: time="2024-06-15T23:42:17+02:00" level=error msg="2024/06/15 21:42:17 DEBUG : pacer: low level retry 1/1 (error couldn't connect SSH: dial tcp 167.235.176.251:22: connect: no route to host)" plugin=4a2fac8ad8>
Jun 15 23:42:17 smarti dockerd[752]: time="2024-06-15T23:42:17+02:00" level=error msg="2024/06/15 21:42:17 ERROR : /: Dir.Stat error: List: couldn't connect SSH: dial tcp 167.235.176.251:22: connect: no route to host" plugin=4a2fac8ad830490b9feb>
Jun 15 23:42:17 smarti dockerd[752]: time="2024-06-15T23:42:17+02:00" level=error msg="2024/06/15 21:42:17 ERROR : IO error: List: couldn't connect SSH: dial tcp 167.235.176.251:22: connect: no route to host" plugin=4a2fac8ad830490b9feb7abc509aa>
Jun 15 23:42:17 smarti dockerd[752]: time="2024-06-15T23:42:17+02:00" level=error msg="2024/06/15 21:42:17 DEBUG : /: >Lookup: node=<nil>, err=List: couldn't connect SSH: dial tcp 167.235.176.251:22: connect: no route to host" plugin=4a2fac8ad83>
Jun 15 23:42:17 smarti dockerd[752]: time="2024-06-15T23:42:17+02:00" level=error msg="2024/06/15 21:42:17 DEBUG : /: Lookup: name=\"video\"" plugin=4a2fac8ad830490b9feb7abc509aa68184c162c718479bd1a7e657540dca14d1

kapitainsky · June 16, 2024, 11:07am

Then it is problem with your host... why you think it is rclone bug?

banhans · June 27, 2024, 12:30pm

Because I see all those IO and DNS errors. The system runs fine unless I stress it with this task. Also, the problem seems to have gone away since I limited the cpu usage to 50% of the cores in docker-compose.yml. So rclone seems to have hogged all ressources?

nle · June 29, 2024, 1:45pm

Try setting custom DNS in your docker compose file, and see if that helps:

  dns:
   - 1.1.1.1 #cloudflare
   - 8.8.8.8 #google

Either an issue with routing or a network connection goes down before it gets to shutdown properly.

banhans · June 29, 2024, 2:06pm

Great idea, I had suspected the internal dns as well… Alas, no avail.

system · July 29, 2024, 2:07pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.