Volume created by docker plugin suddenly disappears while in use

Suspected Bug
1

What is the problem you are having with rclone?

The volume created by the docker plugin just disappears after some time of writing to it. The plugin does not get deactivated. Reading seems ok.

Run the command 'rclone version' and share the full output of the command.

amd64-1.62.2

:~# docker plugin list
ID             NAME            DESCRIPTION                       ENABLED
ebf402c27f84   rclone:latest   Rclone volume plugin for Docker   true


:~# docker plugin inspect rclone
[
    {
        "Config": {
            "Args": {
                "Description": "",
                "Name": "args",
                "Settable": [
                    "value"
                ],
                "Value": []
            },
            "Description": "Rclone volume plugin for Docker",
            "DockerVersion": "20.10.23+azure-2",
            "Documentation": "https://rclone.org/docker",
            "Entrypoint": [
                "rclone",
                "serve",
                "docker"
            ],
            "Env": [
                {
                    "Description": "",
                    "Name": "RCLONE_VERBOSE",
                    "Settable": [
                        "value"
                    ],
                    "Value": "0"
                },
                {
                    "Description": "",
                    "Name": "RCLONE_CONFIG",
                    "Settable": null,
                    "Value": "/data/config/rclone.conf"
                },
                {
                    "Description": "",
                    "Name": "RCLONE_CACHE_DIR",
                    "Settable": null,
                    "Value": "/data/cache"
                },
                {
                    "Description": "",
                    "Name": "RCLONE_BASE_DIR",
                    "Settable": null,
                    "Value": "/mnt"
                },
                {
                    "Description": "",
                    "Name": "HTTP_PROXY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "HTTPS_PROXY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                },
                {
                    "Description": "",
                    "Name": "NO_PROXY",
                    "Settable": [
                        "value"
                    ],
                    "Value": ""
                }
            ],
            "Interface": {
                "Socket": "rclone.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": false,
                "Capabilities": [
                    "CAP_SYS_ADMIN"
                ],
                "Devices": [
                    {
                        "Description": "",
                        "Name": "",
                        "Path": "/dev/fuse",
                        "Settable": null
                    }
                ]
            },
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/data/config",
                    "Name": "config",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/config",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/data/cache",
                    "Name": "cache",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/cache",
                    "Type": "bind"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/mnt",
            "User": {},
            "WorkDir": "/data",
            "rootfs": {
                "diff_ids": [
                    "sha256:e98a346341e1525aeda77c2705d4ccbbd8570d58e90cc2b03eab064a4fd0456f"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "ebf402c27f846e590cb4166880d38e0ed6ca0a9bdab6699b299d7df91333be25",
        "Name": "rclone:latest",
        "PluginReference": "docker.io/rclone/docker-volume-rclone:amd64",
        "Settings": {
            "Args": [
                "-v"
            ],
            "Devices": [
                {
                    "Description": "",
                    "Name": "",
                    "Path": "/dev/fuse",
                    "Settable": null
                }
            ],
            "Env": [
                "RCLONE_VERBOSE=0",
                "RCLONE_CONFIG=/data/config/rclone.conf",
                "RCLONE_CACHE_DIR=/data/cache",
                "RCLONE_BASE_DIR=/mnt",
                "HTTP_PROXY=",
                "HTTPS_PROXY=",
                "NO_PROXY="
            ],
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/data/config",
                    "Name": "config",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/config",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/data/cache",
                    "Name": "cache",
                    "Options": [
                        "rbind"
                    ],
                    "Settable": [
                        "source"
                    ],
                    "Source": "/var/lib/docker-plugins/rclone/cache",
                    "Type": "bind"
                }
            ]
        }
    }
]

Which cloud storage system are you using? (eg Google Drive)

Strato HiDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

aptitude install fuse3

sudo mkdir -p /var/lib/docker-plugins/rclone/config
sudo mkdir -p /var/lib/docker-plugins/rclone/cache

docker plugin install rclone/docker-volume-rclone:amd64 args="-v" --alias rclone --grant-all-permissions

docker volume create vol-<myHidriveName> -d rclone -o remote=<myHidriveName>:/users/<myHidriveName>/mounted -o vfs-cache-mode=writes -o allow-other=true

Volume was mounted to the following container (command was executed after the volume had disappeared):

~# docker inspect syncthing
[
    {
        "Id": "69d5eacfbbd9f12c7770db2cd37978aaaf45b26a585a14ba258af8208271d96d",
        "Created": "2023-03-21T22:59:08.963551245Z",
        "Path": "/bin/entrypoint.sh",
        "Args": [
            "/bin/syncthing",
            "-home",
            "/var/syncthing/config"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-03-21T23:04:46.097513674Z",
            "FinishedAt": "2023-03-22T00:10:24.583896944Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2023-03-22T01:05:42.530912887+01:00",
                        "End": "2023-03-22T01:05:42.645981427+01:00",
                        "ExitCode": 0,
                        "Output": "OK\n"
                    },
                    {
                        "Start": "2023-03-22T01:06:42.647154793+01:00",
                        "End": "2023-03-22T01:06:42.726139828+01:00",
                        "ExitCode": 0,
                        "Output": "OK\n"
                    },
                    {
                        "Start": "2023-03-22T01:07:42.726958252+01:00",
                        "End": "2023-03-22T01:07:42.793521844+01:00",
                        "ExitCode": 0,
                        "Output": "OK\n"
                    },
                    {
                        "Start": "2023-03-22T01:08:42.795150672+01:00",
                        "End": "2023-03-22T01:08:42.875837854+01:00",
                        "ExitCode": 0,
                        "Output": "OK\n"
                    },
                    {
                        "Start": "2023-03-22T01:09:42.876537821+01:00",
                        "End": "2023-03-22T01:09:44.59085489+01:00",
                        "ExitCode": 0,
                        "Output": "OK\n"
                    }
                ]
            }
        },
        "Image": "sha256:1c7ffde2688cecff995ccf5ad039e1c1a264b2fb86f596882e0f9704af776914",
        "ResolvConfPath": "/var/lib/docker/containers/69d5eacfbbd9f12c7770db2cd37978aaaf45b26a585a14ba258af8208271d96d/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/69d5eacfbbd9f12c7770db2cd37978aaaf45b26a585a14ba258af8208271d96d/hostname",
        "HostsPath": "/var/lib/docker/containers/69d5eacfbbd9f12c7770db2cd37978aaaf45b26a585a14ba258af8208271d96d/hosts",
        "LogPath": "/var/lib/docker/containers/69d5eacfbbd9f12c7770db2cd37978aaaf45b26a585a14ba258af8208271d96d/69d5eacfbbd9f12c7770db2cd37978aaaf45b26a585a14ba258af8208271d96d-json.log",
        "Name": "/syncthing",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/root/docker/syncthing:/var/syncthing",
                "vol-<myHidriveName>:/mnt/<myHidriveName>"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge",
            "PortBindings": {
                "21027/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "21027"
                    }
                ],
                "22000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "22000"
                    }
                ],
                "22000/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "22000"
                    }
                ],
                "8384/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8384"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                55,
                204
            ],
            "CapAdd": [
                "AUDIT_WRITE",
                "CHOWN",
                "DAC_OVERRIDE",
                "FOWNER",
                "FSETID",
                "KILL",
                "MKNOD",
                "NET_BIND_SERVICE",
                "NET_RAW",
                "SETFCAP",
                "SETGID",
                "SETPCAP",
                "SETUID",
                "SYS_CHROOT"
            ],
            "CapDrop": [
                "AUDIT_CONTROL",
                "BLOCK_SUSPEND",
                "DAC_READ_SEARCH",
                "IPC_LOCK",
                "IPC_OWNER",
                "LEASE",
                "LINUX_IMMUTABLE",
                "MAC_ADMIN",
                "MAC_OVERRIDE",
                "NET_ADMIN",
                "NET_BROADCAST",
                "SYSLOG",
                "SYS_ADMIN",
                "SYS_BOOT",
                "SYS_MODULE",
                "SYS_NICE",
                "SYS_PACCT",
                "SYS_PTRACE",
                "SYS_RAWIO",
                "SYS_RESOURCE",
                "SYS_TIME",
                "SYS_TTY_CONFIG",
                "WAKE_ALARM"
            ],
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/dc08bfd4885ee00d4808f17f44e18eec141058689886b547313d8e0861539042-init/diff:/var/lib/docker/overlay2/ed10bb61c02f95e1ed177b492fa1249548f941641b82500a75521635cf37b79d/diff:/var/lib/docker/overlay2/0494bdf96b31b8233b0a8593588d205024e8129b0d79771006c6a536380d8d26/diff:/var/lib/docker/overlay2/69817441b8e6a8ebb6207f380ae4b751bda1c69b9448687efa899ea26885bb1a/diff:/var/lib/docker/overlay2/04528fcd9902d9fdd323d62ff2b99e924646bd966b5fcdd87821984606125767/diff",
                "MergedDir": "/var/lib/docker/overlay2/dc08bfd4885ee00d4808f17f44e18eec141058689886b547313d8e0861539042/merged",
                "UpperDir": "/var/lib/docker/overlay2/dc08bfd4885ee00d4808f17f44e18eec141058689886b547313d8e0861539042/diff",
                "WorkDir": "/var/lib/docker/overlay2/dc08bfd4885ee00d4808f17f44e18eec141058689886b547313d8e0861539042/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/root/docker/syncthing",
                "Destination": "/var/syncthing",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "vol-<myHidriveName>",
                "Source": "",
                "Destination": "/mnt/<myHidriveName>",
                "Driver": "rclone:latest",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "syncthing-strato",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "21027/udp": {},
                "22000/tcp": {},
                "22000/udp": {},
                "8384/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PUID=1000",
                "PGID=1000",
                "HOME=/var/syncthing",
                "STGUIADDRESS=0.0.0.0:8384"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o --color=never OK || exit 1"
                ],
                "Interval": 60000000000,
                "Timeout": 10000000000
            },
            "Image": "syncthing/syncthing:latest",
            "Volumes": {
                "/mnt/<myHidriveName>": {},
                "/var/syncthing": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/bin/entrypoint.sh",
                "/bin/syncthing",
                "-home",
                "/var/syncthing/config"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "858c4e7df819893d0985e0514f20ae9c0fec50748b977dbb82855085504a481b",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/858c4e7df819",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "500cf9ca3a41a4c0a7c072c1ef382c3c3baa9e5855aa855b54f125ba3ad3b4ee",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

The rclone config contents with secrets removed.

:~# cat /var/lib/docker-plugins/rclone/config/rclone.conf
[<myHidriveName>]
type = hidrive
token = {"access_token":"…","token_type":"Bearer","refresh_token":"…","expiry":"2023-03-22T00:26:29.458229183Z"}

A log from the command with the -vv flag

Before the error occured:

~# df -h
Dateisystem       Größe Benutzt Verf. Verw% Eingehängt auf
/dev/ploop15305p1  9,9G    3,2G  6,2G   34% /
none               4,0M       0  4,0M    0% /sys/fs/cgroup
none               256M       0  256M    0% /dev
tmpfs              256M       0  256M    0% /dev/shm
tmpfs              103M    1,3M  102M    2% /run
tmpfs              5,0M       0  5,0M    0% /run/lock
none               256M       0  256M    0% /run/shm
overlay            9,9G    3,2G  6,2G   34% /var/lib/docker/overlay2/6c3908f9071027e2beebbc635f7cd0cb73c9309df75ba8d9e61548bb5925b6eb/merged
tmpfs               52M       0   52M    0% /run/user/0

After the volume disappeared from docker:

~# df -h
df: /var/lib/docker/plugins/ebf402c27f846e590cb4166880d38e0ed6ca0a9bdab6699b299d7df91333be25/propagated-mount/vol-<myHidriveName>: Der Socket ist nicht verbunden -> The Socket is not connected
df: /var/lib/docker/plugins/ebf402c27f846e590cb4166880d38e0ed6ca0a9bdab6699b299d7df91333be25/propagated-mount/vol-<myHidriveName>: Der Socket ist nicht verbunden
Dateisystem       Größe Benutzt Verf. Verw% Eingehängt auf
/dev/ploop15305p1  9,9G    3,2G  6,2G   34% /
none               4,0M       0  4,0M    0% /sys/fs/cgroup
none               256M       0  256M    0% /dev
tmpfs              256M       0  256M    0% /dev/shm
tmpfs              103M    1,4M  102M    2% /run
tmpfs              5,0M       0  5,0M    0% /run/lock
none               256M       0  256M    0% /run/shm
overlay            9,9G    3,2G  6,2G   34% /var/lib/docker/overlay2/6c3908f9071027e2beebbc635f7cd0cb73c9309df75ba8d9e61548bb5925b6eb/merged
tmpfs               52M       0   52M    0% /run/user/0
overlay            9,9G    3,2G  6,2G   34% /var/lib/docker/overlay2/dc08bfd4885ee00d4808f17f44e18eec141058689886b547313d8e0861539042/merged

~# cat /var/lib/docker-plugins/rclone/cache/docker-plugin.state
[{"name":"vol-<myHidriveName>","mountpoint":"/mnt/vol-<myHidriveName>","created":"2023-03-21T22:26:39.315691789Z","fs":"<myHidriveName>:/users/<myHidriveName>/mounted","options":{"allow-other":"true","vfs-cache-mode":"writes"},"mounts":["0f0c5498d8aa73b0c2d6108a211a9df925a4d937cd88d7a5a2f6c984ef102223"]}]r

:~# docker volume ls
DRIVER    VOLUME NAME
local     portainer_data

Log from Syncthing (after succesful initial scan, some files were synced successfully):

2023-03-22 00:39:06: Error on folder "xxx1" (…): stat /mnt/<myHidriveName>/xxx1: transport endpoint is not connected

2023-03-22 00:39:07: Error on folder "xxx2" (…): stat /mnt/<myHidriveName>/xxx2: transport endpoint is not connected

2023-03-22 00:39:09: Error on folder "xxx3" (…): stat /mnt/<myHidriveName>/xxx3: transport endpoint is not connected
2

That error means the mount was killed.

3

Do you have any advice on how to find out why it was killed?
Are there any other logs I could read?

It happend multiple times.

Also: Is it expected that the volume disappears from docker volume ls?

4

I don't use the docker plugin and can only comment the actual error you shared.

Rclone is a single binary with a single config file so for me, I don't use docker for that.

That error means the mount was killed and not stopped cleanly, which leaves in that state.

1 Like
5

My next try would be to use rclone standalone and pass the mountpoint as a bind instead of a volume.

But of course, I would be thankful for any further advice what went wrong and how to avoid it.

6

Shall I post a bugreport at github?

~# df -h
df: /var/lib/docker/plugins/ebf402c27f846e590cb4166880d38e0ed6ca0a9bdab6699b299d7df91333be25/propagated-mount/vol-<myHidriveName>: Der Socket ist nicht verbunden
df: /var/lib/docker/plugins/ebf402c27f846e590cb4166880d38e0ed6ca0a9bdab6699b299d7df91333be25/propagated-mount/vol-<myHidriveName>: Der Socket ist nicht verbunden
Dateisystem       Größe Benutzt Verf. Verw% Eingehängt auf
/dev/ploop15305p1  9,9G    3,3G  6,1G   35% /
none               4,0M       0  4,0M    0% /sys/fs/cgroup
none               256M       0  256M    0% /dev
tmpfs              256M       0  256M    0% /dev/shm
tmpfs              103M    1,3M  102M    2% /run
tmpfs              5,0M       0  5,0M    0% /run/lock
none               256M       0  256M    0% /run/shm
overlay            9,9G    3,3G  6,1G   35% /var/lib/docker/overlay2/6c3908f9071027e2beebbc635f7cd0cb73c9309df75ba8d9e61548bb5925b6eb/merged
tmpfs               52M       0   52M    0% /run/user/0
root@<myServer>:~# docker volume ls
DRIVER    VOLUME NAME
local     portainer_data
root@<myServer>:~# docker volume create vol-<myHidriveName> -d rclone -o remote=<myHidriveName>:/users/<myHidriveName>/mounted -o vfs-cache-mode=writes -o allow-other=true
Error response from daemon: create vol-<myHidriveName>: VolumeDriver.Create: lstat /mnt/vol-<myHidriveName>: transport endpoint is not connected
root@<myServer>:~# ^C
root@<myServer>:~# docker plugin disable rclone
rclone
root@<myServer>:~# docker plugin enable rclone
rclone
root@<myServer>:~# docker volume create vol-<myHidriveName> -d rclone -o remote=<myHidriveName>:/users/<myHidriveName>/mounted -o vfs-cache-mode=writes -o allow-other=true
vol-<myHidriveName>
root@<myServer>:~# df -h
Dateisystem       Größe Benutzt Verf. Verw% Eingehängt auf
/dev/ploop15305p1  9,9G    3,3G  6,1G   35% /
none               4,0M       0  4,0M    0% /sys/fs/cgroup
none               256M       0  256M    0% /dev
tmpfs              256M       0  256M    0% /dev/shm
tmpfs              103M    1,3M  102M    2% /run
tmpfs              5,0M       0  5,0M    0% /run/lock
none               256M       0  256M    0% /run/shm
overlay            9,9G    3,3G  6,1G   35% /var/lib/docker/overlay2/6c3908f9071027e2beebbc635f7cd0cb73c9309df75ba8d9e61548bb5925b6eb/merged
tmpfs               52M       0   52M    0% /run/user/0
7

You can but it's really a docker question not related to rclone.

The issue is something in the process is killing it.

8

I tried it again without docker and it ran into fatal error: runtime: out of memory.
That vps has 512mb ram…

When this error occurred, it left the vps in a state where it was not usable and it took multiple tries to boot it afterwards. If I remember correct, there also was some error related to dbus and I reinstalled it after reading a hint at stackexchange.

Run the command 'rclone version' and share the full output of the command.

:~# rclone version
rclone v1.62.2
- os/version: debian 11.6 (64 bit)
- os/kernel: 4.19.0 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

HiDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount <myHidrive>:/users/<myHidrive>/mounted /mnt/<myHidrive> --vfs-cache-mode writes --allow-other -vv --log-file=/root/rclone-log/rclone.log --daemon

The rclone config contents with secrets removed.

cat /root/.config/rclone/rclone.conf
[<myHidrive>]
type = hidrive
token = {"access_token":"…","token_type":"Bearer","refresh_token":"…","expiry":"2023-03-26T…"}

A log from the command with the -vv flag

I can provide the full log via pm if it helps.
Syncthing's logs showed lots of transport endpoint is not connected but it did not crash.

https://pastebin.com/Trq3gYin

9

That generally means you ran out of memory :slight_smile:

So you can add more or try reducing quite a number of parameters but not sure a mount will work well with such a small bit of memory.

10

What would you suggest to reduce memory usage?

--transfers 1
--buffer-size 0M

Anything else?

11

Only other I can think of is:

1 Like
12

Thank you!

It seems to be running a bit longer but still crashes.
I could really live with it being slow, but solid.

Looks like I have to try again on another vps with more ram.

If I remember correctly, I had this running with pure sftp / sshfs for more than a year before it became unstable. I was hoping it might be more reliable with rclone, but maybe the files to be synced just grew too much for rclone, syncthing and 512mb ram…

13

i have run rclone mount in just 128MiB, worked fine. i posted about that in forum.

decrease these values. the greater the reduction, the less memory used.
--hidrive-chunk-size
--hidrive-upload-cutoff
--hidrive-upload-concurrency

for example,
--transfers=1 --buffer-size=0 --hidrive-chunk-size=5M --hidrive-upload-cutoff=5M --hidrive-upload-concurrency=1

1 Like
14

With this it stays below 250mb of ram in total (rclone and syncthing together):

rclone mount <myHidrive>:/users/<myHidrive>/mounted /mnt/<myHidrive> --vfs-cache-mode writes --allow-other --transfers=1 --buffer-size=0 --use-mmap --hidrive-chunk-size=5M --hidrive-upload-cutoff=5M --hidrive-upload-concurrency=1 --daemon

I will let it run for a few more days and then hopefully mark it as solved :wink:

Edit: less than 320mb of ram, but still no crash

15

good to know those tweaks works with hidrive. in the past, i had tested on S3.

let us know.....

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.