Mount error "Transport endpoint is not connected"

[esturniolo (Emiliano)]

Hi all guys.
I'm having a little problem with a mount that I don't fully understand and maybe you can help me with it.

What is the problem you are having with rclone?

I'm using rclone to backup my proxmox stack into a WebDAV mount into a Prxmox Debian OS.
The problem is that suddenly (2 minutes, 2 hours or 2 days after run the mount command or even in the middle of a backup) the mount itself fails with: Transport endpoint is not connected

And when I try to list the directory I've this:

my-user@hostname:~# ls -l
ls: cannot access 'rcloneMount': Transport endpoint is not connected
d????????? ? ?    ?       ?            ? rcloneMount/

May be the rclone mount flags used are not the correct for this type of use.

Thanks in advance!

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: debian 11.8 (64 bit)
- os/kernel: 5.15.131-2-pve (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: non

NOTE: I had the issue with version 1.63.1. I must upgrade to 1.66 to get the "rclone config redacted" command available

Which cloud storage system are you using? (eg Google Drive)

WebDAV (Hetzner Cloud Storage - Nextcloud)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --daemon --daemon-timeout=5m --allow-non-empty --buffer-size=64M --dir-cache-time=24h --vfs-cache-mode full  --vfs-read-chunk-size 100M webdav:/backups/proxmox /root/rcloneMount/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[webdavHetzner]
type = webdav
url = https://nx12345.your-storageshare.de/remote.php/dav/files/myuser/
vendor = nextcloud
user = XXX
pass = XXX

A log from the command that you were trying to run with the -vv flag

2024/03/23 13:07:12 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "mount" "--daemon" "--daemon-timeout=5m" "--allow-non-empty" "--buffer-size=64M" "--dir-cache-time=24h" "--vfs-cache-mode" "full" "--vfs-read-chunk-size" "100M" "webdav:/backups/proxmox" "/root/rcloneMount/" "-vv"]
2024/03/23 13:07:12 DEBUG : Creating backend with remote "webdav:/backups/proxmox"
2024/03/23 13:07:12 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/03/23 13:07:12 DEBUG : found headers: 
2024/03/23 13:07:12 DEBUG : Chunks temporary upload directory: https://nx12345.your-storageshare.de/remote.php/dav/files/myuser/
2024/03/23 13:07:12 DEBUG : fs cache: renaming cache item "webdavHetzner:/backups/proxmox" to be canonical "webdav:backups/proxmox"
2024/03/23 13:07:19 DEBUG : rclone: Version "v1.66.0" finishing with parameters ["/usr/bin/rclone" "mount" "--daemon" "--daemon-timeout=5m" "--allow-non-empty" "--buffer-size=64M" "--dir-cache-time=24h" "--vfs-cache-mode" "full" "--vfs-read-chunk-size" "100M" "webdav:/backups/proxmox" "/root/rcloneMount/" "-vv"]

[asdffdsa (jojothehumanmonkey)]

@esturniolo

The problem is that suddenly (2 minutes, 2 hours or 2 days after run the mount command or even in the middle of a backup) the mount itself fails with: Transport endpoint is not connected

that log snippet is just seven seconds and does not show the issue?

fwiw, i would test without --daemon --daemon-timeout=5m --allow-non-empty

[esturniolo (Emiliano)]

You're right. I'll try to rerun the command and send all the logs to a file.

[asdffdsa (jojothehumanmonkey)]

@esturniolo

I'll try to rerun the command and send all the logs to a file.

great, but we do not need the full log, just yet.
for now, run the command, copy the top 30 lines into a new post.

[esturniolo (Emiliano)]

Here is the log that I ran with this command:

rclone mount --daemon --daemon-timeout=5m --allow-non-empty --buffer-size=64M --dir-cache-time=24h --vfs-cache-mode full --log-file /root/rclone.log --log-level INFO --vfs-read-chunk-size 100M webdav:/backups/proxmox /root/rcloneMount/

2024/03/23 19:31:43 INFO  : webdav root 'backups/proxmox': poll-interval is not supported by this remote
2024/03/23 19:31:43 INFO  : backups/dump: Removing directory
2024/03/23 19:31:43 INFO  : backups: Removing directory
2024/03/23 19:31:43 INFO  : backups/dump: Removing directory
2024/03/23 19:31:43 INFO  : backups: Removing directory
2024/03/23 19:31:43 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2024/03/23 19:32:43 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 0, total size 2.610Gi (was 2.610Gi)
2024/03/23 19:33:43 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 0, total size 8.369Gi (was 8.369Gi)
2024/03/23 19:34:33 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.dat: vfs cache: queuing for upload in 5s
2024/03/23 19:34:33 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.dat: vfs cache: renamed in cache to "backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst"
2024/03/23 19:34:34 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes.tmp.521494: vfs cache: queuing for upload in 5s
2024/03/23 19:34:34 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes.tmp.521494: vfs cache: renamed in cache to "backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes"
2024/03/23 19:34:35 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes: vfs cache: queuing for upload in 5s
2024/03/23 19:34:36 INFO  : backups/dump/vzdump-lxc-100-2024_02_23-02_10_00.tar.zst.notes: vfs cache: removed cache file as file deleted
2024/03/23 19:34:37 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.log: vfs cache: queuing for upload in 5s
2024/03/23 19:34:41 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes: Copied (new)
2024/03/23 19:34:41 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes: vfs cache: upload succeeded try #1
2024/03/23 19:34:43 INFO  : vfs cache: cleaned: objects 8 (was 8) in use 2, to upload 0, uploading 2, total size 12.457Gi (was 12.457Gi)
2024/03/23 19:34:43 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.log: Copied (new)
2024/03/23 19:34:43 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.log: vfs cache: upload succeeded try #1
2024/03/23 19:35:43 INFO  : vfs cache: cleaned: objects 8 (was 8) in use 1, to upload 0, uploading 1, total size 12.457Gi (was 12.457Gi)
2024/03/23 19:36:43 INFO  : vfs cache: cleaned: objects 9 (was 9) in use 2, to upload 0, uploading 1, total size 17.737Gi (was 17.737Gi)
2024/03/23 19:37:43 INFO  : vfs cache: cleaned: objects 9 (was 9) in use 2, to upload 0, uploading 1, total size 21.701Gi (was 21.701Gi)
2024/03/23 19:38:43 INFO  : vfs cache: cleaned: objects 9 (was 9) in use 2, to upload 0, uploading 1, total size 23.012Gi (was 23.012Gi)
2024/03/23 19:39:34 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.dat: vfs cache: queuing for upload in 5s
2024/03/23 19:39:34 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.dat: vfs cache: renamed in cache to "backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst"
2024/03/23 19:39:34 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes.tmp.524695: vfs cache: queuing for upload in 5s
2024/03/23 19:39:34 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes.tmp.524695: vfs cache: renamed in cache to "backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes"
2024/03/23 19:39:35 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes: vfs cache: queuing for upload in 5s
2024/03/23 19:39:35 INFO  : backups/dump/vzdump-lxc-105-2024_02_18-02_00_04.tar.zst.notes: vfs cache: removed cache file as file deleted
2024/03/23 19:39:38 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.log: vfs cache: queuing for upload in 5s
2024/03/23 19:39:41 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes: Copied (new)
2024/03/23 19:39:41 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes: vfs cache: upload succeeded try #1
2024/03/23 19:39:43 INFO  : vfs cache: cleaned: objects 14 (was 14) in use 3, to upload 0, uploading 3, total size 24.235Gi (was 24.235Gi)
2024/03/23 19:39:44 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.log: Copied (new)
2024/03/23 19:39:44 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.log: vfs cache: upload succeeded try #1
2024/03/23 19:40:43 INFO  : vfs cache: cleaned: objects 14 (was 14) in use 2, to upload 0, uploading 2, total size 24.235Gi (was 24.235Gi)
2024/03/23 19:41:43 INFO  : vfs cache: cleaned: objects 15 (was 15) in use 3, to upload 0, uploading 2, total size 24.633Gi (was 24.633Gi)
2024/03/23 19:41:44 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.dat: vfs cache: queuing for upload in 5s
2024/03/23 19:41:44 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.dat: vfs cache: renamed in cache to "backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst"
2024/03/23 19:41:44 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst.notes.tmp.527084: vfs cache: queuing for upload in 5s
2024/03/23 19:41:44 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst.notes.tmp.527084: vfs cache: renamed in cache to "backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst.notes"
2024/03/23 19:41:45 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst.notes: vfs cache: queuing for upload in 5s
2024/03/23 19:41:46 INFO  : backups/dump/vzdump-lxc-103-2024_03_22-17_53_25.tar.zst.notes: vfs cache: removed cache file as file deleted
2024/03/23 19:41:49 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.log: vfs cache: queuing for upload in 5s
2024/03/23 19:41:51 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst.notes: Copied (new)
2024/03/23 19:41:51 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst.notes: vfs cache: upload succeeded try #1
2024/03/23 19:41:56 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.log: Copied (new)
2024/03/23 19:41:56 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.log: vfs cache: upload succeeded try #1
2024/03/23 19:42:43 INFO  : vfs cache: cleaned: objects 20 (was 20) in use 3, to upload 0, uploading 3, total size 24.684Gi (was 24.684Gi)
2024/03/23 19:43:31 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst: Copied (new)
2024/03/23 19:43:31 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst: vfs cache: upload succeeded try #1
2024/03/23 19:43:43 INFO  : vfs cache: cleaned: objects 20 (was 20) in use 2, to upload 0, uploading 2, total size 24.684Gi (was 24.684Gi)
2024/03/23 19:44:43 INFO  : vfs cache: cleaned: objects 20 (was 20) in use 2, to upload 0, uploading 2, total size 24.684Gi (was 24.684Gi)
2024/03/23 19:45:43 INFO  : vfs cache: cleaned: objects 20 (was 20) in use 2, to upload 0, uploading 2, total size 24.684Gi (was 24.684Gi)
2024/03/23 19:45:49 INFO  : Signal received: terminated
2024/03/23 19:45:49 ERROR : /root/webdavMount/: Unmounted rclone mount
2024/03/23 19:45:49 INFO  : Exiting...

I hope that this be useful.

Thanks in advance!

[asdffdsa (jojothehumanmonkey)]

@esturniolo

I hope that this be useful.
Thanks in advance!

good, but need top thirty lines of a debug log
--log-level DEBUG

[esturniolo (Emiliano)]

Sorry for the delay.
TBH it seems that the new version works more stable with the same command and flags.
3 days in a row without a single disconnection.

I've the log anyway just for feed our curiosity.

2024/03/25 22:55:33 DEBUG : Creating backend with remote "webdav:/backups/proxmox"
2024/03/25 22:55:33 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/03/25 22:55:33 DEBUG : found headers:
2024/03/25 22:55:33 DEBUG : Chunks temporary upload directory: https://nx12345.your-storageshare.de/remote.php/dav/uploads/esturniolo/
2024/03/25 22:55:34 DEBUG : fs cache: renaming cache item "webdav:/backups/proxmox" to be canonical "webdav:backups/proxmox"
2024/03/25 22:55:34 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/usr/bin/rclone" "mount" "--daemon" "--daemon-timeout=5m" "--allow-non-empty" "--buffer-size=64M" "--dir-cache-time=24h" "--vfs-cache-mode" "full" "--log-file" "/root/rclone.log" "--log-level" "DEBUG" "--vfs-read-chunk-size" "100M" "webdav:/backups/proxmox" "/root/rcloneMount/"]
2024/03/25 22:55:34 DEBUG : Creating backend with remote "webdav:/backups/proxmox"
2024/03/25 22:55:34 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/03/25 22:55:34 DEBUG : found headers:
2024/03/25 22:55:34 DEBUG : Chunks temporary upload directory: https://nx12345.your-storageshare.de/remote.php/dav/uploads/esturniolo/
2024/03/25 22:55:34 DEBUG : fs cache: renaming cache item "webdav:/backups/proxmox" to be canonical "webdav:backups/proxmox"
2024/03/25 22:55:34 INFO  : webdav root 'backups/proxmox': poll-interval is not supported by this remote
2024/03/25 22:55:34 DEBUG : vfs cache: root is "/root/.cache/rclone"
2024/03/25 22:55:34 DEBUG : vfs cache: data root is "/root/.cache/rclone/vfs/webdav/backups/proxmox"
2024/03/25 22:55:34 DEBUG : vfs cache: metadata root is "/root/.cache/rclone/vfsMeta/webdav/backups/proxmox"
2024/03/25 22:55:34 DEBUG : Creating backend with remote "/root/.cache/rclone/vfs/webdav/backups/proxmox"
2024/03/25 22:55:34 DEBUG : Creating backend with remote "/root/.cache/rclone/vfsMeta/webdav/backups/proxmox"
2024/03/25 22:55:34 DEBUG : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst: vfs cache: truncate to size=13375254423 (not needed as size correct)
2024/03/25 22:55:34 DEBUG : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst: vfs cache: setting modification time to 2024-03-23 19:34:33.265454117 +0100 CET
2024/03/25 22:55:34 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst: vfs cache: queuing for upload in 5s
2024/03/25 22:55:35 DEBUG : backups/dump: Added virtual directory entry vAddFile: "vzdump-lxc-100-2024_03_23-19_32_13.tar.zst"
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst: vfs cache: truncate to size=12646655931 (not needed as size correct)
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst: vfs cache: setting modification time to 2024-03-23 19:39:34.140174593 +0100 CET
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst: vfs cache: queuing for upload in 5s
2024/03/25 22:55:35 DEBUG : backups/dump: Added virtual directory entry vAddFile: "vzdump-lxc-105-2024_03_23-19_35_56.tar.zst"
2024/03/25 22:55:35 DEBUG : webdav root 'backups/proxmox': Mounting on "/root/rcloneMount/"
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.log: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.log: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.log was removed, freed 1107 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-105-2024_02_23-02_00_01.tar.zst.notes: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-105-2024_02_23-02_00_01.tar.zst.notes: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-105-2024_02_23-02_00_01.tar.zst.notes was removed, freed 15 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.log: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.log: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.log was removed, freed 1084 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.log: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.log: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.log was removed, freed 951 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-100-2024_03_23-19_32_13.tar.zst.notes was removed, freed 6 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-103-2024_03_23-19_41_28.tar.zst was removed, freed 482730024 bytes
2024/03/25 22:55:35 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst not removed, freed 0 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/03/25 22:55:35 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item backups/dump/vzdump-lxc-105-2024_03_23-19_35_56.tar.zst.notes was removed, freed 15 bytes
2024/03/25 22:55:35 INFO  : backups/dump/vzdump-qemu-100-2023_12_10-18_01_27.vma.zst.notes: vfs cache: removed cache file as Removing old cache file not in use
2024/03/25 22:55:35 DEBUG : backups/dump/vzdump-qemu-100-2023_12_10-18_01_27.vma.zst.notes: vfs cache: removed metadata from cache as Removing old cache file not in use

BTW, the flags that I'm using seems to be the correct ones for my use case?
A 24/7 mounted WebDAV into a directory to get accessed around 10 times a day during 1 or 2 hours during midnight.
Why a mounted directory instead a copy flag on demand?
Because with this I try to "cheat" Proxmox. It only see volumes, so with this Proxmox can see my remote storage and treat it as if it were a physical volume, dealing with the backup retention policies and another Proxmox stuff.

Due this last experience it seems that this configurations works with this newest version, but may be I can improve something.

Thanks!