Finder cannot copy big file onto self-hosted (nginx dav+ext) WebDAV macFUSE mount

Suspected Bug
1

What is the problem you are having with rclone?

When I drag & drop a big (660 MB, but I tried bigger) file from one folder to the WebDAV mount folder, the shown progress either freezes at zero or immediately shows a huge (403 MB - impossible with my network connection so fast) amount of progress, stops at exactly this point even though network transfer is going at fast pace (see logs and I double-checked through monitoring network activity through Activity Monitor), and eventually Finder fails with the message "The operation can't be completed because the device disappeared."

Despite the error, mount is live and continues working.

I've tried restarting (not reloading) nginx, restarting computer with rclone (to reset macFUSE state if any), restarting rclone itself, and combinations of those, removing all remnants of the file I'm trying to copy from remote through ssh if anything left every time. Reproduces every time.

I've been monitoring the client_body_temp_path folder on remote via ssh while the transfer was shown as stalled in Finder before failing, and the file is there and is steadily growing, until eventually deleted. So it seems rclone is transferring the file, Finder misrepresents the progress, and then abruptly fails.

I did not find any specific (albeit big) size or longetivity of the transfer at which it fails.

I've tried

dd if=same_file of=mount/file bs=16M
cp same_file mount/

Similar problem.

As far as I'm concerned these should work just fine without caching.

I've tried

curl same/url/as/mounted/test.file --upload-file same_file -v

It works without issue.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

  • os/version: darwin 15.0.1 (64 bit)
  • os/kernel: 24.0.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.23.1
  • go/linking: dynamic
  • go/tags: cmount

macFUSE 4.82 DMG from the official website

Which cloud storage system are you using? (eg Google Drive)

Self-hosted WebDAV (nginx 1.27.2 ngx_http_dav_module+ngx_http_dav_ext_module). Config excerpt:

server {
    location *** {
        alias ***;

        auth_basic "closed site";
        auth_basic_user_file htpasswd;

        dav_methods PUT DELETE MKCOL COPY MOVE;
        dav_access user:rw group:r all:r;

        client_body_temp_path *** 1 2 3;
        client_max_body_size 0;

        dav_ext_methods PROPFIND OPTIONS;
    }
}

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount \
  --daemon \
  --log-file=*** \
  --log-level=DEBUG \
  --use-server-modtime \
  --vfs-cache-mode=off \
  --vfs-case-insensitive=false \
  ***: ***

I tried adding --direct-io.

The rclone config contents with secrets removed.

[***]
type = webdav
url = ***
vendor = other
user = ***
pass = ***
pacer_min_sleep = 0s

A log from the command with the -vv flag

rclone logs https://pastebin.com/raw/yMjufW31 (expires in a week)
It starts exactly at the log entry where I drop the file to the WebDAV folder.
I replaced repeated similar entries with ... and some names with ***.

The only relevant nginx logs

*** - *** [26/Oct/2024:03:16:34 +0000] "PUT /*** HTTP/2.0" 201 0 "***" "rclone/v1.68.1" "-"
*** - *** [26/Oct/2024:03:16:34 +0000] "PROPFIND /*** HTTP/2.0" 207 619 "***" "rclone/v1.68.1" "-"
*** - *** [26/Oct/2024:03:16:34 +0000] "DELETE /*** HTTP/2.0" 204 0 "***" "rclone/v1.68.1" "-"
*** - *** [26/Oct/2024:03:17:19 +0000] "PUT /*** HTTP/2.0" 201 0 "***" "rclone/v1.68.1" "-"
*** - *** [26/Oct/2024:03:17:19 +0000] "PROPFIND /*** HTTP/2.0" 207 627 "***" "rclone/v1.68.1" "-"
2

welcome to the forum,

rclone mount is more than a middle-man for file copying.
has to emulate a local filesystem, including random access read+writes.

well, you are comparing two different approaches to copying files.
the rclone equivalent to curl, is rclone copy, not rclone mount

@kapitainsky ?

3

I would try FUSE-T. You can install it using brew:

brew tap macos-fuse-t/homebrew-cask
brew install fuse-t

I think it is better to remove macFUSE before. I stopped using it ages ago as it was becoming more and more problematic with every macOS release. FUSE-T is kext-less and is using Apple NFS server.

OR

use rclone own rclone nfsmount instead. Like FUSE-T it is using NFS server - but one implemented in golang.

4

Do you mind me asking if NFS approach through FUSE-T has any cache of their own? And I don't mean low-level cache in a sense that kernel might buffer writes until the very small buffer is filled, the miniscule amount of time passes, or flush is explicitly called. I rather mean something high-level similar to vfs-cache-mode=minimal/writes/full.

I'm trying to avoid any such cache, even though I understand it limits possible write modes, and sometimes performance.

5

Thanks, I tried rclone copy and it worked without problems. I need mounted FS, though.

I provided curl simply to demonstrate that remote is willing to accept such a file, that the problem is not in remote configuration or infra. rclone copy would have been a better demonstration.

6

rclone nfsmount is read-only with vfs-cache-mode=off so unfortunately does not work for me.

FUSE-T is not read-only with vfs-cache-mode=off; seems to be fully caching the file somewhere before starting the transfer as I see spike of very fast traffic in Activity Monitor, and only then transfer starts; fails instantly when trying to drag & drop the file in Finder; running cp from Terminal uploads the file fine eventually but never exits, shows "nfs server fuse-t:/***: not responding" repeatedly while copying, once the file is uploaded rclone crashes with Fatal error leaving FUSE-T mount dangling and unresponsive.

In other words, so far macFUSE works the best despite the original post.

7

Same behaviour with dd if=/dev/zero of=mount/test bs=16M count=X.

Seems to be breaking either between 512M (count=32) and 1024M (count=64) copied or between 110s and 150s time spent. Smaller files and shorter transfers do not lead to Fatal error.