Windows webdav mount thinks it may write to read-only files on webdav server

It does as it's written to the log file as an error after 3 retries as that's the default.

It doesn't know that until it tries to write a file as you have ACLs I'm guessing on the specific folder as it's not at the sharing level on the folder. If items have ACLs on them, it doesn't know until it tries to write as it would be hugely inefficient to read every single file on startup and maintain a list of file permissions.

You'd have to share a full debug log of the issue as guessing or assuming wastes your time and my time.

To add some context, it's similar in Linux if you have a directory that has read permissions but no write.

If I don't use cache mode as a user, I get an error:

felix@gemini:~/testmount/readonly$ cp /etc/hosts .
cp: failed to close './hosts': Input/output error

With writes on, you'd want to look for errors:

2021/12/13 09:42:11 DEBUG : readonly/hosts: vfs cache: starting upload
2021/12/13 09:42:11 ERROR : readonly/hosts: Failed to copy: open /home/felix/test/readonly/hosts: permission denied
2021/12/13 09:42:11 ERROR : readonly/hosts: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: open /home/felix/test/readonly/hosts: permission denied

Since the write failed and from the user's perspective, it'll keep retrying the upload and persist in the cache so the file isn't lost.

I mistakenly stated that 3 is the default as that's the normal number for copy/sync:

      --retries int                          Retry operations this many times if they fail (default 3)

With full/writes mode on cache, it exponentially keeps going until it succeeds (I think). I need to dig a bit and see and perhaps maybe @ivandeex or @ncw can chime in on the changes as I don't use writes since it's been updated semi recently.

Not giving up yet... :wink:

  • The Windows OS does not learn the file could not be written. How would it know about some text file that happens to be a log file for rclone? Windows Explorer never generates an error.

  • From what I see in the log: rclone knows that it could not write the file, but it seems not to understand why that is the case. It just reports "Failed to copy" and keeps retrying. NB: I will upload a longer log file (currently at 3500 lines) just to prove to you that this is indeed happening. I am not making this up just to keep you busy.

  • The webdav server is not using any ACLs, just simple Linux file permissions. But none of these get carried over to the Windows mount; all files/folders inside the mounted drive (on Windows) claim that the current Windows user has "Full control" over them (which is not true on the server). What would be needed is for the mounted file system to indicate that one of its items is read-only...

I think I've answered this a few times. It writes it in the background and returns to the prompt to queue the upload with cache mode writes..

See above post as I've answered this there.

See above post as I was giving a Windows example as you mentioned Windows. Linux is the same as my example above.

Please find the extended log file at 2021/12/13 14:42:20 DEBUG : rclone: Version "v1.57.0" starting with parameters [ - Pastebin.com

As you can see, rclone keeps retrying to save the file rclone.exe to the remote folder readonlyfolder sixteen times between 15:25:33 and 16:21:16 and not giving up...

Right I've said that:

Are you seeing my posts?

What about the option --retries 1 that was used here - according to Documentation this is supposed to disable retries?

FWIW, I get the same result when using --vfs-cache-mode minimal, so it is not related to write caching.

When using --vfs-cache-mode off I still do not get any obvious file write error but a slightly different log file section:

2021/12/13 16:42:59 DEBUG : /readonlyfolder: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/12/13 16:42:59 DEBUG : /readonlyfolder: >Getattr: errc=0
2021/12/13 16:42:59 DEBUG : /readonlyfolder: Releasedir: fh=0x6
2021/12/13 16:42:59 DEBUG : /readonlyfolder: >Releasedir: errc=0
2021/12/13 16:43:01 ERROR : readonlyfolder/rclone.exe: Failed to copy: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 ERROR : readonlyfolder/rclone.exe: WriteFileHandle.New Rcat failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 ERROR : readonlyfolder/rclone.exe: WriteFileHandle.Flush error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 ERROR : IO error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 DEBUG : /readonlyfolder/rclone.exe: >Flush: errc=-5
2021/12/13 16:43:01 DEBUG : /readonlyfolder/rclone.exe: Release: fh=0x2
2021/12/13 16:43:01 DEBUG : readonlyfolder/rclone.exe: WriteFileHandle.Release nothing to do
2021/12/13 16:43:01 DEBUG : /readonlyfolder/rclone.exe: >Release: errc=0

After this, the mounted Windows drive still shows the file in the (readonly) folder even though rclone has given up.

This is what I get when trying to delete the "copied" file from the mounted drive:

hi,
just wanted to point out a few things.

  1. windows can connect directly to webdav servers and mount it as local storage without the need for rclone mount. might try testing with that.
  2. rclone can emulate a webdav server, i used this command, tried to edit a file and got the following error message.
    rclone serve webdav remote: --read-only --addr=:8080 --user=user --pass=pass --vfs-cache-mode=full --cache-dir=D:\data\rclone\cache\remote

image

A., thank you for joining!

To your points:

  1. Unfortunately, Windows webdav is about five times slower than rclone, so it is not a final solution. But I can of course use it for testing! Will do so asap and report back.
  2. That error looks exactly like what I am looking for - but you are using --read-only (NB: seems strange to combine this with --vfs-cache-mode=full?).
    With --read-only option I also get an appropriate error message - but in our setup, we have some folders that are read-only and others (depending on the current user) that are read-write.
    Would you mind doing the same without --read-only and try to write to a read-only folder?

Thanks!

Edit: Small addition for the time being: I realize that when I try to write to the readonlyfolder using one of the other (non-mounting) webdav clients (Cyberduck and WinSCP), they also start uploading the file and only after uploading is complete, at the very end, report a 403 error...

  1. ok. report back.
  2. well, good point. i quickly took a rclone mount command and turned it into a rclone serve webdav.
    and now i better understand your use case.

is there a specific reason to use a webdav server, instead of samba which has very detailed folder/file permissions.

not sure what read-only means?

  1. prevent the win user from editing/saving an existing document such as a .doc
    and/or
  2. prevent the win user from copying/deleting files on the webdav server.

"read-only" folder here means a folder into which the user (connected through webdav) cannot edit/delete existing files, nor create new ones.

So this does not apply to the entire webdav share visible to the user, only some folders in it; this is controlled by the normal UNIX file permissions, groups and ownerships on the server.

again, curious about the requirement to use a webdav server instead of samba?

Excellent question!

And yes, Samba or NFS would have been my preference, too, but unfortunately the server behind the webdav server does not do any of these.
Currently, webdav seems to be the only semi-standard way to get files in and out - and we were delighted to see the speed at which rclone does this; now we are trying to figure out whether we have to make this a --read-only mount...

a shame, on that linux server, cannot install samba or some spin up a docker or some container with samba support.

imho, a rclone mount --vfs-cache-mode=writes over a webdav server with a combintation of ro+rw folders is not a viable solution.

Regarding "ro+rw" - thanks for confirming.

I start to wonder whether webdav in general is capable of transmitting/translating the permissions at all...

That leaves us with the remaining issue of rclone never giving up retrying to write something it cannot write to, despite the --retries 1 option... (see above) and the (IMO) failure to promote that error to the Windows user

to be clear, i was just stating my opinion, not confirming anything.
it might be possible to battle test a solution or perhaps wonder about another out of the box idea.

So is this a bug that the Windows file system mounted by rclone does not get notified of the write error (unless --read-only is used)?

I've answered that numerous times already...

Please see my above posts as I've answered this one too...

No, it is generally impossible to return file update errors from a mount using a write-behind cache, this is nothing specific to rclone, webdav or Windows. This is what @Animosity022 has been trying to tell you all the time.

You see a similar situation with the native OneDrive client on Windows if you provoke a sync/write conflict – it only shows the issue/error in the extended OneDrive status view. I guess you can see the same in the clients for Google Drive, Dropbox, etc.

The benefit of write-behind caching is the improved resilience in case of network glitches and the improved user experience:

but it sometimes comes at a price, as you have seen.

It may be possible to report an error if using --vfs-cache-mode=off, but this limits the usability significantly and it probably has the same response times as the native webdav support in Windows.

I would either go for the native webdav support in Windows, or split the data such that each user has two rclone mounts (one writeable and one --read-only).