Windows webdav mount thinks it may write to read-only files on webdav server

What is the problem you are having with rclone?

Windows thinks it can write files to folders on a webdav mount which really are read-only on the server. This results in rclone error messages which never show up on the Windows File Explorer level. So the Windows user thinks he/she has successfully written a file, while really the file was never written to the server.

Context: The webdav server is shared by multiple users who each have their own user/login on the server. So for any given user, some of the files/folders on the webdav share are read/writeable, but others are not.

What should have happened: rclone should "know" those files are read-only and produce a Windows error message, just as if the user were writing to some other read-only location on disk.

What is your rclone version (output from rclone version)

1.57.0 64-bit on Windows 10

Which cloud storage system are you using? (eg Google Drive)

webdav on an internal Linux server

The command you were trying to run (eg rclone copy /tmp remote:tmp)

not using a command for this but the Windows Explorer after having mounted it with

rclone.exe mount webdav: X: --network-mode --vfs-cache-mode writes --log-file ./rclone.log --log-level DEBUG --retries 1 --low-level-retries 2

The rclone config contents with secrets removed.

type = webdav
pass = xxxxx
url = https://10.XX.YY.ZZ
user = username

A log from the command with the -vv flag

See 2021/12/13 14:42:20 DEBUG : rclone: Version "v1.57.0" starting with parameters [ -
(expires in 6 months)

Thanks for any pointers!

Not sure how rclone would 'know' it's read only.

It looks like it properly generated an error saying you can't write to it:

<p>You don't have permission to access this resource.</p>

I'm not aware of any programs that 'know' as they try to write and would generate an error.

felix@gemini:~$ sudo touch readonly
felix@gemini:~$ cp /etc/hosts readonly
cp: cannot create regular file 'readonly': Permission denied

Thanks for the fast reply!

The key problem is that Windows never learns about the write failure, so the user will never know (without looking into the rclone logs).

Maybe using the word "know" is misleading: all I was hoping for is some way for a user to learn that the file could not be written - and also a way to stop those writing attempts (as now rclone retries to write those files whenever it is started, again and again).

And actually that error message you quoted is generated by the webdav server, not by rclone, so I am not even sure rclone understands the error?

With that, you have pushed the error back into the background as it'll retry a few times to see if it fails and finally fail in the background and write an error to the log.

You'd want to scrape the log for errors or properly setup permissions for the mount to make sure it can write. You can try without cache mode but not sure that would work with your application and how it writes.

  • how can the Windows user learn about the write failures if the error is "back into the background"?
  • why does rclone not stop retrying? Here it has not stopped, only waited for even longer time.
  • with this shared server, I cannot "setup permissions" on the server to fix this; these permissions are exactly as they need to be. The problem comes from Windows/rclone not presenting the files/folders in the mounted drive as read-only.

I'm not a Windows guy so I won't even guess on this other than scraping the log.

The default is 3 times but I guess you stopped before the 3rd one.

2021/12/13 14:42:27 ERROR : readonlydir/ vfs cache: failed to upload try #1, will retry in 10s:
2021/12/13 14:42:38 ERROR : readonlydir/ vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

I'd rephrase that as the issue is the same for any application but you've specifically setup caching and caching writes in the background as you've asked it to cache. Any program that would do background writes / retries would hit the same issue. The file isn't know to any other program/application as read only until you try to write to it and it tries to.

Applications generally don't precache file permissions.

To clarify some more the issue:

  • In our current setup, Windows never learns the file could not be written. It thinks everything is fine, while it really is not. This is dangerous (file not on server) and confusing (other users cannot see the uploaded file).

  • From my naive view, the fundamental source of the problem is that the folder I am writing to has read-only permissions on the server, but rclone ignores this.

  • While not included in the uploaded log, rclone keeps retrying, for many hours, with increasing delay between the retries. I would like to change this, so it stops and fails fatally after a few attempts.

All this really looks like a configuration problem on our side but I do not know what to change where...

It does as it's written to the log file as an error after 3 retries as that's the default.

It doesn't know that until it tries to write a file as you have ACLs I'm guessing on the specific folder as it's not at the sharing level on the folder. If items have ACLs on them, it doesn't know until it tries to write as it would be hugely inefficient to read every single file on startup and maintain a list of file permissions.

You'd have to share a full debug log of the issue as guessing or assuming wastes your time and my time.

To add some context, it's similar in Linux if you have a directory that has read permissions but no write.

If I don't use cache mode as a user, I get an error:

felix@gemini:~/testmount/readonly$ cp /etc/hosts .
cp: failed to close './hosts': Input/output error

With writes on, you'd want to look for errors:

2021/12/13 09:42:11 DEBUG : readonly/hosts: vfs cache: starting upload
2021/12/13 09:42:11 ERROR : readonly/hosts: Failed to copy: open /home/felix/test/readonly/hosts: permission denied
2021/12/13 09:42:11 ERROR : readonly/hosts: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: open /home/felix/test/readonly/hosts: permission denied

Since the write failed and from the user's perspective, it'll keep retrying the upload and persist in the cache so the file isn't lost.

I mistakenly stated that 3 is the default as that's the normal number for copy/sync:

      --retries int                          Retry operations this many times if they fail (default 3)

With full/writes mode on cache, it exponentially keeps going until it succeeds (I think). I need to dig a bit and see and perhaps maybe @ivandeex or @ncw can chime in on the changes as I don't use writes since it's been updated semi recently.

Not giving up yet... :wink:

  • The Windows OS does not learn the file could not be written. How would it know about some text file that happens to be a log file for rclone? Windows Explorer never generates an error.

  • From what I see in the log: rclone knows that it could not write the file, but it seems not to understand why that is the case. It just reports "Failed to copy" and keeps retrying. NB: I will upload a longer log file (currently at 3500 lines) just to prove to you that this is indeed happening. I am not making this up just to keep you busy.

  • The webdav server is not using any ACLs, just simple Linux file permissions. But none of these get carried over to the Windows mount; all files/folders inside the mounted drive (on Windows) claim that the current Windows user has "Full control" over them (which is not true on the server). What would be needed is for the mounted file system to indicate that one of its items is read-only...

I think I've answered this a few times. It writes it in the background and returns to the prompt to queue the upload with cache mode writes..

See above post as I've answered this there.

See above post as I was giving a Windows example as you mentioned Windows. Linux is the same as my example above.

Please find the extended log file at 2021/12/13 14:42:20 DEBUG : rclone: Version "v1.57.0" starting with parameters [ -

As you can see, rclone keeps retrying to save the file rclone.exe to the remote folder readonlyfolder sixteen times between 15:25:33 and 16:21:16 and not giving up...

Right I've said that:

Are you seeing my posts?

What about the option --retries 1 that was used here - according to Documentation this is supposed to disable retries?

FWIW, I get the same result when using --vfs-cache-mode minimal, so it is not related to write caching.

When using --vfs-cache-mode off I still do not get any obvious file write error but a slightly different log file section:

2021/12/13 16:42:59 DEBUG : /readonlyfolder: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/12/13 16:42:59 DEBUG : /readonlyfolder: >Getattr: errc=0
2021/12/13 16:42:59 DEBUG : /readonlyfolder: Releasedir: fh=0x6
2021/12/13 16:42:59 DEBUG : /readonlyfolder: >Releasedir: errc=0
2021/12/13 16:43:01 ERROR : readonlyfolder/rclone.exe: Failed to copy: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<title>403 Forbidden</title>
<p>You don't have permission to access this resource.</p>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 ERROR : readonlyfolder/rclone.exe: WriteFileHandle.New Rcat failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<title>403 Forbidden</title>
<p>You don't have permission to access this resource.</p>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 ERROR : readonlyfolder/rclone.exe: WriteFileHandle.Flush error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<title>403 Forbidden</title>
<p>You don't have permission to access this resource.</p>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 ERROR : IO error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<title>403 Forbidden</title>
<p>You don't have permission to access this resource.</p>
<address>Apache/2.4.29 (Ubuntu) Server at webdavserver Port 80</address>
</body></html>: 403 Forbidden
2021/12/13 16:43:01 DEBUG : /readonlyfolder/rclone.exe: >Flush: errc=-5
2021/12/13 16:43:01 DEBUG : /readonlyfolder/rclone.exe: Release: fh=0x2
2021/12/13 16:43:01 DEBUG : readonlyfolder/rclone.exe: WriteFileHandle.Release nothing to do
2021/12/13 16:43:01 DEBUG : /readonlyfolder/rclone.exe: >Release: errc=0

After this, the mounted Windows drive still shows the file in the (readonly) folder even though rclone has given up.

This is what I get when trying to delete the "copied" file from the mounted drive:

just wanted to point out a few things.

  1. windows can connect directly to webdav servers and mount it as local storage without the need for rclone mount. might try testing with that.
  2. rclone can emulate a webdav server, i used this command, tried to edit a file and got the following error message.
    rclone serve webdav remote: --read-only --addr=:8080 --user=user --pass=pass --vfs-cache-mode=full --cache-dir=D:\data\rclone\cache\remote


A., thank you for joining!

To your points:

  1. Unfortunately, Windows webdav is about five times slower than rclone, so it is not a final solution. But I can of course use it for testing! Will do so asap and report back.
  2. That error looks exactly like what I am looking for - but you are using --read-only (NB: seems strange to combine this with --vfs-cache-mode=full?).
    With --read-only option I also get an appropriate error message - but in our setup, we have some folders that are read-only and others (depending on the current user) that are read-write.
    Would you mind doing the same without --read-only and try to write to a read-only folder?


Edit: Small addition for the time being: I realize that when I try to write to the readonlyfolder using one of the other (non-mounting) webdav clients (Cyberduck and WinSCP), they also start uploading the file and only after uploading is complete, at the very end, report a 403 error...

  1. ok. report back.
  2. well, good point. i quickly took a rclone mount command and turned it into a rclone serve webdav.
    and now i better understand your use case.

is there a specific reason to use a webdav server, instead of samba which has very detailed folder/file permissions.

not sure what read-only means?

  1. prevent the win user from editing/saving an existing document such as a .doc
  2. prevent the win user from copying/deleting files on the webdav server.

"read-only" folder here means a folder into which the user (connected through webdav) cannot edit/delete existing files, nor create new ones.

So this does not apply to the entire webdav share visible to the user, only some folders in it; this is controlled by the normal UNIX file permissions, groups and ownerships on the server.

again, curious about the requirement to use a webdav server instead of samba?