Windows thinks it can write files to folders on a webdav mount which really are read-only on the server. This results in rclone error messages which never show up on the Windows File Explorer level. So the Windows user thinks he/she has successfully written a file, while really the file was never written to the server.
Context: The webdav server is shared by multiple users who each have their own user/login on the server. So for any given user, some of the files/folders on the webdav share are read/writeable, but others are not.
What should have happened: rclone should "know" those files are read-only and produce a Windows error message, just as if the user were writing to some other read-only location on disk.
What is your rclone version (output from rclone version)
1.57.0 64-bit on Windows 10
Which cloud storage system are you using? (eg Google Drive)
webdav on an internal Linux server
The command you were trying to run (eg rclone copy /tmp remote:tmp)
not using a command for this but the Windows Explorer after having mounted it with
The key problem is that Windows never learns about the write failure, so the user will never know (without looking into the rclone logs).
Maybe using the word "know" is misleading: all I was hoping for is some way for a user to learn that the file could not be written - and also a way to stop those writing attempts (as now rclone retries to write those files whenever it is started, again and again).
And actually that error message you quoted is generated by the webdav server, not by rclone, so I am not even sure rclone understands the error?
With that, you have pushed the error back into the background as it'll retry a few times to see if it fails and finally fail in the background and write an error to the log.
You'd want to scrape the log for errors or properly setup permissions for the mount to make sure it can write. You can try without cache mode but not sure that would work with your application and how it writes.
how can the Windows user learn about the write failures if the error is "back into the background"?
why does rclone not stop retrying? Here it has not stopped, only waited for even longer time.
with this shared server, I cannot "setup permissions" on the server to fix this; these permissions are exactly as they need to be. The problem comes from Windows/rclone not presenting the files/folders in the mounted drive as read-only.
I'm not a Windows guy so I won't even guess on this other than scraping the log.
The default is 3 times but I guess you stopped before the 3rd one.
2021/12/13 14:42:27 ERROR : readonlydir/rclone-v1.57.0-windows-amd64.zip: vfs cache: failed to upload try #1, will retry in 10s:
2021/12/13 14:42:38 ERROR : readonlydir/rclone-v1.57.0-windows-amd64.zip: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
I'd rephrase that as the issue is the same for any application but you've specifically setup caching and caching writes in the background as you've asked it to cache. Any program that would do background writes / retries would hit the same issue. The file isn't know to any other program/application as read only until you try to write to it and it tries to.
Applications generally don't precache file permissions.
In our current setup, Windows never learns the file could not be written. It thinks everything is fine, while it really is not. This is dangerous (file not on server) and confusing (other users cannot see the uploaded file).
From my naive view, the fundamental source of the problem is that the folder I am writing to has read-only permissions on the server, but rclone ignores this.
While not included in the uploaded log, rclone keeps retrying, for many hours, with increasing delay between the retries. I would like to change this, so it stops and fails fatally after a few attempts.
All this really looks like a configuration problem on our side but I do not know what to change where...
It does as it's written to the log file as an error after 3 retries as that's the default.
It doesn't know that until it tries to write a file as you have ACLs I'm guessing on the specific folder as it's not at the sharing level on the folder. If items have ACLs on them, it doesn't know until it tries to write as it would be hugely inefficient to read every single file on startup and maintain a list of file permissions.
You'd have to share a full debug log of the issue as guessing or assuming wastes your time and my time.
To add some context, it's similar in Linux if you have a directory that has read permissions but no write.
If I don't use cache mode as a user, I get an error:
felix@gemini:~/testmount/readonly$ cp /etc/hosts .
cp: failed to close './hosts': Input/output error
With writes on, you'd want to look for errors:
2021/12/13 09:42:11 DEBUG : readonly/hosts: vfs cache: starting upload
2021/12/13 09:42:11 ERROR : readonly/hosts: Failed to copy: open /home/felix/test/readonly/hosts: permission denied
2021/12/13 09:42:11 ERROR : readonly/hosts: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: open /home/felix/test/readonly/hosts: permission denied
Since the write failed and from the user's perspective, it'll keep retrying the upload and persist in the cache so the file isn't lost.
I mistakenly stated that 3 is the default as that's the normal number for copy/sync:
--retries int Retry operations this many times if they fail (default 3)
With full/writes mode on cache, it exponentially keeps going until it succeeds (I think). I need to dig a bit and see and perhaps maybe @ivandeex or @ncw can chime in on the changes as I don't use writes since it's been updated semi recently.
The Windows OS does not learn the file could not be written. How would it know about some text file that happens to be a log file for rclone? Windows Explorer never generates an error.
From what I see in the log: rclone knows that it could not write the file, but it seems not to understand why that is the case. It just reports "Failed to copy" and keeps retrying. NB: I will upload a longer log file (currently at 3500 lines) just to prove to you that this is indeed happening. I am not making this up just to keep you busy.
The webdav server is not using any ACLs, just simple Linux file permissions. But none of these get carried over to the Windows mount; all files/folders inside the mounted drive (on Windows) claim that the current Windows user has "Full control" over them (which is not true on the server). What would be needed is for the mounted file system to indicate that one of its items is read-only...
As you can see, rclone keeps retrying to save the file rclone.exe to the remote folder readonlyfolder sixteen times between 15:25:33 and 16:21:16 and not giving up...
windows can connect directly to webdav servers and mount it as local storage without the need for rclone mount. might try testing with that.
rclone can emulate a webdav server, i used this command, tried to edit a file and got the following error message. rclone serve webdav remote: --read-only --addr=:8080 --user=user --pass=pass --vfs-cache-mode=full --cache-dir=D:\data\rclone\cache\remote
Unfortunately, Windows webdav is about five times slower than rclone, so it is not a final solution. But I can of course use it for testing! Will do so asap and report back.
That error looks exactly like what I am looking for - but you are using --read-only (NB: seems strange to combine this with --vfs-cache-mode=full?).
With --read-only option I also get an appropriate error message - but in our setup, we have some folders that are read-only and others (depending on the current user) that are read-write.
Would you mind doing the same without --read-only and try to write to a read-only folder?
Thanks!
Edit: Small addition for the time being: I realize that when I try to write to the readonlyfolder using one of the other (non-mounting) webdav clients (Cyberduck and WinSCP), they also start uploading the file and only after uploading is complete, at the very end, report a 403 error...
"read-only" folder here means a folder into which the user (connected through webdav) cannot edit/delete existing files, nor create new ones.
So this does not apply to the entire webdav share visible to the user, only some folders in it; this is controlled by the normal UNIX file permissions, groups and ownerships on the server.
