Support `PutStream` for webdav servers with partial update support

What problem are you are trying to solve?

Currently, WebDAV's Putstream feature is disabled for all vendors except owncloud:

Which will result in 1. A warning when using rclone mount with webdav server:

and 2. RCat operation in rclone mount will create a temporary local fs, copy the object to tmpLocalFs, finally sending the whole object to webdav server:

Therefore, when copying a file to a mounted webdav server, the progress bar will finish in a second, but the file is actually being uploaded in the background.

This is not user friendly as completed file uploading cannot be indicated by a finished progress bar.

How do you think rclone should be changed to solve that?

If a webdav remote supports partial update, then we could provide PutStream feature based on its partial update specification, so the problem is resolved.

Although how partial update should work is not specified in webdav specification, there are several non-standard ways implemented by some widely used webdav providers:

New open source webdav servers generally recognize these two implementations as de facto standard, like this rust implementation for webdav server. So, by supporting these two partial update methods, varieties of webdav remotes will benefit from it.

These two methods all provide some ways to inform clients of their existence.

For SABREDAV:

If you want to be compliant with SabreDAV's implementation of PATCH, you must also return 'sabredav-partialupdate' in the 'DAV:' header:

HTTP/1.1 204 No Content
DAV: 1, 2, 3, sabredav-partialupdate, extended-mkcol

This is only required if you are adding this feature to a DAV server. For non-webdav implementations such as REST services this is optional.

For Apache:

There is no official way to know if PUT-with-content-range is supported by a webserver. For a client it's probably best to do an OPTIONS request, and then check two things:

  • the Server header must contain the word Apache
  • the DAV header must contain <http://apache.org/dav/propset/fs/1>.

In that case, your are sure to talk to an Apache webserver with mod_dav enabled.

Therefore, support of partial update feature could be auto-detected and hence no need to change the option (Or maybe some vendors could be added to allow users to manually indicate partial update is supported?).

If this is a feature worth to support, I am willing to implement it myself, and hence seek for suggestions and guidance about the desired design, where to put the auto detection code, etc.

Related Posts

That is probably unnecessarily restrictive...

All that is required for PutStream is that the server supports chunked transfer encoding without a Content-Length: header.

This shouldn't be too hard and maybe the servers you want to use really do support it.

Before implementing the partial update (which looks interesting) it might be worth just setting canStream unconditionally and see if your backend works?

I'd also like to implement a quirks system so you can override the quirks rclone chooses for you. This would be useful in a number of backends.

I specified the vendor of my webdav remote to be "owncloud", and did a simple file upload (about 170MB):

  1. Upload succeeded with progress bar working as expected;
  2. The file is still valid after downloading from remote.

But the rclone mount log shows some errors:

ERROR : Script.pvf: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
ERROR : Script.pvf: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
ERROR : Script.pvf: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
ERROR : Script.pvf: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes

ERROR : Script.pvf: WriteFileHandle.New Rcat failed: corrupted on transfer
ERROR : Script.pvf: WriteFileHandle.Flush error: corrupted on transfer
ERROR : IO error: corrupted on transfer

Relevant debug output:

(Upload begins)

DEBUG : /Script.pvf: Truncate: size=181355213, fh=0x0
ERROR : Script.pvf: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
DEBUG : /Script.pvf: >Truncate: errc=-1

... (After upload finishes) 

DEBUG : Script.pvf: Modification times differ by 17.4935035s: 2021-11-04 00:53:02.5064965 +0800 CST m=+6767.635558101, 2021-11-03 16:53:20 +0000 GMT
ERROR : Script.pvf: corrupted on transfer
ERROR : Script.pvf: WriteFileHandle.New Rcat failed: corrupted on transfer
ERROR : Script.pvf: WriteFileHandle.Flush error: corrupted on transfer
ERROR : IO error: corrupted on transfer

Below is the trace I've done until now:

Focus on the errors after upload finished, the error occured during RCat's compare function:

Notice the log message Modification times differ..., there is no more log message between it and the final corrupted on transfer message.

Looking through the source code, only one place will not log any message before return:

So that means CheckHashs(ctx, src, dst) returned (true, hash.None, _), meaning that the hash couldn't be checked. Anything I'm missing? (or derived from incompatibility with "owncloud" vendor?)

All that is required for PutStream is that the server supports chunked transfer encoding without a Content-Length: header.

About the server support, I learned that rclone will pass object of size -1 when calling PutStream:

and golang's http module will convert Content-Length of -1 to Transfer-Encoding: trunked automatically.

My webdav server uses rust's hyper package as http server backend, and it correctly processed requests without Content-Length.

Therefore, the webdav handler could normally read and process body from PUT request as a stream.

So that may mean PutStream could be easily supported by those webdav servers if based on some mature http implementation like hyper (thus no need to introduce partial update for now), the "owncloud" constraint seems unnecessarily restrictive.

Owncloud sets too many quirks for your provider

        case "owncloud":
                f.canStream = true
                f.precision = time.Second
                f.useOCMtime = true
                f.hasMD5 = true
                f.hasSHA1 = true

You want to set only f.canStream = true for this experiment.

What we should really do is make the quirks configurable from the command line.

I think if you try that (you'll have to modify the source) then it should work.

I am so looking forward to this fix! I'll be happy to test it and report the results.