How to specify a webdav backend root path using connection string syntax

Help and Support
1

What is the problem you are having with rclone?

When doing an rc copy (see command below) it fails on the PROPFIND of the src webdav backend. The issue is that the src webdav service (running at a host with a path: https://app.cs3mesh-iop.k8s.surfsara.nl/iop, which in this case is a particular server setup) returns the resource items with a href starting from that path: eg. /remote.php/webdav/home/folder1/test1.txt

Now the rclone webdav backend implementation compares the returned items' paths with the path of the base url which reads /iop/remote.php/webdav/home/folder1/test1.txt (ie. including the /iop part). And thus it fails to recognize the path (see debug output below).

Shifting the /iop part to the path (url=\"https://app.cs3mesh-iop.k8s.surfsara.nl\":/iop/remote.php/webdav/home/folder1") gives the same result.

So, I tried to use the _root special value to specify a backend root as follows:
rclone -vv copy ":webdav,headers=\"x-access-token,...token...\",url=\"https://app.cs3mesh-iop.k8s.surfsara.nl\",_root=/iop:/remote.php/webdav/home/folder1" /tmp/folder2
but that does not seem to do anything. The root part is missing in the PROPFIND request.

Two questions:

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: centos 8.5.2111 (64 bit)
  • os/kernel: 5.10.102.1-microsoft-standard-WSL2 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

owncloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv copy ":webdav,headers=\"x-access-token,...token...\",url=\"https://app.cs3mesh-iop.k8s.surfsara.nl/iop\":/remote.php/webdav/home/folder1" /tmp/folder2

The rclone config contents with secrets removed.

using connection strings syntax; see command above

A log from the command with the -vv flag

2022/06/02 14:03:11 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "-vv" "copy" ":webdav,headers=\"x-access-token,...token...\",url=\"https://app.cs3mesh-iop.k8s.surfsara.nl/iop\":/remote.php/webdav/home/folder1" "/tmp/folder2"]
2022/06/02 14:03:11 DEBUG : Creating backend with remote ":webdav,headers=\"x-access-token,...token...\",url=\"https://app.cs3mesh-iop.k8s.surfsara.nl/iop\":/remote.php/webdav/home/folder1"
2022/06/02 14:03:11 DEBUG : :webdav: detected overridden config - adding "{kKeur}" suffix to name
2022/06/02 14:03:11 DEBUG : Using config file from "/config/rclone/rclone.conf"
2022/06/02 14:03:11 DEBUG : found headers: x-access-token,...token...
2022/06/02 14:03:11 DEBUG : fs cache: renaming cache item ":webdav,headers=\"x-access-token,...token...\",url=\"https://app.cs3mesh-iop.k8s.surfsara.nl/iop\":/remote.php/webdav/home/folder1" to be canonical ":webdav{kKeur}:remote.php/webdav/home/folder1"
2022/06/02 14:03:11 DEBUG : Creating backend with remote "/tmp/folder2"
2022/06/02 14:03:11 DEBUG : Item with unknown path received: "/remote.php/webdav/home/folder1/", "/iop/remote.php/webdav/home/folder1/"
2022/06/02 14:03:11 DEBUG : Item with unknown path received: "/remote.php/webdav/home/folder1/test1.txt", "/iop/remote.php/webdav/home/folder1/"
2022/06/02 14:03:11 DEBUG : Local file system at /tmp/folder2: Waiting for checks to finish
2022/06/02 14:03:11 DEBUG : Local file system at /tmp/folder2: Waiting for transfers to finish
2022/06/02 14:03:11 INFO  : There was nothing to transfer
2022/06/02 14:03:11 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s

2022/06/02 14:03:11 DEBUG : 4 go routines active
2

A few things:

  1. I've never heard of _root and it isn't in the docs either. Where is that from?
  2. I know you eventually want to use connection strings, but what happens when you move the configurable options out of them and into a config file? You still will need things like the headers in the string (I think).
    • Are you 100% sure your escaping and joining is correct? I've struggled with that a lot and finally just made a quick python tool to CSV encode and then shell escape it to be certain.
3
  1. It's documented here Remote Control / API
  2. For our use case connection strings is the only way.

As far as escaping&joining is concerned, I can only say that I've tried about every conf that I can think of ;). So far I've found that the _root value doesn't seem to be picked up (the way I've hoped).

4

The _root parameter is only for use over the API and will be ignored otherwise.

What you want to do is move the path to the url parameter passed to webdav eg https://app.cs3mesh-iop.k8s.surfsara.nl/iop/remote.php/. That might not work - you might need to move more to the url parameter, but hopefully you get the idea.

5

Hi @ncw ,
Unfortunately that doesn't work either. The baseURL.Path in the check always starts with /iop of the full url path. But our webdav service never returns item paths starting with /iop. It considers that part of the webdav endpoint so to speak.
And so the check fails. Here's eg. the relevant debug output of that check after using your suggestion:
Item with unknown path received: "/remote.php/webdav/home/folder1/", "/iop/remote.php/webdav/home/folder1/"
Moving more fragments over to the url parameter gives a similar result.

I'm still wondering about the nature of this check, why it checks the items' paths against the full request url path even though the first part of it is not included in the request path to begin with? But I also don't know if there's a standard for this.

6

Its because rclone needs to work out the paths of things relative to the root of the webdav server. It therefore wants to take /remote.php/webdav/home/folder1/ and turn it into home/folder1 or something like that.

All the webdav servers I've seen so far work like that, but webdav isn't a very well standardized protocol and there is a lot of variety in servers.

What you are asking for is a URL which is different to the service URL for rclone to remove from all front of all the paths.

It wouldn't be impossible to add such a URL as an additional parameter.

Maybe rclone could even work it out itself...

7

Thanks for the explanation. So after re-checking (the code) I realized that for our use case a simple modification is enough. By not always trimming the root for a leading slash here (also removing the same redundant! line here) one can play with the path having a leading slash or not.
Now the check now makes sense and works.

Eg. Path with a leading slash:
:webdav,url="http://somewhere.org/api":/remote.php/webdav/folder1 will now test the returned items for starting with /remote.php/webdav/folder1

and path without a leading slash:
:webdav,url="http://somewhere.org/api":remote.php/webdav/folder1 will now test the returned items for starting with /api/remote.php/webdav/folder1

After doing this code modification I ran the code tests locally(!) and they appear to be fully in order (and real-world copying tests now also work as expected).

I'm curious to hear what you think, and possibly overlooked.

8

That is an interesting workaround.

Rclone users are fond of writing remote:/path/to/file even though everywhere the docs state use remote:path/to/file and I think your patch would break that.

I think we probably do need to keep trimming the slashes off the right hand side too as users quite often write remote:path/to/dir/

Here is the alternative approach with a base_path parameter which you'll need to set to /remote.php/webdav/home/folder1/ assuming you are using the url https://app.cs3mesh-iop.k8s.surfsara.nl/iop/webdav/home/folder1/

v1.59.0-beta.6181.dfc5b0460.fix-webdav-base-path on branch fix-webdav-base-path (uploaded in 15-30 mins)

That said, I'm not convinced that this is the right thing to do for your webdav server. It seems perfectly happy to respond without the /iop/ part, so have you tried url https://app.cs3mesh-iop.k8s.surfsara.nl/remote.php/webdav/ in the original rclone?

That might need to be https://app.cs3mesh-iop.k8s.surfsara.nl/remote.php/webdav/home/folder1 or similar, it depends on exactly what you are trying to achieve.

Any webdav server with remote.php/webdav in the URL is most likely based on the same PHP serving library as used by nextcloud and owncloud and their URLs look like https://example.com/remote.php/webdav/ so I would expect that form to work here too.

9

The issue there is that if the /iop part is missing the request doesn't get directed to the webdav service. However the underlying platform webservice is oblivious to this, and so it doesn't return that part. But the project (you are aware of it :slight_smile: ) is also under development and configurations change constantly. I do understand the confusion it can cause in webdav clients. Initially I thought the _root special value would help in a situation like this ...

So I think we should leave this for now, and try to solve it from our end.

--
ps. Here's what I experienced trying out you base_path parameter fix on our webservices:

That works for rclone lsd. But I do get some errors on folders when using ls and lsl.
This is the output for .../folder1 which contains the 2 files and a folder tmp:

     1143 test1.txt
       20 test2.txt
2022/06/09 09:09:23 ERROR : tmp/tmp: error listing: directory not found

I have yet to find out what happens here.

(Thanks for the support!)

10

Hi Nick, just a little clarification...

This is not quite right, the leading / is indeed needed in a few situations and recommended in the docs:

Note that some SFTP servers will need the leading / - Synology is a good example of this. rsync.net, on the other hand, requires users to OMIT the leading /.

I agree.

11

Ok!

Let me know if you want more help :slight_smile:

Hmm, not sure entirely what is going on there - would probable need access to the server to debug!

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.