Serve webdav Dir.ReadDirAll error

jautus · October 11, 2019, 4:38pm

What is the problem you are having with rclone?

2019/10/11 09:16:33 DEBUG : : Dir.ReadDirAll error: couldn't list files: Propfind /: unsupported protocol scheme ""
2019/10/11 09:16:33 http: superfluous response.WriteHeader call from github.com/rclone/rclone/vendor/golang.org/x/net/webdav.(*Handler).ServeHTTP (webdav.go:74)

What is your rclone version (output from `rclone version`)

rclone v1.49.5

os/arch: windows/amd64
go version: go1.12.10

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows Server 2012 R2 (Build 9600)

Which cloud storage system are you using? (eg Google Drive)

using rclone serve webdav as a server

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

On the server:
rclone serve webdav --addr :8081 -vv --dir-cache-time 5s --auth-proxy E:\ProgramData\rclone\auth-proxy\auth-proxy.exe

On the client:
rclone tree WebDAV: -vv --dump bodies

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

On the server:

2019/10/11 09:16:24 DEBUG : rclone: Version "v1.49.5" starting with parameters ["rclone" "serve" "webdav" "--addr" ":8081" "-vv" "--dir-cache-time" "5s" "--auth-proxy" "E:\ProgramData\rclone\auth-proxy\auth-proxy.exe"]
2019/10/11 09:16:24 NOTICE: WebDav Server started on http://[::]:8081/
2019/10/11 09:16:33 DEBUG : Calling proxy [E:\ProgramData\rclone\auth-proxy\auth-proxy.exe]
2019/10/11 09:16:33 DEBUG : Proxy returned in 326.5985ms
2019/10/11 09:16:33 INFO : webdav root '': poll-interval is not supported by this remote
2019/10/11 09:16:33 DEBUG : Adding path "vfs/forget" to remote control registry
2019/10/11 09:16:33 DEBUG : Adding path "vfs/refresh" to remote control registry

2019/10/11 09:16:33 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2019/10/11 09:16:33 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/11 09:16:33 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/11 09:16:33 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/11 09:16:33 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/11 09:16:33 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/11 09:16:33 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/11 09:16:33 DEBUG : : Dir.ReadDirAll error: couldn't list files: Propfind /: unsupported protocol scheme ""
2019/10/11 09:16:33 http: superfluous response.WriteHeader call from github.com/rclone/rclone/vendor/golang.org/x/net/webdav.(*Handler).ServeHTTP (webdav.go:74)

2019/10/11 09:16:33 INFO : /: PROPFIND from 10.10.40.112:50968

On the client:

2019/10/11 09:27:38 DEBUG : rclone: Version "v1.49.5" starting with parameters ["rclone" "tree" "WebDAV:" "-vv" "--dump" "bodies"]
2019/10/11 09:27:38 DEBUG : Using config file from "E:\ProgramData\rclone\rclone.config"
2019/10/11 09:27:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/10/11 09:27:38 DEBUG : HTTP REQUEST (req 0xc000108c00)
2019/10/11 09:27:38 DEBUG : PROPFIND / HTTP/1.1
Host: syncserver:8081
User-Agent: rclone/v1.49.5
Authorization: XXXX
Depth: 1
Accept-Encoding: gzip

2019/10/11 09:27:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/10/11 09:27:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/10/11 09:27:39 DEBUG : HTTP RESPONSE (req 0xc000108c00)
2019/10/11 09:27:39 DEBUG : HTTP/1.1 207 Multi-Status
Content-Length: 541
Content-Type: text/xml; charset=utf-8
Date: Fri, 11 Oct 2019 16:28:09 GMT

<?xml version="1.0" encoding="UTF-8"?>

<D:href>/</D:href><D:propstat><D:prop><D:resourcetype><D:collection xmlns:D="DAV
:"/></D:resourcetype><D:displayname></D:displayname><D:getlastmodified>Fri, 11 O
ct 2019 16:28:09 GMT</D:getlastmodified><D:supportedlock><D:lockentry xmlns:D="D
AV:"><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype

</D:lockentry></D:supportedlock></D:prop><D:status>HTTP/1.1 200 OK</D:status></
D:propstat></D:response></D:multistatus>Internal Server Error
2019/10/11 09:27:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/10/11 09:27:39 DEBUG : Stat: filePath="/"
2019/10/11 09:27:39 DEBUG : >Stat: fi=, err=
2019/10/11 09:27:39 DEBUG : ReadDir: dir=/
2019/10/11 09:27:39 DEBUG : >ReadDir: names=[], err=
/

0 directories, 0 files
2019/10/11 09:27:39 DEBUG : 5 go routines active
2019/10/11 09:27:39 DEBUG : rclone: Version "v1.49.5" finishing with parameters
["rclone" "tree" "WebDAV:" "-vv" "--dump" "bodies"]

Standard in/out of auth-proxy.exe:

Using the --auth-proxy with auth-proxy.exe program

In:
{ "pass": "mypassword", "user": "me"}

Out:
{"type":"webdav","_root":"","_obscure":"pass","user":"me","pass":"K6GysoBe1GDPeyT1mL_AmrBhzizxqVxSOO8\n","host":"localhost"}

ncw · October 11, 2019, 9:20pm

It looks like the output of the auth proxy output is missing the url parameter for configuring the WebDAV server.

Just out of interest what are you using the auth proxy for?

jautus · October 11, 2019, 10:30pm

Thanks for the quick reply.

I’m not familiar with the golang way of coding. With that said I wrote a C# console app that the auth-proxy calls then check LDAP to validate a user based on the username and password provided by the rclone client. I’m using the username as the certificate name and the password as the base64 public key to verify client connection for added security and restriction since enabling the SSL/TLS option any client with the trusted ca certificate can connect to the serve webdav server.

Let me know if i’m doing this wrong...

Also when you said the URL parameter you mean http://:?

ncw · October 12, 2019, 10:37am

Nice one - that is exactly what the AUTH proxy is for.

You are configuring a webdav remote in the output of the auth proxy but you haven't set the "url" parameter

https://rclone.org/webdav/#webdav-url

So you need to add

{
    "type":"webdav",
    "url": "https://wherever",
...

}

Note that host isn't a parameter for the webdav backend.

jautus · October 12, 2019, 12:12pm

Made the changes, had an issue with incorrect credentials by obscuring the password twice but got pass that.

Now I'm not sure what else I'm missing, rclone on the server is stuck at the waiting phase asking for response headers...

See below:

Server:

E:\ProgramData\rclone>rclone serve webdav --addr :8081 -vv --dir-cache-time 5s --auth-proxy E:\ProgramData\rclone\auth-proxy\auth-proxy.exe
2019/10/12 04:49:58 DEBUG : rclone: Version "v1.49.5" starting with parameters ["rclone" "serve" "webdav" "--addr" ":8081" "-vv" "--dir-cache-time" "5s" "--auth-proxy" "E:\ProgramData\rclone\auth-proxy\auth-proxy.exe"]
2019/10/12 04:49:58 NOTICE: WebDav Server started on http://[::]:8081/2019/10/12 04:50:06 DEBUG : Calling proxy [E:\ProgramData\rclone\auth-proxy\auth-proxy.exe]
2019/10/12 04:50:06 DEBUG : Proxy returned in 262.5529ms
2019/10/12 04:50:06 INFO : webdav root '': poll-interval is not supported by this remote
2019/10/12 04:50:06 DEBUG : Adding path "vfs/forget" to remote control registry
2019/10/12 04:50:06 DEBUG : Adding path "vfs/refresh" to remote control registry

2019/10/12 04:50:06 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2019/10/12 04:50:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:50:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:50:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:50:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:50:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:50:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:50:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:50:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:50:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:50:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:50:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:50:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:55:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:55:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:55:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:55:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:55:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:55:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:55:06 DEBUG : pacer: low level retry 1/10 (error Propfind http://syncserver:8081/: net/http: timeout awaiting response headers)
2019/10/12 04:55:06 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2019/10/12 04:55:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:55:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:55:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:55:06 DEBUG : /: >OpenFile: fd=/ (r), err=
2019/10/12 04:55:06 DEBUG : /: OpenFile: flags=O_RDONLY, perm=----------
2019/10/12 04:55:06 DEBUG : /: >OpenFile: fd=/ (r), err=

Client:

E:\ProgramData\rclone>rclone tree WebDAV: -vv --dump bodies
2019/10/12 04:49:35 DEBUG : rclone: Version "v1.49.5" starting with parameters ["rclone" "tree" "WebDAV:" "-vv" "--dump" "bodies"]
2019/10/12 04:49:35 DEBUG : Using config file from "E:\ProgramData\rclone\rclone.config"
2019/10/12 04:49:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>
2019/10/12 04:49:35 DEBUG : HTTP REQUEST (req 0xc000108000)
2019/10/12 04:49:35 DEBUG : PROPFIND / HTTP/1.1
Host: syncserver:8081
User-Agent: rclone/v1.49.5
Authorization: XXXX
Depth: 1
Accept-Encoding: gzip

2019/10/12 04:49:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>
2019/10/12 04:54:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<
2019/10/12 04:54:35 DEBUG : HTTP RESPONSE (req 0xc000108000)
2019/10/12 04:54:35 DEBUG : Error: net/http: timeout awaiting response headers
2019/10/12 04:54:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<
2019/10/12 04:54:35 DEBUG : pacer: low level retry 1/10 (error Propfind http://syncserver:8081/: net/http: timeout awaiting response headers)
2019/10/12 04:54:35 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2019/10/12 04:54:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>
2019/10/12 04:54:35 DEBUG : HTTP REQUEST (req 0xc000164a00)
2019/10/12 04:54:35 DEBUG : PROPFIND / HTTP/1.1
Host: syncserver:8081
User-Agent: rclone/v1.49.5
Authorization: XXXX
Depth: 1
Accept-Encoding: gzip

2019/10/12 04:54:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>

stdout:

{"type":"webdav","_root":"","_obscure":"pass","user":"me","pass":"mypassword","url":"http://syncserver:8081"}

Question: On the server where or how do I specify the drive or UNC path to where the WebDAV folder root is at? I tried changing the root in the stdout from "" to "G:\Sync" rclone just kept calling Proxy returned in xx.xxms Calling proxy [program.exe]

Last Question: Does the backend server dies/terminates after the transaction is complete?

ncw · October 12, 2019, 2:24pm

I think this explains all the problems... You don't want to setup a WebDAV backend, you want a local backend. So set type = local and _root = path/to/folder you want to serve in the output of your auth proxy.

It will be cached for 60 seconds after the last connect then be destroyed.

jautus · October 13, 2019, 4:06pm

Looks like that was it. You got me working. Thank you.

system · January 11, 2020, 4:13pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.