Fail to download/list files serve webdav - Item with unknown path received

Help and Support
1

What is the problem you are having with rclone?

I'm using caddy to reverse proxy to rclone serve webdav. I'm able to UPLOAD files but not able to LIST and/or DOWNLOAD.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

Using caddy server to reverse proxy to rclone serve webdav

The command you were trying to run (eg rclone copy /tmp remote:tmp)

DOWNLOAD COMMAND
rclone.exe --config=rclone.conf --client-cert client.pem --client-key client.key -vv copy WEBDAV:download G:\download

LIST COMMAND
rclone.exe --config=rclone.conf --client-cert client.pem --client-key client.key -vv ls WEBDAV:

The rclone config contents with secrets removed.

[HTTP]
type = http
url = https://webportal.freeddns.org/http/

[WEBDAV]
type = webdav
url = https://webportal.freeddns.org/webdav/

A log from the command with the -vv flag

E:\ProgramData\rclone>rclone.exe --config=rclone.conf --client-cert client.pem --client-key client.key -vv ls WEBDAV:
2023/01/11 23:05:02 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone.exe" "--config=rclone.conf" "--client-cert" "client.pem" "--client-key" "client.key" "-vv" "ls" "WEBDAV:"]
2023/01/11 23:05:02 DEBUG : Creating backend with remote "WEBDAV:"
2023/01/11 23:05:02 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/11 23:05:02 DEBUG : found headers:
2023/01/11 23:05:02 DEBUG : Item with unknown path received: "/7000/", "/webdav/7000/"
2023/01/11 23:05:02 DEBUG : Item with unknown path received: "/7000/20230109093411_UploadTestFile.txt", "/webdav/7000/"
2023/01/11 23:05:02 DEBUG : Item with unknown path received: "/7000/20230111205158_UploadTestFile.txt", "/webdav/7000/"
2023/01/11 23:05:02 DEBUG : Item with unknown path received: "/7000/20230111215131_UploadTestFile.txt", "/webdav/7000/"
2023/01/11 23:05:02 DEBUG : Item with unknown path received: "/7000/download/", "/webdav/7000/"
2023/01/11 23:05:02 DEBUG : 3 go routines active

E:\ProgramData\rclone>rclone.exe --config=rclone.conf --client-cert client.pem --client-key client.key -vv copy WEBDAV:DOWNLOAD G:\DOWNLOAD
2023/01/11 23:21:22 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone.exe" "--config=rclone.conf" "--client-cert" "client.pem" "--client-key" "client.key" "-vv" "copy" "WEBDAV:DOWNLOAD" "G:\\DOWNLOAD"]
2023/01/11 23:21:22 DEBUG : Creating backend with remote "WEBDAV:DOWNLOAD"
2023/01/11 23:21:22 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/11 23:21:22 DEBUG : found headers:
2023/01/11 23:21:22 DEBUG : Creating backend with remote "G:\\DOWNLOAD"
2023/01/11 23:21:22 DEBUG : fs cache: renaming cache item "G:\\DOWNLOAD" to be canonical "//?/G:/DOWNLOAD"
2023/01/11 23:21:22 ERROR : webdav root 'DOWNLOAD': error reading source root directory: directory not found
2023/01/11 23:21:22 DEBUG : Local file system at //?/G:/DOWNLOAD: Waiting for checks to finish
2023/01/11 23:21:22 DEBUG : Local file system at //?/G:/DOWNLOAD: Waiting for transfers to finish
2023/01/11 23:21:22 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2023/01/11 23:21:22 ERROR : webdav root 'DOWNLOAD': error reading source root directory: directory not found
2023/01/11 23:21:22 DEBUG : Local file system at //?/G:/DOWNLOAD: Waiting for checks to finish
2023/01/11 23:21:22 DEBUG : Local file system at //?/G:/DOWNLOAD: Waiting for transfers to finish
2023/01/11 23:21:22 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2023/01/11 23:21:22 ERROR : webdav root 'DOWNLOAD': error reading source root directory: directory not found
2023/01/11 23:21:22 DEBUG : Local file system at //?/G:/DOWNLOAD: Waiting for checks to finish
2023/01/11 23:21:22 DEBUG : Local file system at //?/G:/DOWNLOAD: Waiting for transfers to finish
2023/01/11 23:21:22 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2023/01/11 23:21:22 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.3s

2023/01/11 23:21:22 DEBUG : 3 go routines active
2023/01/11 23:21:22 Failed to copy: directory not found
2

Hi Jay,

I cannot say exactly why you are getting this error, seems to be something like the webdav server returned this item URL "/7000/20230109093411_UploadTestFile.txt" and rclone expected this URL "/webdav/7000/20230109093411_UploadTestFile.txt".

I see you are developer with good knowledge of JavaScript, so perhaps you can find out more with this reference to the code returning the error:

3

@Ole Thank you for pointing that out. I did not understand the error.

I'm not sure if your familiar with CADDY (https://caddyserver.com/)
I'm utilizing CADDY to route traffic to different servers and use reverse proxy to handle where it would go.

main url
|-------- url/webdav/
|-------- url/filebrowser/
|-------- url/http/

this is from the caddyfile config

handle_path /webdav/* {
	reverse_proxy localhost:8888
}

But this is something I can test out from your comment/explanation

4

Missed this initially, please update to the latest version from here: https://rclone.org/downloads/

I know Caddy by concept, and that it serves rclone.org, but never tried it. The same goes for reverse proxy and webdav, but guess we can fix this with our combined knowledge and a little trial and error.

I am pretty sure this routing is our challenge, the good question is how to (best) make the webdav server handle the path prefix (/webdav/*) - or perhaps change your setup to be more like rclone.org having subdomains like: beta.rclone.org and download.rclone.org pointing to different servers (I assume).

Here are a few things I suggest you/we try:

What exactly is happening when you successfully upload a file? Which path does rclone see/use?

Enable logging on the webdav server something like this:

rclone server webdav -vv --log-file=serverlog.txt ...

then upload a very simple testfile:

echo "Hello World" > ./testfile.txt
rclone copy ./testfile.txt WEBDAV: -vv ...
rclone copyto ./testfile.txt WEBDAV:webdav/testfile2.txt -vv ...

Anything to be learned from the logs?
Where did the files end in the webdav server?

Next see if you can list any of the files:

rclone lsl WEBDAV: -vv

Then see if you can read any of the files:

rclone cat WEBDAV:testfile -vv ...
rclone cat WEBDAV:webdav/testfile -vv ...
rclone cat WEBDAV:testfile2 -vv ...
rclone cat WEBDAV:webdav/testfile2 -vv ...

Depending on the above, I would see if this is possible and perhaps does the trick (note the added path):

handle_path /webdav/* {
	reverse_proxy localhost:8888/webdav/
}
5

For the most part copy/upload commands all worked!. The cat/list commands all failed...
Here are the commands and output logs.

Enabled this on the server to get the logs generated server side.

E:\ProgramData\rclone>rclone --config=rclone.conf copy testfile.txt WEBDAV: -vv
2023/01/13 07:01:00 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "copy" "testfile.txt" "WEBDAV:" "-vv"]
2023/01/13 07:01:00 DEBUG : Creating backend with remote "testfile.txt"
2023/01/13 07:01:00 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:01:00 DEBUG : fs cache: adding new entry for parent of "testfile.txt", "//?/E:/ProgramData/rclone"
2023/01/13 07:01:00 DEBUG : Creating backend with remote "WEBDAV:"
2023/01/13 07:01:00 DEBUG : found headers:
2023/01/13 07:01:01 DEBUG : testfile.txt: Need to transfer - File not found at Destination
2023/01/13 07:01:01 INFO  : testfile.txt: Copied (new)
2023/01/13 07:01:01 INFO  :
Transferred:             16 B / 16 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2023/01/13 07:01:01 DEBUG : 4 go routines active

E:\ProgramData\rclone>rclone --config=rclone.conf copyto testfile.txt WEBDAV:webdav/testfile2.txt -vv
2023/01/13 07:02:11 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "copyto" "testfile.txt" "WEBDAV:webdav/testfile2.txt" "-vv"]
2023/01/13 07:02:11 DEBUG : Creating backend with remote "testfile.txt"
2023/01/13 07:02:11 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:02:11 DEBUG : fs cache: adding new entry for parent of "testfile.txt", "//?/E:/ProgramData/rclone"
2023/01/13 07:02:11 DEBUG : Creating backend with remote "WEBDAV:webdav/"
2023/01/13 07:02:11 DEBUG : found headers:
2023/01/13 07:02:11 DEBUG : fs cache: renaming cache item "WEBDAV:webdav/" to be canonical "WEBDAV:webdav"
2023/01/13 07:02:11 DEBUG : testfile.txt: Need to transfer - File not found at Destination
2023/01/13 07:02:11 INFO  : testfile.txt: Copied (new) to: testfile2.txt
2023/01/13 07:02:11 INFO  :
Transferred:             16 B / 16 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2023/01/13 07:02:11 DEBUG : 4 go routines active

Using the copy command, testfile.txt was created and successfully uploaded to the root of the fileserver

Using the copyto command, testfile.txt on the server was successfully copied to the sub directory called webdav as a new named testfile2.txt

The list command failed... see error output.

E:\ProgramData\rclone>rclone --config=rclone.conf lsl WEBDAV: -vv
2023/01/13 07:13:21 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "lsl" "WEBDAV:" "-vv"]
2023/01/13 07:13:21 DEBUG : Creating backend with remote "WEBDAV:"
2023/01/13 07:13:21 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:13:21 DEBUG : found headers:
2023/01/13 07:13:21 DEBUG : Item with unknown path received: "/", "/webdav/"
2023/01/13 07:13:21 DEBUG : Item with unknown path received: "/7000/", "/webdav/"
2023/01/13 07:13:21 DEBUG : Item with unknown path received: "/7001/", "/webdav/"
2023/01/13 07:13:21 DEBUG : Item with unknown path received: "/root.txt", "/webdav/"
2023/01/13 07:13:21 DEBUG : Item with unknown path received: "/testfile.txt", "/webdav/"
2023/01/13 07:13:21 DEBUG : 3 go routines active

E:\ProgramData\rclone>rclone --config=rclone.conf cat WEBDAV:testfile.txt -vv
2023/01/13 07:15:28 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "cat" "WEBDAV:testfile.txt" "-vv"]
2023/01/13 07:15:28 DEBUG : Creating backend with remote "WEBDAV:testfile.txt"
2023/01/13 07:15:28 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:15:28 DEBUG : found headers:
2023/01/13 07:15:28 DEBUG : fs cache: adding new entry for parent of "WEBDAV:testfile.txt", "WEBDAV:"
2023/01/13 07:15:28 DEBUG : Item with unknown path received: "/", "/webdav/"
2023/01/13 07:15:28 DEBUG : Item with unknown path received: "/7000/", "/webdav/"
2023/01/13 07:15:28 DEBUG : Item with unknown path received: "/7001/", "/webdav/"
2023/01/13 07:15:28 DEBUG : Item with unknown path received: "/root.txt", "/webdav/"
2023/01/13 07:15:28 DEBUG : Item with unknown path received: "/testfile.txt", "/webdav/"
2023/01/13 07:15:28 DEBUG : 3 go routines active

E:\ProgramData\rclone>rclone --config=rclone.conf cat WEBDAV:webdav/testfile.txt -vv
2023/01/13 07:16:21 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "cat" "WEBDAV:webdav/testfile.txt" "-vv"]
2023/01/13 07:16:21 DEBUG : Creating backend with remote "WEBDAV:webdav/testfile.txt"
2023/01/13 07:16:21 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:16:21 DEBUG : found headers:
2023/01/13 07:16:21 ERROR : : error listing: directory not found
2023/01/13 07:16:21 DEBUG : 3 go routines active
2023/01/13 07:16:21 Failed to cat with 2 errors: last error was: directory not found

E:\ProgramData\rclone>rclone --config=rclone.conf cat WEBDAV:testfile2.txt -vv
2023/01/13 07:16:53 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "cat" "WEBDAV:testfile2.txt" "-vv"]
2023/01/13 07:16:53 DEBUG : Creating backend with remote "WEBDAV:testfile2.txt"
2023/01/13 07:16:53 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:16:53 DEBUG : found headers:
2023/01/13 07:16:53 ERROR : : error listing: directory not found
2023/01/13 07:16:53 DEBUG : 3 go routines active
2023/01/13 07:16:53 Failed to cat with 2 errors: last error was: directory not found

E:\ProgramData\rclone>rclone --config=rclone.conf cat WEBDAV:webdav/testfile2.txt -vv
2023/01/13 07:17:15 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "cat" "WEBDAV:webdav/testfile2.txt" "-vv"]
2023/01/13 07:17:15 DEBUG : Creating backend with remote "WEBDAV:webdav/testfile2.txt"
2023/01/13 07:17:15 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:17:15 DEBUG : found headers:
2023/01/13 07:17:15 DEBUG : fs cache: adding new entry for parent of "WEBDAV:webdav/testfile2.txt", "WEBDAV:webdav"
2023/01/13 07:17:15 DEBUG : Item with unknown path received: "/webdav/", "/webdav/webdav/"
2023/01/13 07:17:15 DEBUG : Item with unknown path received: "/webdav/testfile2.txt", "/webdav/webdav/"
2023/01/13 07:17:15 DEBUG : 3 go routines active

Added the prefix: /webdav/ which resulted in error 502 bad gateway. I did try this one time and got the same results...

E:\ProgramData\rclone>rclone --config=rclone.conf lsl WEBDAV: -vv
2023/01/13 07:19:52 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "--config=rclone.conf" "lsl" "WEBDAV:" "-vv"]
2023/01/13 07:19:52 DEBUG : Creating backend with remote "WEBDAV:"
2023/01/13 07:19:52 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:19:52 DEBUG : found headers:
2023/01/13 07:19:52 DEBUG : pacer: low level retry 1/10 (error 502 Bad Gateway)
2023/01/13 07:19:52 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023/01/13 07:19:52 DEBUG : pacer: low level retry 2/10 (error 502 Bad Gateway)
2023/01/13 07:19:52 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2023/01/13 07:19:52 DEBUG : pacer: low level retry 3/10 (error 502 Bad Gateway)
2023/01/13 07:19:52 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2023/01/13 07:19:52 DEBUG : pacer: low level retry 4/10 (error 502 Bad Gateway)
2023/01/13 07:19:52 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2023/01/13 07:19:52 DEBUG : pacer: low level retry 5/10 (error 502 Bad Gateway)
2023/01/13 07:19:52 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2023/01/13 07:19:53 DEBUG : pacer: low level retry 6/10 (error 502 Bad Gateway)
2023/01/13 07:19:53 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2023/01/13 07:19:53 DEBUG : pacer: low level retry 7/10 (error 502 Bad Gateway)
2023/01/13 07:19:53 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2023/01/13 07:19:54 DEBUG : pacer: low level retry 8/10 (error 502 Bad Gateway)
2023/01/13 07:19:54 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/01/13 07:19:55 DEBUG : pacer: low level retry 9/10 (error 502 Bad Gateway)
2023/01/13 07:19:57 DEBUG : pacer: low level retry 10/10 (error 502 Bad Gateway)
2023/01/13 07:19:57 ERROR : : error listing: couldn't list files: 502 Bad Gateway
2023/01/13 07:19:57 DEBUG : 4 go routines active
2023/01/13 07:19:57 Failed to lsl with 2 errors: last error was: couldn't list files: 502 Bad Gateway

Here is the generated serverlog.txt.
serverlog.txt (12.7 KB)

6

I tried using the handle directive which retains the prefix instead of handle_path which removes it.

Caddy_Handle-Directives

handle /wedav/* {
	# The path still retains "/wedav"
        reverse_proxy localhost:8888
}

It seems like it cut everything out only listing the the files in root/webdav (Note: There were additional sub folders in the root directory)

E:\ProgramData\rclone>rclone.exe --config=rclone.conf -vv ls WEBDAV:
2023/01/13 07:51:26 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone.exe" "--config=rclone.conf" "-vv" "ls" "WEBDAV:"]
2023/01/13 07:51:26 DEBUG : Creating backend with remote "WEBDAV:"
2023/01/13 07:51:26 DEBUG : Using config file from "E:\\ProgramData\\rclone\\rclone.conf"
2023/01/13 07:51:26 DEBUG : found headers:
       16 testfile2.txt
2023/01/13 07:51:30 DEBUG : 3 go routines active

From what you stated earlier, its something with the return back response from the server...? thoughts?

7

Great, making progress, now we can see the files - just need to find out how to upload them using the new handle.

Now try these (adding --config etc as needed):

rclone copyto ./testfile.txt WEBDAV:testfile3
rclone copyto ./testfile.txt WEBDAV:webdav/testfile4
rclone ls WEBDAV: -vv

Not forgotten, we will either have to move these - or make a better handle - depending on the above.

8

After some more troubleshooting... This combination worked!

Caddy configuration:

handle /webdav/* {
	reverse_proxy localhost:8888
}

Rclone command on the server:

rclone serve webdav -vv --baseurl webdav --addr :8888 G:\webdav

The handle retains the webdav and adding the baseurl rclone serve to the root/webdav/ - like how you had in your recommendation. I appreciate all the help and debugging.

1 Like
9

You are welcome, happy you got it running with a nice clean setup!

I appreciate your patience with my lack of in-depth knowledge in this area. I learned a lot too.

1 Like
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.