Rclone webdav nextcloud can't see files

Help and Support
1

Hello rclone community,

I’ve been trying the latest beta ( 1.38-236 as of writing ), and I’m very interested in webdav ( Nextcloud ) support.
I have 3 Nextcloud servers available to me, and none of them show files with the lsd, ls, or mount commands. I’m able to upload fine, which is great, however, I can’t seem to view/download ( When I mount, nothing shows up until I upload something, then only items uploaded during this current mount show up. ) I haven’t tried other commands, as something is clearly not right, and I don’t want to cause problems to the servers.

With dav2fs, I’m able to mount, view, edit and browse everything fine, so my first guess was that it’s an rclone bug. However, I didn’t want to post a bug report, because it could be a configuration problem ( weird that it’s on all 3 servers with different setups though ).

What I did notice, is that if I run “rclone --max-depth=1 ls -vv xxxx:”. I get the files listed with messages like this:
2017/12/17 20:46:27 DEBUG : test.img: Ignoring item with bad status "HTTP/1.1 404 Not Found"

When I use other webdav clients, I see no such error however. Is this a configuration issue in Nextcloud ( In which case any help fixing it would be greatly appreciated, I can provide more details as needed. ), or is this a possible issue with rclone?

Thanks for the help, and for rclone!
jedi453

P.S. The first Nextcloud server was set up as an OwnCloud server a few years ago by someone else, who then upgraded it to Nextcloud. It is running on an x86 virtual machine. Servers 2 and 3 are on ARM SBCs, one ARMv7 (Debian Jessie, Apache2, php5) one ARMv8 ( Ubuntu 16.04, nginx, php-fpm7.1 ). So I’ve tried a few different setups, which makes me think it could be an rclone bug ( combined with the fact that dav2fs shows everything fine ). All of them are using SSL ( HTTPS ). Thinking about it, is it possible rclone connects initially with https, then tries to read files over http? It’s possible port 80 is blocked, or http not allowed by the web server… Thanks again.

1 Like
2

I came here looking for a similar answer to my issues with rclone and nextcloud… I am having the exact same issue…

While I can mount it from davfs and other clients work fine… rclone gives me " Ignoring item with bad status “HTTP/1.1 404 Not Found” for every item on the remote. I can tell it is seeing them at least, but something isn’t quite correct.

edit: Moving to the latest and greatest nginx stable and double checking that the dav_ext module was loaded/configured correctly didn’t help the problem either.

3

That explains the problem. Webdav has a status for each file which is normally “HTTP/1.1 200 OK”. So it is telling rclone that these files don’t actually exists…

Do you have any idea why the files might have that status?

I run rclone integration tests against both NextCloud and OwnCloud daily so there really shouldn’t be a problem with it!

You can use -vv --dump-bodies to investigate further.

4

Hello,

Thanks for the reply. It’s possible this is an error in my configuration, but it is happening on 3 different Nextcloud Servers, each with different configurations and setups.

Looking at what --dump-bodies gives me, I see that “HTTP/1.1 200 OK” appears to be sent as well, which is very odd… What do you make of this?
Here’s the snippets that I think are important (It was one line, but I’ve added some line breaks for formatting):

<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
<d:response><d:href>/remote.php/webdav/test-rclone/</d:href>
<d:propstat><d:prop><d:getlastmodified>Mon, 18 Dec 2017 22:29:18 GMT</d:getlastmodified>
<d:resourcetype><d:collection/></d:resourcetype>
<d:quota-used-bytes>5</d:quota-used-bytes>
<d:quota-available-bytes>-3</d:quota-available-bytes>
<d:getetag>&quot;5a38413e1a24a&quot;</d:getetag>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat></d:response>
<d:response><d:href>/remote.php/webdav/test-rclone/test1</d:href>
<d:propstat><d:prop><d:getlastmodified>Mon, 18 Dec 2017 22:26:10 GMT</d:getlastmodified>
<d:getcontentlength>0</d:getcontentlength>
<d:resourcetype/><d:getetag>&quot;6779e25a95f2fbcce7aaba2aa51ceafe&quot;</d:getetag>
<d:getcontenttype>application/octet-stream</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat>
<d:propstat>
<d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status></d:propstat>
</d:response>
<d:response><d:href>/remote.php/webdav/test-rclone/test2.txt</d:href>
<d:propstat><d:prop><d:getlastmodified>Mon, 18 Dec 2017 22:26:24 GMT</d:getlastmodified><d:getcontentlength>5</d:getcontentlength>
<d:resourcetype/><d:getetag>&quot;1a37b4f3b1f50122d7e8d391d6ad3b04&quot;</d:getetag><d:getcontenttype>text/plain</d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
<d:propstat><d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
</d:multistatus>

FYI: For testing, within the root of the NextCloud share, I’ve created a directory “test-rclone” with files “test1” and “test2.txt”.
Yet later, I get again:

2017/12/18 17:36:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/12/18 17:36:57 DEBUG : test1: Ignoring item with bad status "HTTP/1.1 404 Not Found"
2017/12/18 17:36:57 DEBUG : test2.txt: Ignoring item with bad status "HTTP/1.1 404 Not Found"
2017/12/18 17:36:57 DEBUG : Go routines at exit 4

Thanks again for your reply, and even more so for rclone!
jedi453

5

I ran that through xmllint --format to get

<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
  <d:response>
    <d:href>/remote.php/webdav/test-rclone/</d:href>
    <d:propstat>
      <d:prop>
        <d:getlastmodified>Mon, 18 Dec 2017 22:29:18 GMT</d:getlastmodified>
        <d:resourcetype>
          <d:collection/>
        </d:resourcetype>
        <d:quota-used-bytes>5</d:quota-used-bytes>
        <d:quota-available-bytes>-3</d:quota-available-bytes>
        <d:getetag>"5a38413e1a24a"</d:getetag>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
  </d:response>
  <d:response>
    <d:href>/remote.php/webdav/test-rclone/test1</d:href>
    <d:propstat>
      <d:prop>
        <d:getlastmodified>Mon, 18 Dec 2017 22:26:10 GMT</d:getlastmodified>
        <d:getcontentlength>0</d:getcontentlength>
        <d:resourcetype/>
        <d:getetag>"6779e25a95f2fbcce7aaba2aa51ceafe"</d:getetag>
        <d:getcontenttype>application/octet-stream</d:getcontenttype>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
      <d:prop>
        <d:quota-used-bytes/>
        <d:quota-available-bytes/>
      </d:prop>
      <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
  </d:response>
  <d:response>
    <d:href>/remote.php/webdav/test-rclone/test2.txt</d:href>
    <d:propstat>
      <d:prop>
        <d:getlastmodified>Mon, 18 Dec 2017 22:26:24 GMT</d:getlastmodified>
        <d:getcontentlength>5</d:getcontentlength>
        <d:resourcetype/>
        <d:getetag>"1a37b4f3b1f50122d7e8d391d6ad3b04"</d:getetag>
        <d:getcontenttype>text/plain</d:getcontenttype>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
      <d:prop>
        <d:quota-used-bytes/>
        <d:quota-available-bytes/>
      </d:prop>
      <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
  </d:response>
</d:multistatus>

So it appears from that the file is found, but the quota for it isn’t, and somehow rclone is getting them mixed up.

Is there any chance you could give me a login to your nextcloud temporarily so I can test stuff?

6

If so can you email details to nick@craig-wood.com please? Thanks!

7

I sent you an account to mine… hopefully it gets you something…

8

Hello @ncw

I just sent you credentials to a test Nextcloud server I set up with the same problem.

Interestingly ( just for the sake of eliminating differences ) I set it up with Nextcloud 11.0.4, and rclone connected and saw everything, no issue. Then I did a manual upgrade to Nextcloud 12.0.4, and the problem re-appeared. Is it possible that this problem is specific to 12.0.4, or the manual upgrade somehow triggers it?

@tonyroomz Are you on 12.0.4, or have you performed an upgrade on your server?

Thanks again,
jedi453

9

I am on 12.0.4 as well… I also upgraded manually…

10

Thank you both for sending me credentials to test.

I’ve managed to fix this - see this beta

https://beta.rclone.org/v1.38-241-g4ba58884/ (uploaded in 15-30 mins)

I also ran the unit tests, which indicate that nextcloud is ignoring rclone’s Range requests now. This means, for instance, that seeking files on an rclone mount won’t work.

I’m pretty sure this is a nextcloud bug - I can see rclone puts the range requests in the request, but nextcloud just ignores them

2017/12/20 11:48:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/12/20 11:48:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/12/20 11:48:58 DEBUG : HTTP REQUEST (req 0xc4204c2000)
2017/12/20 11:48:58 DEBUG : GET /remote.php/webdav/rclone-test-lowuhoz8kabewem4vojoxep3/file%20name.txt HTTP/1.1
Host: test.nsupdate.info:8443
User-Agent: rclone/v1.38-DEV
Authorization: XXXX
Range: bytes=50-

2017/12/20 11:48:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/12/20 11:48:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/12/20 11:48:59 DEBUG : HTTP RESPONSE (req 0xc4204c2000)
2017/12/20 11:48:59 DEBUG : HTTP/1.1 200 OK
Content-Length: 100
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Content-Disposition: attachment; filename*=UTF-8''file%20name.txt; filename="file%20name.txt"
Content-Security-Policy: default-src 'none';
Content-Type: text/plain;charset=UTF-8
Date: Wed, 20 Dec 2017 11:48:59 GMT
Etag: "817461593e1df48f0f723648660aa7b4"
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Last-Modified: Sat, 03 Feb 2001 04:05:06 GMT
Oc-Etag: "817461593e1df48f0f723648660aa7b4"
Pragma: no-cache
Referrer-Policy: same-origin
Server: nginx
Set-Cookie: X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-Xss-Protection: 1; mode=block

suxeqiv7xoyuvol7yiyupap1vuhokut4cezidos9sukelay2yileqob4kecucas7cevayux5toxokur1xapopam5woduteg7raci
2017/12/20 11:48:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
--- FAIL: TestObjectOpenSeek (0.88s)
	Error Trace:	fstests.go:805
	Error:      	Not equal: 
	            	expected: "leqob4kecucas7cevayux5toxokur1xapopam5woduteg7raci"
	            	actual: "suxeqiv7xoyuvol7yiyupap1vuhokut4cezidos9sukelay2yileqob4kecucas7cevayux5toxokur1xapopam5woduteg7raci"
	Messages:   	contents of file1 differ after seek

It would be quite easy to make a repro using curl if anyone fancies reporting this?

1 Like