Unable to rename and move directories via WebDAV

cristianlivella · February 16, 2021, 8:35pm

What is the problem you are having with rclone?

When I try to rename or move a directory in a WebDAV remote I get error 409 Conflict.

I'm using the WebDAV server created by MEGAcmd, however if I connect directly to it using Nautilus I can rename and move directories without problem.

I tried both moving and renaming in the mounted remote, and directly with rclone move.

What is your rclone version (output from `rclone version`)

rclone v1.54.0
os/arch: linux/amd64
go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Fedora 33, 64 bit

Which cloud storage system are you using? (eg Google Drive)

WebDAV (MEGAcmd)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount mega:/ /mnt/mega --vfs-cache-mode full -vv

rclone move mega:/dir1 mega:/dir1-new -vv

The rclone config contents with secrets removed.

[mega]
type = webdav
url = http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive
vendor = other

A log from the command with the `-vv` flag

# rclone mount, trying to move
2021/02/16 21:11:44 DEBUG : dir1/: Attr: 
2021/02/16 21:11:44 DEBUG : dir1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/16 21:11:44 DEBUG : : Statfs: 
2021/02/16 21:11:44 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/02/16 21:11:44 DEBUG : /: Attr: 
2021/02/16 21:11:44 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/16 21:11:44 DEBUG : dir2/: Attr: 
2021/02/16 21:11:44 DEBUG : dir2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/16 21:11:44 DEBUG : dir1/: Lookup: name="dir2"
2021/02/16 21:11:44 DEBUG : dir1/: >Lookup: node=<nil>, err=no such file or directory
2021/02/16 21:11:44 DEBUG : dir1/: Lookup: name="dir2"
2021/02/16 21:11:44 DEBUG : dir1/: >Lookup: node=<nil>, err=no such file or directory
2021/02/16 21:11:44 DEBUG : /: Rename: oldName="dir2", newName="dir2", newDir=dir1/
2021/02/16 21:11:44 ERROR : dir2: Dir.Rename error: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:11:44 DEBUG : /: >Rename: err=DirMove MOVE call failed: 409 Conflict

# rclone mount, trying to rename
2021/02/16 21:21:47 DEBUG : : Statfs: 
2021/02/16 21:21:47 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/02/16 21:21:47 DEBUG : /: Lookup: name="dir1-new"
2021/02/16 21:21:47 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/02/16 21:21:47 DEBUG : /: Lookup: name="dir1"
2021/02/16 21:21:47 DEBUG : /: >Lookup: node=dir1/, err=<nil>
2021/02/16 21:21:47 DEBUG : dir1/: Attr: 
2021/02/16 21:21:47 DEBUG : dir1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/16 21:21:47 DEBUG : /: Lookup: name="dir1-new"
2021/02/16 21:21:47 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/02/16 21:21:47 DEBUG : /: Rename: oldName="dir1", newName="dir1-new", newDir=/
2021/02/16 21:21:47 ERROR : dir1: Dir.Rename error: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:21:47 DEBUG : /: >Rename: err=DirMove MOVE call failed: 409 Conflict

# rclone move mega:/dir1 mega:/dir1-new -vv
2021/02/16 21:24:28 DEBUG : Using config file from "xxx/rclone.conf"
2021/02/16 21:24:28 DEBUG : Creating backend with remote "mega:/dir1"
2021/02/16 21:24:28 DEBUG : fs cache: renaming cache item "mega:/dir1" to be canonical "mega:dir1"
2021/02/16 21:24:28 DEBUG : Creating backend with remote "mega:/dir1-new"
2021/02/16 21:24:29 DEBUG : fs cache: renaming cache item "mega:/dir1-new" to be canonical "mega:dir1-new"
2021/02/16 21:24:29 DEBUG : webdav root 'dir1-new': Using server-side directory move
2021/02/16 21:24:29 ERROR : webdav root 'dir1-new': Server side directory move failed: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:24:29 ERROR : Attempt 1/3 failed with 1 errors and: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:24:29 DEBUG : webdav root 'dir1-new': Using server-side directory move
2021/02/16 21:24:29 ERROR : webdav root 'dir1-new': Server side directory move failed: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:24:29 ERROR : Attempt 2/3 failed with 1 errors and: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:24:29 DEBUG : webdav root 'dir1-new': Using server-side directory move
2021/02/16 21:24:29 ERROR : webdav root 'dir1-new': Server side directory move failed: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:24:29 ERROR : Attempt 3/3 failed with 1 errors and: DirMove MOVE call failed: 409 Conflict
2021/02/16 21:24:29 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         1.3s

ncw · February 17, 2021, 1:15pm

Can you do this one with a bit more logging, so

rclone move mega:/dir1 mega:/dir1-new -vv --dump bodies --retries 1 --low-level-retries 1

I suspect this is webdav being non standard but the above trace should show.

Thanks

cristianlivella · February 17, 2021, 2:23pm

Actually I think I have misunderstood how the move command works. I thought that with that command I could rename dir1 in dir1-new, instead I seem to understand that it only moves the content, and this seems to work correctly. The error was probably because the dir1-new folder didn't exist, I believe this is a desired behavior.

I post instead the result of this command, while I'm trying to move dir1 into dir2.

rclone mount mega:/ /mnt/mega --vfs-cache-mode full -vv --dump bodies --retries 1 --low-level-retries 1

2021/02/17 15:21:03 DEBUG : /: Lookup: name="dir2"
2021/02/17 15:21:03 DEBUG : /: >Lookup: node=dir2/, err=<nil>
2021/02/17 15:21:03 DEBUG : dir2/: Attr: 
2021/02/17 15:21:03 DEBUG : dir2/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/17 15:21:03 DEBUG : : Statfs: 
2021/02/17 15:21:03 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/02/17 15:21:03 DEBUG : /: Attr: 
2021/02/17 15:21:03 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/17 15:21:03 DEBUG : /: Lookup: name="dir1"
2021/02/17 15:21:03 DEBUG : /: >Lookup: node=dir1/, err=<nil>
2021/02/17 15:21:03 DEBUG : dir1/: Attr: 
2021/02/17 15:21:03 DEBUG : dir1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/02/17 15:21:03 DEBUG : dir2/: Lookup: name="dir1"
2021/02/17 15:21:03 DEBUG : dir2/: >Lookup: node=<nil>, err=no such file or directory
2021/02/17 15:21:03 DEBUG : dir2/: Lookup: name="dir1"
2021/02/17 15:21:03 DEBUG : dir2/: >Lookup: node=<nil>, err=no such file or directory
2021/02/17 15:21:03 DEBUG : /: Rename: oldName="dir1", newName="dir1", newDir=dir2/
2021/02/17 15:21:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/17 15:21:03 DEBUG : HTTP REQUEST (req 0xc0003b2b00)
2021/02/17 15:21:03 DEBUG : PROPFIND /XXXXXXXX/Cloud%20Drive/dir2/dir1/ HTTP/1.1
Host: 127.0.0.1:4443
User-Agent: rclone/v1.54.0
Depth: 1
Referer: http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive/
Accept-Encoding: gzip

2021/02/17 15:21:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/17 15:21:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/17 15:21:03 DEBUG : HTTP RESPONSE (req 0xc0003b2b00)
2021/02/17 15:21:03 DEBUG : HTTP/1.1 404 Not Found
Connection: close

2021/02/17 15:21:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/17 15:21:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/17 15:21:03 DEBUG : HTTP REQUEST (req 0xc0000edb00)
2021/02/17 15:21:03 DEBUG : MKCOL /XXXXXXXX/Cloud%20Drive/dir2/ HTTP/1.1
Host: 127.0.0.1:4443
User-Agent: rclone/v1.54.0
Referer: http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive/
Accept-Encoding: gzip

2021/02/17 15:21:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/17 15:21:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/17 15:21:03 DEBUG : HTTP RESPONSE (req 0xc0000edb00)
2021/02/17 15:21:03 DEBUG : HTTP/1.1 405 Unknown Error
Connection: close

2021/02/17 15:21:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/17 15:21:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/17 15:21:03 DEBUG : HTTP REQUEST (req 0xc000314100)
2021/02/17 15:21:03 DEBUG : MOVE /XXXXXXXX/Cloud%20Drive/dir1/ HTTP/1.1
Host: 127.0.0.1:4443
User-Agent: rclone/v1.54.0
Destination: http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive/dir2/dir1/
Overwrite: F
Referer: http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive/
Accept-Encoding: gzip

2021/02/17 15:21:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/17 15:21:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/17 15:21:03 DEBUG : HTTP RESPONSE (req 0xc000314100)
2021/02/17 15:21:03 DEBUG : HTTP/1.1 409 Conflict
Connection: close

2021/02/17 15:21:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/17 15:21:03 ERROR : dir1: Dir.Rename error: DirMove MOVE call failed: 409 Conflict
2021/02/17 15:21:03 DEBUG : /: >Rename: err=DirMove MOVE call failed: 409 Conflict

edit: I'm doing some test, and the problem seems to be the trailing slash at the end of the Destination header.
I tried to make the request to the WebDAV server by hand, and I see that it moves correctly the directory if the destination is like this:

http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive/dir2/dir1

but it won't work and give error 409 Conflict if the destination is like this:

http://127.0.0.1:4443/XXXXXXXX/Cloud%20Drive/dir2/dir1/

I'm not too familiar with WebDAV, so I'm not sure about what should be the correct behaviour, however I see that if I connect to WebDAV using Gnome Nautilus, it doesn't put the trailing slash so I can move and rename directories without problem.

ncw · February 17, 2021, 5:41pm

Well spotted.

My understanding of the WebDAV RFC is that directories (which are known as collections in WebDAV speak) should always have a trailing /...

The example in the RFC has trailing /

You could try changing rclone like this

diff --git a/backend/webdav/webdav.go b/backend/webdav/webdav.go
index e441b06a5..a055a1079 100644
--- a/backend/webdav/webdav.go
+++ b/backend/webdav/webdav.go
@@ -963,7 +963,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
 		Path:       addSlash(srcPath),
 		NoResponse: true,
 		ExtraHeaders: map[string]string{
-			"Destination": addSlash(destinationURL.String()),
+			"Destination": destinationURL.String(),
 			"Overwrite":   "F",
 		},
 	}

I don't think that is correct in the general case (as in the code works with all the other webdav servers rclone supports) but it could be a quirk we add...
And see if that fixes the problem

cristianlivella · February 17, 2021, 8:43pm

This way it works perfectly, thank you!

I took a look at the RFC too, and the correct forms seems to be the one with the trailing slash. I opened an issue in MEGAcmd.

github.com/meganz/MEGAcmd

WebDAV directories move doesn't work with trailing slash in destination

opened 08:42PM - 17 Feb 21 UTC

cristianlivella

I'm trying to use the WebDAV server with [rclone](https://github.com/rclone/rclo…ne) and I noticed that I can't move and rename directories. I observed the packets being exchanged with the WebDAV server, and I noticed that the problem is caused by the trailing slash that rclone put at the end of the destination header. This request doesn't work and return 409 Conflict: ```http MOVE http://127.0.0.1:4443/xxxxxxx/mega/dir/ HTTP/1.1 Destination: http://127.0.0.1:4443/xxxxxxx/mega/new-dir/ Overwrite: F ``` This request instead works as expected: ```http MOVE http://127.0.0.1:4443/xxxxxxx/mega/dir/ HTTP/1.1 Destination: http://127.0.0.1:4443/xxxxxxx/mega/new-dir Overwrite: F ``` According to the [RFC 4918](http://www.webdav.org/specs/rfc4918.html), the correct form seems to be the one with the trailing slash, which is currently not handled properly by MEGAcmd. > There is a standing convention that when a collection is referred to by its name without a trailing slash, the server may handle the request as if the trailing slash were present. [...] In general, clients should use the trailing slash form of collection names. Furthermore, the example of moving a folder in the RFC also has the trailing slash: ```http MOVE /container/ HTTP/1.1 Host: www.example.com Destination: http://www.example.com/othercontainer/ Overwrite: F ``` References: - [http://www.webdav.org/specs/rfc4918.html#rfc.section.5.2.p.8](http://www.webdav.org/specs/rfc4918.html#rfc.section.5.2.p.8) - [http://www.webdav.org/specs/rfc4918.html#rfc.section.9.9.6](http://www.webdav.org/specs/rfc4918.html#rfc.section.9.9.6)

ncw · February 17, 2021, 8:45pm

Well done. See if you get any joy from them fixing the problem. If not we can add a work-around in rclone. We'd make a webdav provider "megacmd" and set a "quirk" that it shouldn't have trailing / in MOVE.

system · February 20, 2021, 8:45pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.