WebDAV COPY request implemented incorrectly

Suspected Bug
1

What is the problem you are having with rclone?

I'm doing copy between two WebDAV endpoints. When issuing the copy call, rclone may try to use the WebDAV COPY method. The COPY request should go to the source endpoint as per current WebDAV specification. Rclone however sends the request to the destination. This is incorrect.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.0-beta.6508.f4a571786

  • os/version: centos 8.5.2111 (64 bit)
  • os/kernel: 5.10.102.1-microsoft-standard-WSL2 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.2
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

webdav

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Run rclone with server-side-accross-configs flag so it will try to do a COPY method request:

/root/go/bin/rclone -vv rcd --server-side-across-configs=true (... other flags ...)

Issue a folder copy call (I'm using the string connection syntax):

curl  -H "Content-Type: application/json"
      -X POST
      -d '{"srcFs":":webdav,headers=\"...eg. auth...\",url=\"http://source-endpoint/\":remote.php/webdav/home/mydata", "dstFs":":webdav,headers=\"...eg. auth...\",url=\"http://dest-endpoint/\":remote.php/webdav/home/DataTransfers/mydata", "_async":"true"}'
     http://rclone-server:5572/sync/copy

In the source folder there are files to be found. Rclone discovers them correctly. And also the folder tree structure is recreated correctly at the destination. However for each file rclone will do a COPY method request to the destination instead of the resource:

COPY http://dest-endpoint/remote.php/webdav/home/DataTransfers/mydata/test1.txt
     Destination: http://dest-endpoint/remote.php/webdav/home/DataTransfers/mydata/test1.txt
     Overwrite: F
     ...

So, Destination is correctly set but the COPY request should be directed to http://source-endpoint/remote.php/webdav/home/mydata/test1.txt

When I 'hack' the code in the webdav copy implementation such that this is corrected everything works as expected.
I'm unfortunately unable to discover the proper fix here.

I hope someone with more in-depth rclone code knowledge can have a look at this.

The rclone config contents with secrets removed.

I'm using the string connection syntax.

A log from the command with the -vv flag

See explanation above.

2

Hmm, interesting.

Have you found some docs to support that the COPY should be sent to the source not the destination?

In the rfc I think it assumes that all copying is on the same server.

Note that --server-side-across-configs=true is a relatively new feature so this is why this hasn't been noticed before.

This should be quite easy to fix, but I'd really like to see some docs if possible. I guess it probably applies to MOVE also?

3

I'm assuming that you are talking about the examples? They use the same server indeed.

But true, the rfc is not very explicit about this, '... implemented incorrectly' may be a bit harsh.

However:
The rfc basically states that source and destination resources are URIs. So that does not rule out different servers.
And further, Status code 502 (Bad Gateway) in the rfc does actually mention the possibility of the destination being on another server.
So in case of different servers that means you would automatically do the request on the source server (or at least the server who has access to the source resource), otherwise having a Destination header would not make sense.
And implementing the WebDAV COPY request this way still supports same server copy.

Have you found some docs to support that the COPY should be sent to the source not the destination?

Projects/servers I know of that interpret WebDAV COPY this way are dCache and Rucio which basically follows HttpTpcTechnical < LCG < TWiki. These servers partly also extend WebDAV COPY from the rfc but don't go against it.

Yes, I would think so. COPY plus delete source resource right?

4

Thank you for the explanations.

I've attempted to fix this - can you give this a go? This passes all the tests with a single server just fine.

v1.61.0-beta.6566.2c78f56d4 on branch master (uploaded in 15-30 mins)

5

Great! My first tests between two different servers look good. Will double check tomorrow if that is ok. Big thanks!!

6

Excellent.

Great - more checks is better!

Note to see http headers use -vv --dump headers which may help you see what is going on.

7

Tested again, everything seems ok. Looks good to go!

8

Excellent! Thank you for testing.

I accidentally merged it to master already so it is in the beta now and will be released in v1.61

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.