I can upload a directory of files via WebDAV to the CyVerse Data Store, but if I repeat the operation with the exact same command, rclone copies all the files again. It appears to fail in checking whether the file already exists. The log includes lines like: 2023/01/28 02:12:18 ERROR : /DJI_0394.JPG: Entry doesn't belong in directory "" (contains subdir) - ignoring
which looks similar to a previous bug with s3 storage: Entry doesn't belong in directory.. errors when using rclone sync with s3 · Issue #1621 · rclone/rclone · GitHub
When the folder contains files and subfolders (like the example I report below), the same Entry doesn't belong in directory "" error is also raised for all the subdirectories in addition to the files.
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1
os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-58-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.19.4
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
WebDAV
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Yes, here's the log for a transfer of a directory with a single file in it, where the remote already has that exact data so the file upload should be skipped but isn't.
Note: when testing this, I first tried to transfer a single file directly (not a directory with a single file) and rclone did see that the file existed (with same file size) and skipped it. But it does not complete this check when uploading a directory with a single file.
Thanks for looking at this!
2023/01/28 17:39:16 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "-vv" "/ofo-share/temp/one-photo/" "wd:dav/iplant/home/djyoung/backups/test-rclone-wd16" "--progress" "--log-file" "rclone-wd16.txt" "--dump" "responses"]
2023/01/28 17:39:16 DEBUG : Creating backend with remote "/ofo-share/temp/one-photo/"
2023/01/28 17:39:16 DEBUG : Using config file from "/home/derek/.config/rclone/rclone.conf"
2023/01/28 17:39:16 DEBUG : fs cache: renaming cache item "/ofo-share/temp/one-photo/" to be canonical "/ofo-share/temp/one-photo"
2023/01/28 17:39:16 DEBUG : Creating backend with remote "wd:dav/iplant/home/djyoung/backups/test-rclone-wd16"
2023/01/28 17:39:16 DEBUG : found headers:
2023/01/28 17:39:16 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/01/28 17:39:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:16 DEBUG : HTTP REQUEST (req 0xc0007d1500)
2023/01/28 17:39:16 DEBUG : PROPFIND /dav/iplant/home/djyoung/backups/test-rclone-wd16 HTTP/1.1
Host: data.cyverse.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Depth: 1
Referer: https://data.cyverse.org/
Accept-Encoding: gzip
2023/01/28 17:39:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:17 DEBUG : HTTP RESPONSE (req 0xc0007d1500)
2023/01/28 17:39:17 DEBUG : HTTP/1.1 207 Multi-Status
Content-Length: 1695
Access-Control-Allow-Headers: Range
Access-Control-Allow-Methods: GET, HEAD, OPTIONS
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public
Content-Type: text/xml; charset="utf-8";charset=utf-8
Date: Sat, 28 Jan 2023 17:39:16 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
Vary: Authorization
Via: 1.1 varnish (Varnish/6.5)
X-Cache: MISS
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:">
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/iplant/home/djyoung/backups/test-rclone-wd16</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Sat, 28 Jan 2023 17:38:48 GMT</D:creationdate>
<D:getetag>"01674927528"</D:getetag>
<D:getlastmodified>Sat, 28 Jan 2023 17:38:48 GMT</D:getlastmodified>
<lp1:resourcetype><D:collection/></lp1:resourcetype>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
<D:getcontenttype>application/octet-stream</D:getcontenttype>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/iplant/home/djyoung/backups/test-rclone-wd16/DJI_0004.JPG</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Sat, 28 Jan 2023 17:38:49 GMT</D:creationdate>
<D:getcontentlength>8658483</D:getcontentlength>
<D:getetag>"841e33-01674927542"</D:getetag>
<D:getlastmodified>Sat, 28 Jan 2023 17:39:02 GMT</D:getlastmodified>
<lp1:resourcetype/>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
<D:getcontenttype>application/octet-stream</D:getcontenttype>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>
2023/01/28 17:39:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:17 DEBUG : HTTP REQUEST (req 0xc000269800)
2023/01/28 17:39:17 DEBUG : PROPFIND /dav/iplant/home/djyoung/backups/test-rclone-wd16/ HTTP/1.1
Host: data.cyverse.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Depth: 1
Referer: https://data.cyverse.org/
Accept-Encoding: gzip
2023/01/28 17:39:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:17 DEBUG : HTTP RESPONSE (req 0xc000269800)
2023/01/28 17:39:17 DEBUG : HTTP/1.1 207 Multi-Status
Content-Length: 1697
Access-Control-Allow-Headers: Range
Access-Control-Allow-Methods: GET, HEAD, OPTIONS
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public
Content-Type: text/xml; charset="utf-8";charset=utf-8
Date: Sat, 28 Jan 2023 17:39:17 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
Vary: Authorization
Via: 1.1 varnish (Varnish/6.5)
X-Cache: MISS
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:">
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/iplant/home/djyoung/backups/test-rclone-wd16/</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Sat, 28 Jan 2023 17:38:48 GMT</D:creationdate>
<D:getetag>"01674927528"</D:getetag>
<D:getlastmodified>Sat, 28 Jan 2023 17:38:48 GMT</D:getlastmodified>
<lp1:resourcetype><D:collection/></lp1:resourcetype>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
<D:getcontenttype>application/octet-stream</D:getcontenttype>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/iplant/home/djyoung/backups/test-rclone-wd16//DJI_0004.JPG</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Sat, 28 Jan 2023 17:38:49 GMT</D:creationdate>
<D:getcontentlength>8658483</D:getcontentlength>
<D:getetag>"841e33-01674927542"</D:getetag>
<D:getlastmodified>Sat, 28 Jan 2023 17:39:02 GMT</D:getlastmodified>
<lp1:resourcetype/>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
<D:getcontenttype>application/octet-stream</D:getcontenttype>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>
2023/01/28 17:39:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:17 ERROR : /DJI_0004.JPG: Entry doesn't belong in directory "" (contains subdir) - ignoring
2023/01/28 17:39:17 DEBUG : webdav root 'dav/iplant/home/djyoung/backups/test-rclone-wd16': Waiting for checks to finish
2023/01/28 17:39:17 DEBUG : webdav root 'dav/iplant/home/djyoung/backups/test-rclone-wd16': Waiting for transfers to finish
2023/01/28 17:39:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:17 DEBUG : HTTP REQUEST (req 0xc0007d0d00)
2023/01/28 17:39:17 DEBUG : MKCOL /dav/iplant/home/djyoung/backups/test-rclone-wd16/ HTTP/1.1
Host: data.cyverse.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Referer: https://data.cyverse.org/
Accept-Encoding: gzip
2023/01/28 17:39:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:17 DEBUG : HTTP RESPONSE (req 0xc0007d0d00)
2023/01/28 17:39:17 DEBUG : HTTP/1.1 405 Method Not Allowed
Content-Length: 272
Allow: TRACE
Content-Type: text/html; charset=iso-8859-1
Date: Sat, 28 Jan 2023 17:39:17 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>405 Method Not Allowed</title>
</head><body>
<h1>Method Not Allowed</h1>
<p>The requested method MKCOL is not allowed for the URL /dav/iplant/home/djyoung/backups/test-rclone-wd16/.</p>
</body></html>
2023/01/28 17:39:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:17 DEBUG : HTTP REQUEST (req 0xc0007d1100)
2023/01/28 17:39:17 DEBUG : PUT /dav/iplant/home/djyoung/backups/test-rclone-wd16/DJI_0004.JPG HTTP/1.1
Host: data.cyverse.org
User-Agent: rclone/v1.61.1
Content-Length: 8658483
Authorization: XXXX
Content-Type: image/jpeg
Referer: https://data.cyverse.org/
Accept-Encoding: gzip
2023/01/28 17:39:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:19 DEBUG : HTTP RESPONSE (req 0xc0007d1100)
2023/01/28 17:39:19 DEBUG : HTTP/1.1 204 No Content
Content-Length: 0
Age: 0
Cache-Control: public
Content-Type: application/octet-stream
Date: Sat, 28 Jan 2023 17:39:17 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
Vary: Accept
Via: 1.1 varnish (Varnish/6.5)
X-Cache: MISS
2023/01/28 17:39:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:19 DEBUG : HTTP REQUEST (req 0xc000590600)
2023/01/28 17:39:19 DEBUG : PROPFIND /dav/iplant/home/djyoung/backups/test-rclone-wd16/DJI_0004.JPG HTTP/1.1
Host: data.cyverse.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Depth: 1
Referer: https://data.cyverse.org/
Accept-Encoding: gzip
2023/01/28 17:39:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/28 17:39:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:19 DEBUG : HTTP RESPONSE (req 0xc000590600)
2023/01/28 17:39:19 DEBUG : HTTP/1.1 207 Multi-Status
Content-Length: 909
Access-Control-Allow-Headers: Range
Access-Control-Allow-Methods: GET, HEAD, OPTIONS
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public
Content-Type: text/xml; charset="utf-8";charset=utf-8
Date: Sat, 28 Jan 2023 17:39:19 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
Vary: Authorization
Via: 1.1 varnish (Varnish/6.5)
X-Cache: MISS
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:">
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/iplant/home/djyoung/backups/test-rclone-wd16/DJI_0004.JPG</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Sat, 28 Jan 2023 17:38:49 GMT</D:creationdate>
<D:getcontentlength>8658483</D:getcontentlength>
<D:getetag>"841e33-01674927558"</D:getetag>
<D:getlastmodified>Sat, 28 Jan 2023 17:39:18 GMT</D:getlastmodified>
<lp1:resourcetype/>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
<D:getcontenttype>application/octet-stream</D:getcontenttype>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>
2023/01/28 17:39:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/28 17:39:19 INFO : DJI_0004.JPG: Copied (new)
2023/01/28 17:39:19 INFO :
Transferred: 8.257 MiB / 8.257 MiB, 100%, 8.247 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 2.9s
2023/01/28 17:39:19 DEBUG : 5 go routines active
The problem appears to be the double slash here // - this is enough to to confuse rclone into thinking that the file DJI_0004.JPG doesn't exist, whereas it sees it is being given files for another directory here.
So this looks like a bug in the provider.
It might be possible to work-around this in rclone.
I checked with the provider (scientific data store CyVerse), and they found that this is a known bug with the WebDAV service they use, davrods.
That bug report suggests that most WebDAV clients are tolerant of the double-slash, but another rclone user there reports the issue with rclone. The davrods developer does not appear to be prioritizing this fix. If it's easy to build in tolerance for this on the rclone end, I would much appreciate it! Otherwise I may be able to work with CyVerse to submit a PR to davrods.