I am using rclone on an android phone (Termux) and synching to WebDav on a synology NAS.
rclone version 1.55.0-DEV
Everything works great except for symlinks when using the --links directive
--copy-links works fine but I don't want to copy the contents of all symbolics
Every symlink fails in the same way. Using --copy-links with the same command works fine so the problem is purely due to the attempt to re-draft the symlinks per --links
A segment of the output log is below
2021/04/26 22:58:46 INFO : usr/share/man/man3/CURLOPT_XFERINFODATA.3.gz: Copied (new)
2021/04/26 22:58:46 INFO : usr/share/man/man3/CURLOPT_XFERINFOFUNCTION.3.gz: Copied (new)
2021/04/26 22:58:46 INFO : usr/share/man/man3/CURLOPT_XOAUTH2_BEARER.3.gz: Copied (new)
2021/04/26 22:59:23 ERROR : usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=30 with Body length 28
2021/04/26 22:59:24 ERROR : usr/share/man/man3/LINES.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/LINES.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=30 with Body length 28
2021/04/26 22:59:29 ERROR : usr/share/man/man3/PAIR_NUMBER.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/PAIR_NUMBER.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=22 with Body length 17
2021/04/26 22:59:31 ERROR : usr/share/man/man3/PC.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/PC.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=22 with Body length 19
2021/04/26 23:01:05 ERROR : usr/share/man/man3/SP.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/SP.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=34 with Body length 30
2021/04/26 23:01:06 ERROR : usr/share/man/man3/TABSIZE.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/TABSIZE.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=30 with Body length 28
2021/04/26 23:01:12 ERROR : usr/share/man/man3/TYPE_ALNUM.3form.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/TYPE_ALNUM.3form.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=26 with Body length 23
2021/04/26 23:01:13 ERROR : usr/share/man/man3/TYPE_ALPHA.3form.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/TYPE_ALPHA.3form.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=26 with Body length 23
2021/04/26 23:10:11 ERROR : usr/share/man/man3/_tracedump.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/_tracedump.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=22 with Body length 17
2021/04/26 23:10:12 ERROR : usr/share/man/man3/_tracemouse.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/_tracemouse.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=22 with Body length 17
2021/04/26 23:10:19 INFO : usr/share/man/man3/add_wch.3ncurses.gz: Copied (new)
2021/04/26 23:10:19 INFO : usr/share/man/man3/add_wchstr.3ncurses.gz: Copied (new)
2021/04/26 23:10:21 INFO : usr/share/man/man3/addch.3ncurses.gz: Copied (new)
2021/04/26 23:10:22 INFO : usr/share/man/man3/addchstr.3ncurses.gz: Copied (new)
2021/04/26 23:11:06 ERROR : usr/share/man/man3/acs_map.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/acs_map.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=34 with Body length 30
2021/04/26 23:11:15 ERROR : usr/share/man/man3/add_wchnstr.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/add_wchnstr.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=26 with Body length 22
2021/04/26 23:11:18 ERROR : usr/share/man/man3/addchnstr.3ncurses.gz.rclonelink: Failed to copy: Put "https://molemoor.synology.me:5006/Fileshares/termux/termux/usr/share/man/man3/addchnstr.3ncurses.gz.rclonelink": net/http: HTTP/1.x transport connection broken: http: ContentLength=22 with Body length 20
2021/04/26 23:11:23 INFO : usr/share/man/man3/addstr.3ncurses.gz: Copied (new)
2021/04/26 23:11:24 INFO : usr/share/man/man3/addwstr.3ncurses.gz: Copied (new)
Bearing in mind I am following instructions robotically and /data/data/com.termux/files/usr/share/man/man3/CURLOPT_XFERINFODATA.3.gz was one of the items in the log is not on an ERROR line
Thanks for your help here. Not sure these are going to help but here goes:
a) rclone --links lsl /data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink
2021/04/28 12:28:07 ERROR : : error listing: directory not found
2021/04/28 12:28:07 Failed to lsl with 2 errors: last error was: directory not found
b) rclone --links cat /data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink | wc
2021/04/28 12:29:55 ERROR : : error listing: directory not found
2021/04/28 12:29:55 Failed to cat with 2 errors: last error was: directory not found
0 0 0
c) rclone --links cat /data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink | hexdump
2021/04/28 12:30:59 ERROR : : error listing: directory not found
2021/04/28 12:30:59 Failed to cat with 2 errors: last error was: directory not found
2021/04/28 12:30:59 ERROR : : error listing: directory not found
2021/04/28 12:30:59 Failed to cat with 2 errors: last error was: directory not found
~/.shortcuts $ rclone copy --links -vv --dump bodies /data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink FilesWebdav:termux/test --retries 1
Enter configuration password:
password:
2021/04/28 12:32:03 DEBUG : Using config file from "/data/data/com.termux/files/home/.config/rclone/rclone.conf"
2021/04/28 12:32:03 DEBUG : rclone: Version "v1.55.0-DEV" starting with parameters ["rclone" "copy" "--links" "-vv" "--dump" "bodies" "/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink" "FilesWebdav:termux/test" "--retries" "1"]
2021/04/28 12:32:03 DEBUG : Creating backend with remote "/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink"
2021/04/28 12:32:03 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2021/04/28 12:32:03 DEBUG : fs cache: renaming cache item "/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink" to be canonical "local{b6816}:/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz.rclonelink"
2021/04/28 12:32:03 DEBUG : Creating backend with remote "FilesWebdav:termux/test"
2021/04/28 12:32:03 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/28 12:32:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/28 12:32:03 DEBUG : HTTP REQUEST (req 0x400066f500)
2021/04/28 12:32:03 DEBUG : PROPFIND /Fileshares/termux/test HTTP/1.1
Host: molemoor.synology.me:5006
User-Agent: rclone/v1.55.0-DEV
Authorization: XXXX
Depth: 1
Referer: https://molemoor.synology.me:5006/Fileshares/
Accept-Encoding: gzip
2021/04/28 12:32:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/28 12:32:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/28 12:32:03 DEBUG : HTTP RESPONSE (req 0x400066f500)
2021/04/28 12:32:03 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 196
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 28 Apr 2021 12:31:58 GMT
Server: Apache
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
2021/04/28 12:32:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/28 12:32:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/28 12:32:03 DEBUG : HTTP REQUEST (req 0x400066e700)
2021/04/28 12:32:03 DEBUG : PROPFIND /Fileshares/termux/test/ HTTP/1.1
Host: molemoor.synology.me:5006
User-Agent: rclone/v1.55.0-DEV
Authorization: XXXX
Depth: 1
Referer: https://molemoor.synology.me:5006/Fileshares/
Accept-Encoding: gzip
2021/04/28 12:32:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/28 12:32:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/28 12:32:03 DEBUG : HTTP RESPONSE (req 0x400066e700)
2021/04/28 12:32:03 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 196
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 28 Apr 2021 12:31:58 GMT
Server: Apache
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
2021/04/28 12:32:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/28 12:32:03 ERROR : : error reading source directory: directory not found
2021/04/28 12:32:03 DEBUG : webdav root 'termux/test': Waiting for checks to finish
2021/04/28 12:32:03 DEBUG : webdav root 'termux/test': Waiting for transfers to finish
2021/04/28 12:32:03 ERROR : Attempt 1/1 failed with 1 errors and: directory not found
2021/04/28 12:32:03 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 4.8s
2021/04/28 12:32:03 DEBUG : 4 go routines active
2021/04/28 12:32:03 Failed to copy: directory not found
ata/com.termux/files/home/.config/rclone/rclone.conf"
2021/04/28 12:33:48 DEBUG : rclone: Version "v1.55.0-DEV" starting with parameters ["rclone" "copy" "--links" "-vv" "--dump" "bodies" "/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz" "FilesWebdav:termux/test" "--retries" "1"]
2021/04/28 12:33:48 DEBUG : Creating backend with remote "/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz"
2021/04/28 12:33:48 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2021/04/28 12:33:48 DEBUG : fs cache: adding new entry for parent of "/data/data/com.termux/files/usr/share/man/man3/ESCDELAY.3ncurses.gz", "local{b6816}:/data/data/com.termux/files/usr/share/man/man3"
2021/04/28 12:33:48 DEBUG : Creating backend with remote "FilesWebdav:termux/test"
2021/04/28 12:33:48 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/28 12:33:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/28 12:33:48 DEBUG : HTTP REQUEST (req 0x40004cfc00)
2021/04/28 12:33:48 DEBUG : PROPFIND /Fileshares/termux/test HTTP/1.1
Host: molemoor.synology.me:5006
User-Agent: rclone/v1.55.0-DEV
Authorization: XXXX
Depth: 1
Referer: https://molemoor.synology.me:5006/Fileshares/
Accept-Encoding: gzip
2021/04/28 12:33:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/28 12:33:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/28 12:33:49 DEBUG : HTTP RESPONSE (req 0x40004cfc00)
2021/04/28 12:33:49 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 196
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 28 Apr 2021 12:33:44 GMT
Server: Apache
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
2021/04/28 12:33:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/28 12:33:49 ERROR : Attempt 1/1 failed with 1 errors and: object not found
2021/04/28 12:33:49 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 5.0s
2021/04/28 12:33:49 DEBUG : 4 go routines active
2021/04/28 12:33:49 Failed to copy: object not found
Hmm, that didn't work as expected! My mistake - I forgot that there is a little bug in rclone which means naming symlinks on the command line is a bit hit and miss.
Which should hopefully just copy one file and it should be found this time. If the file isn't found can you adjust the path or the include so it does find one or more files?
This should fail the same way your original post did, but give a lot more debug.
I can see the errors but I can't see what I really wanted to see which was the http transaction - I think this failed before even sending the http transaction.
I suspect the hexdump bit didn't do its dumping (as both outputs are similar), but I'm mostly a windows person so will need to do some serious research to redraft the command...
b) I tried to create a new simplified source containing just a few symlinks, both to folders and to individual files, and all the symlink names a single word. Both types of symlinks carry the problem.
c) I then tried to change the destination from WebDav to Google Drive, and the problem and error messages are the same, so it's not a webdav-specific problem.
d) I tried to use copy instead of sync, no difference.
The runs complete with a number of errors equal to the number of symlinks, and in the destination no sign of the symlinks.
I wonder if this could be something specific to Termux (this is an unaltered Termux install apart from a few aliases added)?
Can you run the cat and the lsl tests with this test binary please and post the logs.
You'll find an android binary in the testbuilds subdirectory. You'll have to pick one to match your device. If you aren't sure try the armv8a one and if that doesn't work try the armv7a one. You'll need to gunzip the builds - you can use wget to download them straight to the phone and then gunzip them, something like
wget https://beta.rclone.org/branch/fix-local-symlink/v1.56.0-beta.5461.1dc58a924.fix-local-symlink/testbuilds/rclone-android-21-armv8a.gz
gunzip rclone-android-21-armv8a.gz
./rclone-android-21-armv8a version
I've just realised that there is an option which will fix this for you.
Try your sync with the --local-zero-size-links option. Probably not the best named option in the world, but that will make rclone read the links rather than just looking at their size in the listing.
and /data/data/com.termux/files/usr/share/man/man3/CURLOPT_XFERINFODATA.3.gz was one of the items in the log is not on an ERROR line