Rclone does not work if servers redirect within a path

Help and Support
1

What is the problem you are having with rclone?

rclone does not work with servers redirecting to others within a given url-path

Run the command 'rclone version' and share the full output of the command.

rclone v1.72.1
- os/version: ubuntu 25.10 (64 bit)
- os/kernel: 6.17.0-8-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.25.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

  • http, https

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ rclone -vv lsf --http-url https://ftp.netbsd.org/ :http:/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

$ rclone config redacted
; empty config
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

$ rclone -vv lsf --http-url https://ftp.netbsd.org/ :http:/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/
2025/12/13 22:44:29 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "-vv" "lsf" "--http-url" "https://ftp.netbsd.org/" ":http:/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/"]
2025/12/13 22:44:29 DEBUG : Creating backend with remote ":http:/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/"
2025/12/13 22:44:29 DEBUG : Using config file from "/home/tps/.config/rclone/rclone.conf"
2025/12/13 22:44:29 DEBUG : :http: detected overridden config - adding "{WiZVP}" suffix to name
2025/12/13 22:44:29 DEBUG : Root: https://ftp.netbsd.org/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/
2025/12/13 22:44:29 DEBUG : fs cache: renaming cache item ":http:/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/" to be canonical ":http{WiZVP}:/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/"
2025/12/13 22:44:30 ERROR : error listing: error listing "": directory not found
2025/12/13 22:44:30 DEBUG : 4 go routines active
2025/12/13 22:44:30 NOTICE: Failed to lsf with 2 errors: last error was: error in ListJSON: error listing "": directory not found

The directory "HEAD" is extracted from is listed using command:

$ rclone -vv lsf --http-url https://ftp.netbsd.org/ :http:/pub/NetBSD-daily/
2025/12/13 22:15:22 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "-vv" "lsf" "--http-url" "https://ftp.netbsd.org/" ":http:/pub/NetBSD-daily/"]
2025/12/13 22:15:22 DEBUG : Creating backend with remote ":http:/pub/NetBSD-daily/"
2025/12/13 22:15:22 DEBUG : Using config file from "/home/tps/.config/rclone/rclone.conf"
2025/12/13 22:15:22 DEBUG : :http: detected overridden config - adding "{WiZVP}" suffix to name
2025/12/13 22:15:22 DEBUG : Root: https://ftp.netbsd.org/pub/NetBSD-daily/
2025/12/13 22:15:22 DEBUG : fs cache: renaming cache item ":http:/pub/NetBSD-daily/" to be canonical ":http{WiZVP}:/pub/NetBSD-daily/"
HEAD/
HEAD-lint/
HEAD-llvm/
netbsd-10/
netbsd-10-llvm/
netbsd-11/
netbsd-11-llvm/
netbsd-9/
2025/12/13 22:15:23 DEBUG : 6 go routines active

ftp.netbsd.org redirects to other servers for load and traffic balancing. If I use one of the redirected to servers directly it works, until this server itself redirects to another one because of load or because this server is going to maintenance.

Any idea how to overcome this problem, making it unnecessary to find out which servers are behind some name like ftp.netbsd.org using a given url directly as found in documents? As you can do with any browser out in the wild?

2

I've analysed what curl is doing while retrieving https://ftp.netbsd.org/, then https://ftp.netbsd.org/pub/, https://ftp.netbsd.org/pub/NetBSD-daily/:

First step: https://ftp.netbsd.org/:

$ curl -v https://ftp.netbsd.org/
* Host ftp.netbsd.org:443 was resolved.
* IPv6: 2001:470:a085:999::21
* IPv4: 199.233.217.201
*   Trying [2001:470:a085:999::21]:443...
[…]
* Connected to ftp.netbsd.org (2001:470:a085:999::21) port 443
* using HTTP/1.x
> GET / HTTP/1.1
> Host: ftp.netbsd.org
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 200 OK
< Date: Sat, 13 Dec 2025 22:18:56 GMT
< Server: bozohttpd/20240428
< Accept-Ranges: bytes
< Content-Type: text/html
< Connection: close
< 
<!DOCTYPE html>
<html><head><meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<style type="text/css">
table {
	border-top: 1px solid black;
	border-bottom: 1px solid black;
}
th { background: aquamarine; }
tr:nth-child(even) { background: lavender; }
</style>
<title>Index of /</title></head>
<body><h1>Index of /</h1>
<table cols=3>
<thead>
<tr><th>Name<th>Last modified<th align=right>Size
<tbody>
<tr><td><a href="../">Parent Directory</a><td>04-Aug-2025 16:12<td align=right>1kB
<tr><td><a href=".bzremap.old">.bzremap.old</a><td>12-Dec-2025 04:16<td align=right>140kB
<tr><td><a href=".message">.message</a><td>10-Feb-2016 06:22<td align=right>1kB
<tr><td><a href="cgi-bin/">cgi-bin/</a><td>05-Jan-2025 19:24<td align=right>1kB
<tr><td><a href="etc/">etc/</a><td>16-Aug-2009 11:21<td align=right>1kB
<tr><td><a href="ls-lRA.gz">ls-lRA.gz</a><td>13-Dec-2025 03:46<td align=right>40241kB
<tr><td><a href="pub/">pub/</a><td>30-Sep-2017 22:31<td align=right>1kB
<tr><td><a href="robots.txt">robots.txt</a><td>05-Jan-2024 19:19<td align=right>1kB
</table>
</body></html>

* TLSv1.3 (IN), TLS alert, close notify (256):
* shutting down connection #0

Second step https://ftp.netbsd.org/pub/:

$ curl -v https://ftp.netbsd.org/pub/
* Host ftp.netbsd.org:443 was resolved.
* IPv6: 2001:470:a085:999::21
* IPv4: 199.233.217.201
*   Trying [2001:470:a085:999::21]:443...
[...]
* Connected to ftp.netbsd.org (2001:470:a085:999::21) port 443
* using HTTP/1.x
> GET /pub/ HTTP/1.1
> Host: ftp.netbsd.org
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 200 OK
< Date: Sat, 13 Dec 2025 22:20:36 GMT
< Server: bozohttpd/20240428
< Accept-Ranges: bytes
< Content-Type: text/html
< Connection: close
< 
<!DOCTYPE html>
<html><head><meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<style type="text/css">
table {
	border-top: 1px solid black;
	border-bottom: 1px solid black;
}
th { background: aquamarine; }
tr:nth-child(even) { background: lavender; }
</style>
<title>Index of pub/</title></head>
<body><h1>Index of pub/</h1>
<table cols=3>
<thead>
<tr><th>Name<th>Last modified<th align=right>Size
<tbody>
<tr><td><a href="../">Parent Directory</a><td>13-Dec-2025 04:16<td align=right>1kB
<tr><td><a href="NetBSD/">NetBSD/</a><td>31-Jul-2025 17:03<td align=right>1kB
<tr><td><a href="NetBSD-archive/">NetBSD-archive/</a><td>12-Jul-2020 06:17<td align=right>1kB
<tr><td><a href="NetBSD-daily/">NetBSD-daily/</a><td>06-Oct-2016 17:25<td align=right>1kB
<tr><td><a href="pkgsrc/">pkgsrc/</a><td>22-Sep-2025 04:54<td align=right>1kB
</table>
</body></html>

* TLSv1.3 (IN), TLS alert, close notify (256):
* shutting down connection #0

Third step https://ftp.netbsd.org/pub/NetBSD-daily/

$ curl -v https://ftp.netbsd.org/pub/NetBSD-daily/
* Host ftp.netbsd.org:443 was resolved.
* IPv6: 2001:470:a085:999::21
* IPv4: 199.233.217.201
*   Trying [2001:470:a085:999::21]:443...
[...]
* Connected to ftp.netbsd.org (2001:470:a085:999::21) port 443
* using HTTP/1.x
> GET /pub/NetBSD-daily/ HTTP/1.1
> Host: ftp.netbsd.org
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 301 Document Moved
< Date: Sat, 13 Dec 2025 22:25:56 GMT
< Server: bozohttpd/20240428
< Accept-Ranges: bytes
< Content-Type: text/html
< Connection: close
< Location: https://nycdn.netbsd.org/pub/NetBSD-daily/
< 
<html><head><title>Document Moved</title></head>
<body><h1>Document Moved</h1>
This document had moved <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/">here</a>
</body></html>
* TLSv1.3 (OUT), TLS alert, decode error (562):
* OpenSSL SSL_read: OpenSSL/3.5.3: error:0A000126:SSL routines::unexpected eof while reading, errno 0
* closing connection #0
curl: (56) OpenSSL SSL_read: OpenSSL/3.5.3: error:0A000126:SSL routines::unexpected eof while reading, errno 0

Now here is a difference: Location: https://nycdn.netbsd.org/pub/NetBSD-daily/. This tells the browser to change to url https://nycdn.netbsd.org/pub/NetBSD-daily/. rclone respects this, but does not tell me about it. The directory listing retrieved and displayed is not from ftp.netbsd.org but from nycdn.netbsd.org:

$ rclone lsf --http-url https://ftp.netbsd.org :http:/pub/NetBSD-daily/
HEAD/
HEAD-lint/
HEAD-llvm/
netbsd-10/
netbsd-10-llvm/
netbsd-11/
netbsd-11-llvm/
netbsd-9/

Step four https://ftp.netbsd.org/pub/NetBSD-daily/HEAD/:

$ curl -v https://ftp.netbsd.org/pub/NetBSD-daily/HEAD
* Host ftp.netbsd.org:443 was resolved.
* IPv6: 2001:470:a085:999::21
* IPv4: 199.233.217.201
*   Trying [2001:470:a085:999::21]:443...
[...]
* Connected to ftp.netbsd.org (2001:470:a085:999::21) port 443
* using HTTP/1.x
> GET /pub/NetBSD-daily/HEAD HTTP/1.1
> Host: ftp.netbsd.org
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 301 Document Moved
< Date: Sat, 13 Dec 2025 22:38:33 GMT
< Server: bozohttpd/20240428
< Accept-Ranges: bytes
< Content-Type: text/html
< Connection: close
< Location: https://nycdn.netbsd.org/pub/NetBSD-daily/
< 
<html><head><title>Document Moved</title></head>
<body><h1>Document Moved</h1>
This document had moved <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/">here</a>
</body></html>
* TLSv1.3 (OUT), TLS alert, decode error (562):
* OpenSSL SSL_read: OpenSSL/3.5.3: error:0A000126:SSL routines::unexpected eof while reading, errno 0
* closing connection #0
curl: (56) OpenSSL SSL_read: OpenSSL/3.5.3: error:0A000126:SSL routines::unexpected eof while reading, errno 0

curl fails to retrieve https://ftp.netbsd.org/pub/NetBSD-daily/HEAD/ telling me about the new location, while rclone just fails:

$ rclone lsf -vv --http-url https://ftp.netbsd.org :http:/pub/NetBSD-daily/HEAD/
2025/12/13 23:42:43 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "lsf" "-vv" "--http-url" "https://ftp.netbsd.org" ":http:/pub/NetBSD-daily/HEAD/"]
2025/12/13 23:42:43 DEBUG : Creating backend with remote ":http:/pub/NetBSD-daily/HEAD/"
2025/12/13 23:42:43 DEBUG : Using config file from "/home/tps/.config/rclone/rclone.conf"
2025/12/13 23:42:43 DEBUG : :http: detected overridden config - adding "{ei14f}" suffix to name
2025/12/13 23:42:43 DEBUG : Root: https://ftp.netbsd.org/pub/NetBSD-daily/HEAD/
2025/12/13 23:42:43 DEBUG : fs cache: renaming cache item ":http:/pub/NetBSD-daily/HEAD/" to be canonical ":http{ei14f}:/pub/NetBSD-daily/HEAD/"
2025/12/13 23:42:44 ERROR : error listing: error listing "": directory not found
2025/12/13 23:42:44 DEBUG : 4 go routines active
2025/12/13 23:42:44 NOTICE: Failed to lsf with 2 errors: last error was: error in ListJSON: error listing "": directory not found

I see two possible solutions to this problem:

  1. make rclone descend the given path and follow any redirect it finds. Or
  2. make rclone report redirects as curl does. curl fails, but reports the location where the file it tried to retrieve would have been found and then leaves it up to the user to take action.

With browsers it depends: some automatically change the location and try again, but all of them display at least what the server gives back: the link the file could be found at:

<html><head><title>Document Moved</title></head>
<body><h1>Document Moved</h1>
This document had moved <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/">here</a>
</body></html>
3

Maybe the whole thing is related to WebDAV: Redirects for metadata queries are interpreted as "file not found" · Issue #6029 · rclone/rclone · GitHub

But this does not solve redirects of deep links like https://ftp.netbsd.org/pub/NetBSD-daily/HEAD/20251213122542Z/amd64/. This link is redirected at NetBSD-daily. The server tells you you've to follow another url even for HEAD. But after 20251213122542Z you are lost: the server then replies with 404 - not found.
I've found such behaviour for a wide range of servers. At the moment I'd say: don't expect any website to reside on one server only. Expect redirects within paths. Not following them will, in most cases, lead you to 404.