Rclone does not see any files on some pages, while browsers are able do display link-lists

What is the problem you are having with rclone?

rclone does not see any files on some pages, while browsers are able do display link-lists to download. Pages are generated having sources like:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html><head>
  <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">

  <link rel="stylesheet" type="text/css" href="/style/style.css" title="default">
  <link rel="alternate" type="application/rss+xml" title="<function render_body.<locals>.title at 0x7fed9c0e9c60> RSS feed" href="rss/atom.xml" />

  <link rel="shortcut icon" href="favicon.png">
  <title>Haiku 32-bit Hybrid | Haiku Files</title></head><body>
  <div id="wrapper">
   <div id="header">
    <div id="logo">
    <a href="http://download.haiku-os.org/"><img src="/style/haiku_logo_white.svg" alt="logo" style="height: 98px; width: 305px; margin: -5px 0 0 -15px; padding: 0 0 0px 3px;"></a>
    </div>
    <div id="links">
     <a href="http://www.haiku-os.org/"><img src="/style/icon_haiku_32.png" alt="Haiku Project" title="Haiku Project"></a>
     <a href="http://dev.haiku-os.org/"><img src="/style/icon_bug_tracker_32.png" alt="Haiku's Bug Tracker" title="Haiku's Bug Tracker"></a>
     <img src="/style/top-link-separator.png" alt="sep" style="padding: 0pt 10px;">
     <a href="http://www.haiku-os.org/contact"><img src="/style/icon_32_mail.png" alt="contact" title="Contact the Haiku Project"></a>
     <a href="http://www.haiku-os.org/community/donating_to_haiku"><img src="/style/icon_32_donate.png" alt="donate" title="Make a donation to the Haiku Project"></a>
    </div>
   </div>
   <div id="menu">
   </div>
<div id="content">
	<div class="box">
		
		<p>
The Haiku source is continually built by a dedicated machine. These "nightly
images" are provided mainly for development and testing purposes. You can
download and install these snapshots to check out the latest features
and bugfixes, be aware though that they <strong>may be unstable</strong> at times.
		</p>

		<p>
For more information on missing packages, image and archive formats, etc.,
please refer to this site's <a href="http://download.haiku-os.org/">main page</a>.
		</p>
	</div>

	<div class="box">
		<img src="/style/ico_64_folder_dev.png" alt="nightlies">
		<h2>Supported 32-bit Nightly Images (x86 GCC 2 Hybrid)</h2>
		<p>
		</p>

        <table>
            <tr>
                <th>Revision</th>
                    <th>Anyboot ISO</th>
            </tr>

                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev59171">hrev59171</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59171-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59171-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59171-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>
                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev59169">hrev59169</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59169-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59169-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59169-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>
                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev59166">hrev59166</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59166-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59166-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59166-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>
                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev59164">hrev59164</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59164-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59164-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59164-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>
                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev59162">hrev59162</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59162-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59162-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59162-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>
                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev59159">hrev59159</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59159-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59159-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev59159-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>

[…]

                <tr>
                    <td><a href="http://cgit.haiku-os.org/haiku/tag/?id=hrev56737">hrev56737</a></td>
                        <td>
                                    <a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev56737-x86_gcc2h-anyboot.zip">eu</a>&nbsp;
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev56737-x86_gcc2h-anyboot.zip.sha256">checksum</a>]</span>
                                    <span class="checksum">[<a href="https://haiku-nightly.cdn.haiku-os.org/x86_gcc2h/haiku-master-hrev56737-x86_gcc2h-anyboot.zip.minisig">minisig</a>]</span>
                        </td>
                </tr>
        </table>
	</div>
</div>

<div id="footer">
Copyright 2001 - 2021 Haiku, Inc. &#8212; Haiku&#174; and the
HAIKU logo&#174; are registered trademarks of
<a href="http://www.haiku-inc.org">Haiku, Inc.</a>
</div>
		</div>
	</body>
</html>

Run the command 'rclone version' and share the full output of the command.

rclone v1.72.0
- os/version: ubuntu 25.04 (64 bit)
- os/kernel: 6.14.0-36-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.25.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Unknown, http

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls ':http,url="https://download.haiku-os.org/":nightly-images/x86_gcc2h/'

It does not matter if trailing `/` is given or not. Results are the same: rclone does not find anything to display.

The rclone config contents with secrets removed.

No config or empty config.

A log from the command with the -vv flag

$ rclone -vv ls ':http,url="https://download.haiku-os.org/":nightly-images/x86_gcc2h/'
2025/11/26 19:10:45 DEBUG : rclone: Version "v1.72.0" starting with parameters ["rclone" "-vv" "ls" ":http,url=\"https://download.haiku-os.org/\":nightly-images/x86_gcc2h/"]
2025/11/26 19:10:45 DEBUG : Creating backend with remote ":http,url=\"https://download.haiku-os.org/\":nightly-images/x86_gcc2h/"
2025/11/26 19:10:45 DEBUG : Using config file from "/home/tps/.config/rclone/rclone.conf"
2025/11/26 19:10:45 DEBUG : :http: detected overridden config - adding "{hcVhH}" suffix to name
2025/11/26 19:10:45 DEBUG : Root: https://download.haiku-os.org/nightly-images/x86_gcc2h/
2025/11/26 19:10:45 DEBUG : fs cache: renaming cache item ":http,url=\"https://download.haiku-os.org/\":nightly-images/x86_gcc2h/" to be canonical ":http{hcVhH}:nightly-images/x86_gcc2h/"
2025/11/26 19:10:45 DEBUG : 5 go routines active

Because http remote is not all purpose links scrapping utility.

It accepts specific format as per docs:

The webserver should provide file listings which rclone will read and turn into a remote.

Something like this for example:

In general something similar to what Apache, Nginx or Caddy uses for standard file listing.

What you can do is to create a text file (links.txt) with one URL per line and then run something like:

for url in $(cat links.txt); do rclone copyurl "$url" remote:path/to/destination/ -a; done

Or you can always either use “simple” wget:

wget -r -l1 -H -t1 -nd -N -np -A.zip -erobots=off https://download.haiku-os.org/nightly-images/x86_gcc2h/

or search for some dedicated scrapping tools.

I'll just note that rclone copyurl gained the --urls flag recently which means you can give it a file of URLs one per line

1 Like