What must a webpage offer to be recognized as an "http" backend?

Help and Support
1

What is the problem you are having with rclone?

I am trying to mirror files hosted in a backblaze bucket. Links to all files are present in an index.html file. I can edit the index.html to show whatever required, I just don't know what is required. I've also tried with the url Zotero packages for Debian-based systems but am getting the same results.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV

  • os/arch: linux/amd64
  • go version: go1.15.4

Which cloud storage system are you using? (eg Google Drive)

website hosted in backblaze bucket

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsd --http-url https://zotero.retorque.re/file/apt-package-archive/ :http:

The rclone config contents with secrets removed.

[mirror]
type = http
url = https://zotero.retorque.re/file/apt-package-archive/

A log from the command with the -vv flag

2022/03/20 00:17:59 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "lsd" "--http-url" "https://zotero.retorque.re/file/apt-package-archive/" ":http:" "-vv"]
2022/03/20 00:17:59 DEBUG : Creating backend with remote ":http:"
2022/03/20 00:17:59 DEBUG : Using config file from "/home/emile/.config/rclone/rclone.conf"
2022/03/20 00:17:59 ERROR : : error listing: error listing "": failed to readDir: HTTP Error 400: 400 Bad Request
2022/03/20 00:17:59 DEBUG : 5 go routines active
2022/03/20 00:17:59 Failed to lsd with 2 errors: last error was: error listing "": failed to readDir: HTTP Error 400: 400 Bad Request
2

hello and welcome to the forum,

the first thing to do is update rclone to the latest stable v1.58.0
as the version is years old and looks like some custom compiled, development version.

try using https://beta.rclone.org/ as an example.

3

Oh I just used the version that was available on Ubuntu 21.10.

With 1.58.0 I'm getting the same results. Are the CSS classes on that beta page the relevant bits?

4

the internet is full of simple how-tos

https://www.wikihow.com/Create-a-Simple-Web-Page-with-HTML

5

I know how to create a HTML page. What I need to know is what specific features need to be on a page (specific CSS classes? Other metadata?) to have rclone recognize it as a data source. Clearly just a list of links isn't it, or https://zotero.retorque.re/file/apt-package-archive/ would already have worked.

6 
rclone lsd --http-url https://beta.rclone.org/  :http: -vv

doesn't seem to do anything either. I'm not getting errors, but after

rclone lsd --http-url https://beta.rclone.org/  :http: -vv
2022/03/20 01:24:51 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "lsd" "--http-url" "https://beta.rclone.org/" ":http:" "-vv"]
2022/03/20 01:24:51 DEBUG : Creating backend with remote ":http:"
2022/03/20 01:24:51 DEBUG : :http: detected overridden config - adding "{_rLKO}" suffix to name
2022/03/20 01:24:51 DEBUG : Using config file from "/home/emile/.config/rclone/rclone.conf"
2022/03/20 01:24:51 DEBUG : Root: https://beta.rclone.org/
2022/03/20 01:24:51 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{_rLKO}:"

nothing else seems to come.

7

rclone lsd --http-url=https://downloads.rclone.org :http:

8

as i mentioned, using firefox, i tried that link and got the same error as rclone got

code	"bad_request"
message	"File names must contain at least one character"
status	400
9

The page you listed returns a 400 so you can't browse it/etc. You'd ask the person running the site to make it function :slight_smile:

10

<a href=""> links to the files and directories in that directory should be sufficient. URLs linking to directories should end in / links to files shouldn't.

The URLs need to be under the URL of the index.

That's about it.

Example

rclone lsf --http-url https://downloads.rclone.org/  :http:
11

Ah OK. Then it must be the 404 when the slash is present. That will be fixable. Thanks to both!

12

There is a flag to control the slash thing

13

D'oh! How did I miss that?!

1 Like
14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.