Is there a way to log into an HTPP server?

#1

I’ve got an HTTP server that’s behing a user//password combination in order to access it, and I’d like to use rclone. I can’t figure out how to actually get rclone to register that I’m passing it a username//password though.

ash-4.3# rclone lsd remote: -vv
2019/01/25 23:55:26 DEBUG : rclone: Version “v1.45” starting with parameters [“rclone” “lsd” “remote:” “-vv”]
2019/01/25 23:55:26 DEBUG : Using config file from “/root/.config/rclone/rclone.conf”
2019/01/25 23:55:27 ERROR : : error listing: error listing “”: failed to readDir: HTTP Error 401: 401 Unauthorized
2019/01/25 23:55:27 Failed to lsd: error listing “”: failed to readDir: HTTP Error 401: 401 Unauthorized

and the config is

[remote]
type = http
url = https://insert_url_here
username = encrypted
password = encrypted
pass = encrypted
user = encrypted

I added those last 4 fields manually using the

rclone config password field key

but they don’t seem to work. Is there any way to do this?

Thanks.

#2

I’m reasonably sure you can put the username and password in the URL

so http://user:password@www.example.com/path

#3

I’ve seen that with other things, like wget, but it’s not quite working here.

ash-4.3# rclone sync remote:/ /volume1/location
2019/02/03 23:39:55 Failed to create file system for “remote:/”: parse “https://user:password@site”/: first path segment in URL cannot contain colon

Sorry for the late response, been a bit busy.

#4

That works for me when I do it like this against a local http server

rclone lsf --http-url "http://user:pass@localhost:8081" :http:

And also when I put it in the config file

[testhttp]
type = http
url = http://user:pass@localhost:8081

rclone lsf testhttp:
#5

Huh. Somehow when I typed it in myself it didn’t work, but when I copied from your post it did. Not quite sure why but I’ll take it. Thank you!

#6

Sorry to bother again but it isn’t quite working still.

ash-4.3# rclone copy remote: . --retries 10
2019/02/04 12:10:01 ERROR : E: Failed to copy: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:01 ERROR : I: Failed to copy: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:01 ERROR : M: Failed to copy: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:01 ERROR : Oneshots: Failed to copy: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:03 ERROR : Attempt 1/10 failed with 4 errors and: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:10 ERROR : E: Failed to copy: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:10 ERROR : Attempt 2/10 failed with 1 errors and: failed to open source object: Open failed: HTTP Error 404: 404 Not Found
2019/02/04 12:10:17 ERROR : Attempt 3/10 succeeded

It “succeeds” but the end result is a local folder full of HTML files, and it doesn’t crawl through the folders recursively, like how wget would. I think it should be doing this automatically, no? Or is there something that I should be setting.

Sorry for the bothers.

#7

It should be crawling through the files yes. Any chance you could PM me the access details so I can have a quick look?

#8

I don’t think that I can. There’s no way for me to change my password on the site and I can’t create a new account for you to mess with, so I"m not really comfortable sharing the access details. Is there anything else I could provide that could help with diagnosis, logs or anything?

#9

Fair enough!

Try running with rclone lsf -vv remote: - does that print the expected top level files and directories? If not then rclone isn’t parsing the HTML page properly.

#10

ash-4.3# rclone lsf -vv remote:
2019/02/05 12:16:36 DEBUG : rclone: Version “v1.45” starting with parameters [“rclone” “lsf” “-vv” “remote:”]
2019/02/05 12:16:36 DEBUG : Using config file from “/root/.config/rclone/rclone.conf”
A
B
C
D
E
F
G
H
I
J
K
L
M
N
Non-English
O
P
Q
R
README.txt
S
Subfolders.txt
T
U
V
W
X
Y
Z
Some other files
2019/02/05 12:16:45 DEBUG : 4 go routines active
2019/02/05 12:16:45 DEBUG : rclone: Version “v1.45” finishing with parameters [“rclone” “lsf” “-vv” “remote:”]

All of the directories are there, as well as the files, just the same that I see it in a normal web browser.

#11

Are the A -Z directories? rclone thinks they are files - they would end with / if rclone thought they were directories.

Can you save the HTML source of that page and attach it or email it to me (put a link to the forum page) at nick@craig-wood.com

#12

I’ve sent you an email.

#13

Thank you :smile:

Yes it looks like rclone is confused by the links to the directories which appear without a / on the end in the HTML.

If you go to one of the sub pages in your browser, say the A directory, does it show a / on the end in the URL bar? I’m wondering if the site redirects to something with a / on the end - this would be something rclone could detect easily.

If that doesn’t work then I could put a flag in to treat all text/html pages as directories.

#14

If you could send a log from this, it would be very helpful so I can see what exactly the server sends in the headers (which might be helpful).

rclone log -vv --dump bodies --log-file rclone.log lsf remote:
#15

It doesn’t redirect to something with a / at the end, so that might be part of the problem.

My rclone doesn’t seem to have a log command?

ash-4.3# rclone log -vv --dump bodies --log-file rclone.log lsf remote:
Error: unknown command “log” for “rclone”

Did you mean this?
ls
lsd
lsf
lsl

Run ‘rclone --help’ for usage.
2019/02/07 10:20:30 Fatal error: unknown command “log” for “rclone”

Did you mean this?
ls
lsd
lsf
lsl

ash-4.3# rclone --version
rclone v1.45

  • os/arch: linux/amd64
  • go version: go1.11.2
#16

OK. That would be too easy :wink:

Sorry, typo… What I meant was

rclone -vv --dump bodies --log-file rclone.log lsf remote:
#17

That’s kinda what I thought but I was thinking maybe I’m not on the beta version or something.

Sent.

#18

Thanks for the listing - very helpful.

Here is a beta for you to try. Use the --http-no-slash flag, or put no_slash = true into your config.

Hopefully that will work!

https://beta.rclone.org/branch/v1.45-188-ge29b3c9d-fix-http-html-dir-beta/ (uploaded in 15-30 mins)

#19

It’s kind of working?

I’m running it right now, buit I’m getting a lot of errors like this.

2019-02-08 11:19:45 ERROR : A/AM/AMER: error reading source directory: error listing “A/AM/AMER/”: directory not found

or even like this

2019-02-08 11:17:17 ERROR : I/I_/I_HA/I Hate Love/file.zip: Failed to copy: failed to open source object: Open failed: HTTP Error 404: 404 Not Found

I’m going to let it run for the night and see what ends up happening.

#20

OK, let me know what happens!

Can you try going to that URL to see if it exists?

And that one too?