Previously i was using 1.51 version for copying files from https server it was working fine, Now I thought of upgrading it to 1.56.2 but the same copy command is giving an error

What is the problem you are having with rclone?

After serving my local drive in https server. When i try to copy files through copy command which is working for 1.51.0 version is giving some error in 1.56.2 version but when I try to serve my local drive through http removing --cert, --key and --client-ca flags, Copy command works fine. I want to use https

What is your rclone version (output from rclone version)

1.56.2

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe --config "Path" --size-only copy Https_C:"Testing" "C:\Testing" --client-cert "Path" --client-key "Path" --ca-cert "Path" --backup-dir "Path" --bwlimit 0 --transfers 8 --multi-thread-streams 0 --multi-thread-cutoff 1024M --create-empty-src-dirs --stats 5s --stats-one-line --retries 1 --copy-links --no-traverse --timeout 10m0s --contimeout 5m0s --max-backlog 102400 --low-level-retries 3 -v

The rclone config contents with secrets removed.

url with server password

A log from the command with the -vv flag

2021/10/25 16:12:30 Failed to copy: error listing "": failed to readDir: Get "https://cdmoldpc:***@192.168.1.8:8081/dummyfolder/": http: server gave HTTP response to HTTPS client

Are you using any rclone.conf?
Can you run the command with -vv and share the full output?

In rclone.conf I am passing the URL(containing user and password).

2021/10/25 16:12:30 ERROR : {URL}: error reading source root directory: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

2021/10/25 16:12:30 ERROR : Attempt 1/3 failed with 1 errors and: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

2021/10/25 16:12:30 ERROR : URL/: error reading source root directory: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

2021/10/25 16:12:30 ERROR : Attempt 2/3 failed with 1 errors and: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

2021/10/25 16:12:30 ERROR : URL: error reading source root directory: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

2021/10/25 16:12:30 ERROR : Attempt 3/3 failed with 1 errors and: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

2021/10/25 16:12:30 Failed to copy: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

Can you please run the command with -vv and share the full output?

Same logs are coming

Hi Indar,

It is very difficult to help you when we have an incomplete picture of your situation.

What we would like to see is your configuration, that is the output from this command:

rclone config show Https_C:

and then we also need to see the debug output from your command, that is:

rclone.exe … copy … -vv

Note there are two v’s in the end, the output you have posted above is indicates that you only used one v.

Please redact any secrets before posting the output, that is replace secrets by *** or similar.

What do you mean by this? Are you serving your C drive with rclone serve http? If so, then please post the command you are using for this - and shortly explain why you are using this setup.

You have a lot of parameters in your command line which makes it difficult for both you and us to get a clear picture of the situation. I therefore recommend you remove all the parameters that aren't strictly necessary to reproduce your issue. E.g. --size-only, --backup-dir, --bwlimit, --transfers, --multi-thead-streams, --multi-thread-cutoff, --create-empty-src-dirs --stats-one-line --retries --copy-links, --no-traverse, --timeout, --contimeout, --max-backlog, --low-level-retries.

Hi Ole,
Sorry for the previous comment I was missing one thing.
I am using rclone in my personal project for copying files.

Command for starting server-

rclone.exe serve http --config {rclone.conf file path} mylocal:C:\ --cert {Path} --key {Path} --client-ca {Path} --user testpc --pass Welcome@123 --addr 192.168.1.8:8081

Rclone.conf file content-
[mylocal]
type = local
copy_links=true

Command for copy-

rclone.exe --config {rclone.conf file path} --size-only copy Https_C:"Testing" "C:\Testing" --client-cert {Path} --client-key {Path} --ca-cert {Path} -vv

Rclone.conf file content-
[Https_C]
type = http
url = https://testpc:Welcome@123@192.168.1.8:8081

Debug output-

2021/10/26 14:07:14 DEBUG : Creating backend with remote "Https_C:Testing"
2021/10/26 14:07:14 DEBUG : Using config file from "rclone.conf file path"
2021/10/26 14:07:14 DEBUG : Creating backend with remote "C:\Testing"
2021/10/26 14:07:14 DEBUG : fs cache: renaming cache item "C:\Testing" to be canonical "//?/C:/Testing"
2021/10/26 14:07:14 ERROR : {URL }: error reading source root directory: error listing "": failed to readDir: Get "{URL}": http: server gave HTTP response to HTTPS client
2021/10/26 14:07:14 DEBUG : Local file system at //?/C:/Testing: Waiting for checks to finish
2021/10/26 14:07:14 DEBUG : Local file system at //?/C:/Testing: Waiting for transfers to finish
2021/10/26 14:07:14 ERROR : Attempt 1/3 failed with 1 errors and: error listing "": failed to readDir: Get "{URL}": http: server gave HTTP response to HTTPS client
2021/10/26 14:07:14 ERROR : {URL}: error reading source root directory: error listing "": failed to readDir: Get "{URL}": http: server gave HTTP response to HTTPS client
2021/10/26 14:07:14 DEBUG : Local file system at //?/C:/Testing: Waiting for checks to finish
2021/10/26 14:07:14 DEBUG : Local file system at //?/C:/Testing: Waiting for transfers to finish
2021/10/26 14:07:14 ERROR : Attempt 2/3 failed with 1 errors and: error listing "": failed to readDir: Get "{URL}": http: server gave HTTP response to HTTPS client
2021/10/26 14:07:14 ERROR : {URL}: error reading source root directory: error listing "": failed to readDir: Get "{URL}": http: server gave HTTP response to HTTPS client
2021/10/26 14:07:14 DEBUG : Local file system at //?/C:/Testing: Waiting for checks to finish
2021/10/26 14:07:14 DEBUG : Local file system at //?/C:/Testing: Waiting for transfers to finish
2021/10/26 14:07:14 ERROR : Attempt 3/3 failed with 1 errors and: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client
2021/10/26 14:07:14 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 7.0s

2021/10/26 14:07:14 DEBUG : 2 go routines active
2021/10/26 14:07:14 Failed to copy: error listing "": failed to readDir: Get "URL": http: server gave HTTP response to HTTPS client

The URL in debug output is the url present in the rclone.conf file above.
Same copy commands with same path and conf file works well with rclone 1.51.0 version but not with rclone 1.56.2 version.

Thanks, this certainly helps my understanding.

I never used rclone serve http or the http backend, but the error message “server gave HTTP response to HTTPS client” indicates an issue in the setup of your https server. That is the parameters to rclone serve http ....

Have you verified the correct functioning of the https server?

I would try my Chrome browser and check that the lock in the address bar (and inspect site settings) - click a folder link and then verify again.

Have you tried starting the server with -vv to see any debug messages?

Tip: You can make commands and output easier to read by marking it as preformatted text (Ctrl-E) in the editor - or enclosing them in three back ticks.

Yes the https server looks fine
Debug output while starting the server

2021/10/26 17:50:17 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone.exe" "serve" "http" "--config" "rclone.conf Path" "mylocal:C:\" "--cert" "Path" "--key" "Path" "--client-ca" "Path" "--user" "testpc" "--pass" "Welcome@123" "--addr" "192.168.0.108:8081" "-vv"]
2021/10/26 17:50:17 DEBUG : Creating backend with remote "mylocal:C:\"
2021/10/26 17:50:17 DEBUG : Using config file from "rclone.conf path"
2021/10/26 17:50:17 DEBUG : fs cache: renaming cache item "mylocal:C:\" to be canonical "mylocal://?/C:/"
2021/10/26 17:50:17 INFO : Local file system at //?/C:/: poll-interval is not supported by this remote
2021/10/26 17:50:17 INFO : Using --user testpc --pass XXXX as authenticated user
2021/10/26 17:50:55 INFO : /: 192.168.0.108:51921: Unauthorized request from
2021/10/26 17:51:13 INFO : : 192.168.0.108:51921: Serving directory
2021/10/26 17:51:14 INFO : favicon.ico: 192.168.0.108:51921: File not found
2021/10/26 17:51:17 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 1m0.2s

Strange, that doesn't match the error message from the client.

You need somebody more experienced in rclone https serve and backend to guide you from here.

@Animosity022 Do you have any ideas?

It looks like a bug introduced from 1.56 onward.

1.55.1 works fine as it takes the SSL parameters:

felix@gemini:~/rclone-v1.55.1-linux-amd64$ ./rclone serve http /tmp --addr :8686 -vv --cert `pwd`/blah.pem --key `pwd`/apps.animosity.us.key
2021/10/26 10:33:07 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/10/26 10:33:07 DEBUG : rclone: Version "v1.55.1" starting with parameters ["./rclone" "serve" "http" "/tmp" "--addr" ":8686" "-vv" "--cert" "/home/felix/rclone-v1.55.1-linux-amd64/blah.pem" "--key" "/home/felix/rclone-v1.55.1-linux-amd64/apps.animosity.us.key"]
2021/10/26 10:33:07 DEBUG : Creating backend with remote "/tmp"
2021/10/26 10:33:07 INFO  : Local file system at /tmp: poll-interval is not supported by this remote
2021/10/26 10:33:07 NOTICE: Local file system at /tmp: Serving on https://0.0.0.0:8686/

and the new version doesn't take it:

felix@gemini:~$ rclone serve http /tmp --addr :8686 -vv --cert `pwd`/blah.pem --key `pwd`/apps.animosity.us.key
2021/10/26 10:34:17 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/10/26 10:34:17 DEBUG : Setting --user-agent "animosityapp" from environment variable RCLONE_USER_AGENT="animosityapp"
2021/10/26 10:34:17 DEBUG : Setting --rc-user "felix" from environment variable RCLONE_RC_USER="felix"
2021/10/26 10:34:17 DEBUG : Setting --rc-pass "felix" from environment variable RCLONE_RC_PASS="felix"
2021/10/26 10:34:17 DEBUG : Setting default for drive-pacer-min-sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
2021/10/26 10:34:17 DEBUG : Setting default for drive-pacer-burst="1000" from environment variable RCLONE_DRIVE_PACER_BURST
2021/10/26 10:34:17 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "serve" "http" "/tmp" "--addr" ":8686" "-vv" "--cert" "/home/felix/blah.pem" "--key" "/home/felix/apps.animosity.us.key"]
2021/10/26 10:34:17 DEBUG : Creating backend with remote "/tmp"
2021/10/26 10:34:17 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/10/26 10:34:17 INFO  : Local file system at /tmp: poll-interval is not supported by this remote

If you can submit a bug report on github, they can take a look at it. Just reference the forum post.

Confirm. We experienced a number of http serving bugs after 1.56.0. This obviously might be another one. A bug report is welcome.

@innovate-invent Could you take a look?

Yep, I relied a little too heavily on the existing tests...

A fix is here: lib/http: Fix handling of ssl credentials by innovate-invent · Pull Request #5761 · rclone/rclone · GitHub

3 Likes

@iinda
Please try the beta with Nolan's fix applied.

Will it fix your problem? If yes, we'll merge the patch asap.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.