Slow pulls using http with certificate

What is the problem you are having with rclone?

when doing https pulls with https and a certificate. cert is valid wildcard cert from verisign. when using non certificate and key data moves as fast as expected. ~400 megabits/s when using cert and key getting about 40 megabits/s

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: Microsoft Windows Server 2016 Standard 1607 (64 bit)
  • os/kernel: 10.0.14393.4886 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

D:\rclone\rclone.exe -vv serve http ToEpic:\\sgcifs005_dp\test\MVDCVM01RTD021\CAB_PRD_Stage\FULL --addr --cert d:\rclone\StarChsliFullPEM.pem --key d:\rclone\Rclone.key --htpasswd d:\rclone\htpasswd --progress

D:\rclone\rclone.exe -vv serve http ToEpic:\\sgcifs005_dp\test\MVDCVM01RTD021\CAB_PRD_Stage\FULL --addr --htpasswd d:\rclone\htpasswd --progress```

The rclone config contents with secrets removed.

type = local

A log from the command with the -vv flag

2022/02/07 16:34:00 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\rclone\\rclone.exe" "-vv" "serve" "http" "ToEpic:\\\\sgcifs005_dp\\test\\MVDCVM01RTD021\\CAB_PRD_Stage\\FULL" "--addr" "" "--cert" "d:\\rclone\\StarChsliFullPEM.pem" "--key" "d:\\rclone\\Rclone.key" "--htpasswd" "d:\\rclone\\htpasswd" "--progress" "--log-file" "d:\\rclone\\log.txt"]
2022/02/07 16:34:00 DEBUG : Creating backend with remote "ToEpic:\\\\sgcifs005_dp\\test\\MVDCVM01RTD021\\CAB_PRD_Stage\\FULL"
2022/02/07 16:34:00 DEBUG : Using config file from "C:\\Users\\jlamar02\\AppData\\Roaming\\rclone\\rclone.conf"
2022/02/07 16:34:00 DEBUG : fs cache: renaming cache item "ToEpic:\\\\sgcifs005_dp\\test\\MVDCVM01RTD021\\CAB_PRD_Stage\\FULL" to be canonical "ToEpic://?/UNC/sgcifs005_dp/test/MVDCVM01RTD021/CAB_PRD_Stage/FULL"
2022/02/07 16:34:00 INFO  : Local file system at //?/UNC/sgcifs005_dp/test/MVDCVM01RTD021/CAB_PRD_Stage/FULL: poll-interval is not supported by this remote
2022/02/07 16:34:00 INFO  : Using "d:\\rclone\\htpasswd" as htpasswd storage

So what you are saying is that rclone serve http runs much slower when using https rather than http?

I did a quick test here and on my 4 year old Linux laptop, I get https at 310 MiB/s but http at 977 MiB/s which is much much faster than you are getting, but still shows a difference.

This will be to do with the extra CPU load needed for the https connections. Can you throw more CPU at the VM? (assuming it is a VM)

Note also that some ciphers are more efficient than others so tweaking which cipher is in use can help.

thanks nick. that is correct when i add the --cert and --key to the command transfers to my remote location is drastically slower than without the cert.
my vm currently has 8 vcpu currently. and utilization is never above 20%
my cert is from digicert rsa sha 256
I added the chain to the pem perfectly and looks great when connecting through the browser.

also note that i just switched this to a 16 vcpu server and it drastically improved. still using less than 20% of cpu. i would expect if it was cpu bound the processing would be a lot higher

How are you measuring the speed? Can you try measuring it with rclone?

I am looking at the nic on the server to see the look at the throughput. fyi these are 10GB nics

I meant how are you downloading the files? Have you tried with curl and or rclone?

@ncw, we've been doing all the downloads with rclone. The slow speeds with https vs http (~400 Mbps vs ~40 Mbps) were the numbers reported out of rclone

I don't think the CPU can explain the difference entirely. I suspect something to do with the networking stack on Windows.

Can you try a Linux server and see if that is faster? 400 Mbps or 40 Mbps both sound quite slow - rclone should be pumping over 1 Gbit/s for both cases.

I'm not sure I have a Linux box available anywhere, but will see if I can get one for testing. Are you thinking we should try running a Linux server, client, or both?


--- do you have the results of a speedtest on the host and on the vm?

--- this has solved slow windows networking speeds with rclone, might help you too.

--- if a linux machine is not handy, might do initial testing on wsl.

Or on macOS - that would do as well.

thanks for the help. there is a high chance we will be abandoning this to use storage based replication. i will let you know in the next couple days.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.