when doing https pulls with https and a certificate. cert is valid wildcard cert from verisign. when using non certificate and key data moves as fast as expected. ~400 megabits/s when using cert and key getting about 40 megabits/s
Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
os/version: Microsoft Windows Server 2016 Standard 1607 (64 bit)
os/kernel: 10.0.14393.4886 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.17.2
go/linking: dynamic
go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Local Storage
The command you were trying to run (eg rclone copy /tmp remote:tmp)
2022/02/07 16:34:00 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\rclone\\rclone.exe" "-vv" "serve" "http" "ToEpic:\\\\sgcifs005_dp\\test\\MVDCVM01RTD021\\CAB_PRD_Stage\\FULL" "--addr" "10.150.71.30:8443" "--cert" "d:\\rclone\\StarChsliFullPEM.pem" "--key" "d:\\rclone\\Rclone.key" "--htpasswd" "d:\\rclone\\htpasswd" "--progress" "--log-file" "d:\\rclone\\log.txt"]
2022/02/07 16:34:00 DEBUG : Creating backend with remote "ToEpic:\\\\sgcifs005_dp\\test\\MVDCVM01RTD021\\CAB_PRD_Stage\\FULL"
2022/02/07 16:34:00 DEBUG : Using config file from "C:\\Users\\jlamar02\\AppData\\Roaming\\rclone\\rclone.conf"
2022/02/07 16:34:00 DEBUG : fs cache: renaming cache item "ToEpic:\\\\sgcifs005_dp\\test\\MVDCVM01RTD021\\CAB_PRD_Stage\\FULL" to be canonical "ToEpic://?/UNC/sgcifs005_dp/test/MVDCVM01RTD021/CAB_PRD_Stage/FULL"
2022/02/07 16:34:00 INFO : Local file system at //?/UNC/sgcifs005_dp/test/MVDCVM01RTD021/CAB_PRD_Stage/FULL: poll-interval is not supported by this remote
2022/02/07 16:34:00 INFO : Using "d:\\rclone\\htpasswd" as htpasswd storage
So what you are saying is that rclone serve http runs much slower when using https rather than http?
I did a quick test here and on my 4 year old Linux laptop, I get https at 310 MiB/s but http at 977 MiB/s which is much much faster than you are getting, but still shows a difference.
This will be to do with the extra CPU load needed for the https connections. Can you throw more CPU at the VM? (assuming it is a VM)
Note also that some ciphers are more efficient than others so tweaking which cipher is in use can help.
thanks nick. that is correct when i add the --cert and --key to the command transfers to my remote location is drastically slower than without the cert.
my vm currently has 8 vcpu currently. and utilization is never above 20%
my cert is from digicert rsa sha 256
I added the chain to the pem perfectly and looks great when connecting through the browser.
also note that i just switched this to a 16 vcpu server and it drastically improved. still using less than 20% of cpu. i would expect if it was cpu bound the processing would be a lot higher
@ncw, we've been doing all the downloads with rclone. The slow speeds with https vs http (~400 Mbps vs ~40 Mbps) were the numbers reported out of rclone
I don't think the CPU can explain the difference entirely. I suspect something to do with the networking stack on Windows.
Can you try a Linux server and see if that is faster? 400 Mbps or 40 Mbps both sound quite slow - rclone should be pumping over 1 Gbit/s for both cases.
I'm not sure I have a Linux box available anywhere, but will see if I can get one for testing. Are you thinking we should try running a Linux server, client, or both?