CPU idle under (<99%) using rclone

[ninjadark]

What is the problem you are having with rclone?

I have a program that is responsible for streaming videos which are stored in Dropbox and are mounted the unit using rclone on a dedicated server and with "full" cache. The problem is that the CPU idle is getting too low, so it is causing the CPU idle to reach 0% which is not good.

There is no transcoding by my program, since the only thing it does is to read the file and send it to download, so it is just a direct pass, and it has a dedicated ssd for the rclone cache.

An example would be how my program works is that you mount a unit in a directory and expose it by http and many people download the files, well basically it's just that with more security.

To take into account at the time the photo was taken there were approximately 250 connections, so it is quite little, however you can see the idle as there is already a consumption.
When there are more than 1000 connections is when the idle cpu sometimes drops to 65% - 45% - 25%.
I have 4 servers with the same configuration and the same thing happens to all of them.

My Server

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

• os/version: ubuntu 20.04 (64 bit)
• os/kernel: 5.4.0-169-generic (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.21.4
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

It is in a service and this is the command

[Unit]
Description=Rclone (dropbox)
AssertPathIsDirectory=/media/dropbox-series
After=lighttpd.service

[Service]
Type=simple
ExecStart=/usr/bin/rclone mount dropbox-series: /media/dropbox-series \
--config=/root/.config/rclone/rclone.conf \
--log-level INFO --log-file "/ssd/log/dropbox-series.log" \
--allow-other \
--use-mmap \
--dir-cache-time 168h \
--poll-interval 10s \
--cache-dir=/ssd/rclone-cache \
--tpslimit 14 \
--timeout 30m \
--vfs-cache-mode full \
--vfs-cache-max-age 96h \
--vfs-cache-poll-interval 5m \
--vfs-cache-max-size 500G
ExecStop=/bin/fusermount -zu /media/dropbox-series
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[dropbox-series]
type = dropbox
client_id = XXXXX
client_secret = XXXXX
token = XXXXX

A log from the command that you were trying to run with the -vv flag

I put it in debug mode and put it in this page the log

Paste.gd: Copy and Paste Text Tool | Toptal®

[kapitainsky]

@ninjadark

The problem is that the CPU idle is getting too low, so it is causing the CPU idle to reach 0% which is not good.

Why it is not good? What you expect from rclone? To run some extra pointless jobs to satisfy your desire to see CPU load?

[ninjadark]

I don't understand, what do you mean? I have simply seen that the cpu idle drops considerably when I raise rclone and there are a lot of incoming connections. I don't know if there is a way to fix it, because when it is around 250 connections approximately it works perfectly, only when it reaches 1000 conx the idle of the cpu dies.

[Animosity022]

Can you post some more details of the system statistics when you have a 1000 connections?

Your screenshot for 250 is like 7% non idle, which is very low load.

If you times 4 your connections, it shouldn't expoentially increaese.

[ninjadark]

Unfortunately it is not peak time now, but when there are approximately 1000 connections the cpu idle starts to hover around 60 - 50 %.

[Animosity022]

That really doesn't seem too odd based on 1000 connections.

You'd have to share the some details though when the load ramps as that's a CPU back from 2019 and seems to be performing as I'd expect under that load.

If you want less CPU load, replace the CPU with something more current/powerful.

[ninjadark]

I will share my debug file and my top when I have 1000 connections again, however, as which cpu could you recommend for such a demand? Do you think more cores could combat this problem?
I'm all ears

[Animosity022]

Once you have an idea of what the higher performance needs are, you can use something like CPUMark to compare CPUs and see what meets your need.

I use that site a lot when I want to figure out what CPU I need to do a certain function once I benchmark it.

(I don't normally say this, but debug log probably won't add much here as you are just trying to figure out performance. I don't think there is any bug/defect that I can tell offhand)