Very high CPU usage on moving data on mounted SFTP remove

Help and Support
1

What is the problem you are having with rclone?

Copying data to a SFTP remote via rclone mount commant, takes 100% of cpu usage. It breaks everything else in the server. The main usecase for me is to transfer big video files (10-20GB) from this SFTP remote to my Onedrive remote.
The onedrive remote sometimes has high cpu usage but it doesn't go over 50%.

My server is highly performant.

is there any config I can add or remove which would make it use less cpu?

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-78-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/local/bin/rclone mount seedbox:/var/lib/deluge/Downloads/ /mnt/seedbox/ \
    --config=/root/.config/rclone/rclone.conf \
    --allow-other \
    --umask 002 \
    --vfs-cache-mode full \
    --vfs-cache-max-size 100G \
    --vfs-read-chunk-size 512M \
    --vfs-cache-max-age 9999h \
    --tpslimit 50 \
    --log-level INFO \
    --log-file /var/log/rclone/rclone-seedbox.log \
    --bwlimit-file 100M

The rclone config contents with secrets removed.

[seedbox]
type = sftp
host = myhost
user = myuser
key_file = /root/.ssh/id_rsa
key_use_agent = false
use_insecure_cipher = true
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
2

To? From? Both?

Not clear at all what you are doing. Can you show example of command you are using when you see high CPU usage?

3

I mount rclone SFTP to /mnt/sftp

Whenever I copy anything to and from /mnt/sftp, the CPU usage goes to 150%.

Not sure whether you are looking for more info.

The log file doesn't show anything for this mount.

4

Try to remove this - old ciphers can be very slow without hardware acceleration.

5

I deleted it from the rclone config and restarted the mount.
Same problem. CPU usage is high.

6

Try some other SFTP client - e.g. FileZilla to see whether problem is with rclone or SFTP and your machine performance in general.

7

Not sure if I get your point. These are 2 different things. I cannot mount with Filezilla.

This setup was working fine before I deleted the rclone mount command for SFTP and I cannot remember how I had it. It worked fine. Maybe the dev can provide some feedback on which parameters might increase the cpu usage.

8

correct. But you can use FileZilla to e.g. download one of this big files. If it also gets 100%+ CPU usage then problem is not with rclone/mount but with SFTP performance in your specific configuration. If not then it indeed can be rclone issue.

9

For mount itself you can also add:

--vfs-fast-fingerprint

It should improve performance but I would not expect miracle

10

This is on an ubuntu server. I cannot use Filezilla. Also I believe even if I could, Filezilla is doing a simple download. I don't think it would give any feedback.

I will try your other recommentation and report back.

Thank you! :v:

11

Does not have to be FileZilla.

You can just use cmd sftp command.

The purpose is to see what is CPU load when running simple sftp transfer from your seedbox

12

hi,

might check out my howto guide, very simple way to move files from seedbox to cloud.
https://forum.rclone.org/t/how-to-move-a-downloaded-torrent-to-the-cloud/23493

does the seedbox offer any modern protocols such as webdav?

1 Like
13

This didn't do anything. I still see quite a lot of CPU usage 100%+.

14

My rclone is in a different server than my deluge instance. I would prefer to keep it this way.

My seedbox is a VPS where I installed Deluge myself.

15

SFTP protocol is quite CPU intensive - this is how it is.

You can try to throttle it down. Start with for example:

--transfers 1
--tpslimit 5

And tweak based on what CPU usage your machine can afford.

Also you can try to increase sftp-chunk-size.

There is no known issue with CPU usage and rclone so what you experience is your specific system limits or configuration.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.