Today, we are using Rclone for file transfers from Unix Machine to Windows. This is how the Rclone transfer works for our application.
"Installed Rclone Client copies the files from Source Server (RedHAT_Linux) ----> Intermediate Server (Redhat_Linux) ---> Rclone client installed on Windows machine pulls the files from Intermediate Server."
SSH_Connections Allowed on **Intermediate** Server: 1200
CPU: 4-Core &16G Memory
Problem:
During file transfer CPU consumption is 100% and all SSH connections are closed on Intermediate server and does not allow/accept any new connections. Is there any tuning that needs to be done w.r.t Rclone perspective on Source and destination servers where Rclone client is installed on it?
Note: Intermediate server is a plain Unix machine & does not had Rclone installed in it.
What is your rclone version (output from rclone version)
rclone v1.49.5
- os/arch: linux/amd64
- go version: go1.12.10
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Source Server: RHEL Server 6.10 (where Rclone is installed )
Intermediate Server: RedHat 7.8
Destination : Windows Server 2012 (Rclone installed on it )
Which cloud storage system are you using? (eg Google Drive)
N/A
The command you were trying to run (eg rclone copy /tmp remote:tmp)
If latest version does not solve it, what process names does top say are saturating the processor?
Is it sshd, md5sum or sha1sum instances?
Is the rclone command one of a few large transfers or many small transfers sent as separate rclone commands?
If you login from the first server to second with ssh using the sftp credentials rclone is using do you get message of the day or landscape-sysinfo type output?
For frequent, small automated sftp copies, processes triggered at login from .bashrc or similar are a great way to cane the processor.
Thanks @asdffdsa your suggestions has already made a huge difference. I don't even notice a major change in CPU usage after configuring these. Transfer took almost the same time without the performance penalty.
I set the following if anyone else bumps into the same.
--transfers=2
--checkers=4
--bwlimit=10m
The VPS we use runs 8GB Ram and 2vCPU's + 100mbps link.
@ncw Thank you for that insight - I'll look into that. I'm not quite sure if I totally want to omit checksums. But I understand this probably takes a lot of the CPU time in the transfer.