High CPU Usage Copying data to Mounted Drive

Help and Support
1

What is the problem you are having with rclone?

Hope everyone is doing well!
I have been using rclone for a week now and it's been a really good alternative to what I've been using before (google drive's app).
However, I have been facing High CPU spikes when any file is being copied to my mounted drive (Hetzner Storage Box).

The command I'm using:
rclone mount --vfs-cache-mode full backup: O:

Why using mount instead of rclone sync/rclone copy?

I need to keep data in sync with my cloud 100% of the time, so running a backup daily or manually is not an option.
I have achieved the same behavior as google drive using DSynchronize + rclone mounting the drive.
DSynchronize watches my seleced folders and then syncs the changes to the drive.
But the CPU spikes also happen if I manually copy files or if I use robocopy.
This solution is almost perfect, but this high cpu usage is driving me insane.
I have tried other rclone versions, but the same happens.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0

  • os/version: Microsoft Windows 10 Pro 1809 (64 bit)
  • os/kernel: 10.0.17763.1282 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.18.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Hetzner Storage Box

The rclone config contents with secrets removed.

[backup]
type = sftp
host = xxxxxxxx
user = xxxxxxxxx
port = 23
pass = xxxxxxxxx
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
2

Is the CPU spike when the files are being uploaded?

You could try setting a --bwlimit to slow the uploads?

1 Like
3

Hello! Thanks a lot for the answer!
Yes, only when files are uploaded - If not, rclone runs just fine on the background.
I'll try to use this flag and report back.
Thanks a lot!

1 Like
4

Hello again and thanks for the patience on this!
I have made extensive tests over the weekend, and it seems that nothing will change this strange behavior.
Now I've noticed that even without cache, there will be CPU spikes when copying.
This is strange, and I have explored many other options and none of them behave like that.
I suspect I may be doing something wrong, but I am not quite sure.
The problem with the CPU spikes is that it will cause throttle. My mouse will freeze for some milliseconds and my audio will crack absurdly.
If I use windows to map it as a network drive or powershell using net use, the files are copied without any issues.
I really am not sure what else should I look for, but I would really appreciate if you could give me a hint.
Thanks a lot once again!

5

What are the specs on the box?

What's a spike? 100%? What's going on when that happens? Can you share a debug log file? Can you share some server stats to show what's going on and explain what was happening at that time?

Why do you think it's rclone?

1 Like
6

Thanks for the answer!
Yes, 100% usage. It stays around 50% to 60% the whole time but then it jumps to 100% and gets back to 50%/60%.
image

It happens whenever I copy something to the storage box.

Debug:
https://zerobin.net/?bdcda8b816b720f2#jc8lc+Zo/l9hDkBc7AvSiKZkEFHBgqGpGT3S5ySiRgs=
(pastebin was unreachable here)

Server-side is not available because I'm using Hetzner Storage Box, therefore I have no control on anything server related.

I think it's related to rclone because other tools work fine: Air Live Drive, RaiDrive, Windows Map...
But since rclone is much faster, I wish it worked fine :slight_smile:

Thanks again!

7

^^^^

It sounds like it is working fine as if it's going faster, it's going to consume more CPU in general as you doing more.

2022/08/08 12:44:12 ERROR : a/File.mp4: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes

That's going to create some issues without that turned on. Otherwise, that looks like a normal rclone log without anything strange that I can see.

8

Thanks for the quick answer!
I'm sorry, but I really don't think there's a way to see any specs of the box?
It's Hetzner Storage Box BX11, 1tb storage and that's it I think...

I have tried writes/full. Nothing changes, unfortunately.

It sounds like it is working fine as if it's going faster, it's going to consume more CPU in general as you doing more.

The problem is that it renders my computer unusable while the file transfer happens.

9

From looking at the price, I'd imagine that's some very underpowered VPS so you are just taxing it.

For ~$3-4 bucks per month, that's probably the best you can do.

If rclone is too fast, you could possibly tune it down, but not sure you'll see much difference I'd imagine.

In perspective, I run 2 rclone mounts with encryption and generally they hover at a few % of CPU but I have a recent beefy CPU:

Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
10

hi,
i have a hetnzer bx11 that i use with rclone mount, never had an issue with high cpu.
on windows, on hetzner vm.

11

Mine is a i5 2500k + 8gb RAM.
While I understand it's not the fastest one, it is used to heavy cpu load tasks - I am a songwriter and I use lots of VST's/Virtual instruments.

I'll keep on looking!

12

Can you please share your mount settings?
Maybe that can give me a hint!

13

if you are using rclone mount on windows, then you need to use winfsp.
somthing that you have to download and install.

as for settings, just a generic mount values.
sometimes using vfs file cache, sometimes without using vfs file cache.

on the storage box,, i store media files in a crypt remote.
i use emby/jellyfin to stream.

perhaps try without cache
rclone mount backup: O:

14

Win FSP?
I don't think I'm using it?

15

please see my last post.

16

Just saw the update.
Yes, I am using the last version of winfsp (I've installed many programs lately, that's why I was not sure).
I have tried mounting with and without cache, but the behavior is the same.
It's so weird...
I'll keep digging!

17

as a test, try mounting to a folder, not to drive letter.

another test, add --network-mode

18

Ok, I have tested all the advices and I came to the conclusion that the issue lies within Hetzner, not rclone.
And the reason why I'm saying this is because I've tested google drive as well as Jotta Cloud and both of them had good results both on transfer speed and not having such a high CPU usage.
I have also tried connecting hetzner using other options (SFTP, for instance), and it remains exactly the same.
Thanks guys for your patience on this one!
Have a great day!

1 Like
19

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.