High cpu iowait

Help and Support
1

What is the problem you are having with rclone?

High CPU iowait

What is your rclone version (output from rclone version)

v.1.56.0

Which cloud storage system are you using? (eg Google Drive)

Google Drive, Google cloud services, Onedrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount gcloud: /services/gcloud --allow-other --dir-cache-time=24h --timeout=1h --vfs-cache-mode=off --vfs-cache-max-age=1h --vfs-read-chunk-size=64M --ignore-checksum --vfs-read-chunk-size-limit=2G --log-level=DEBUG --log-file=/tmp/gcloud.txt

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

2021/09/13 23:30:24 DEBUG : embyexterno/Series Biblicas/Génesis/Génesis/1x08-thumb.jpg: waiting for in-sequence read to 49152 for 20ms
2021/09/13 23:30:25 DEBUG : embyexterno/Series Biblicas/Génesis/Génesis/1x29-thumb.jpg: waiting for in-sequence read to 49152 for 20ms
2021/09/13 23:30:27 DEBUG : embyexterno/Series Biblicas/Génesis/Génesis/1x80-thumb.jpg: waiting for in-sequence read to 49152 for 20ms
2021/09/13 23:30:27 DEBUG : embyexterno/Series Biblicas/Génesis/Génesis/1x84-thumb.jpg: waiting for in-sequence read to 49152 for 20ms
2021/09/13 23:30:27 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 152207360 for 20ms
2021/09/13 23:30:37 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 153255936 for 20ms
2021/09/13 23:30:43 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 153518080 for 20ms
2021/09/13 23:31:05 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 155353088 for 20ms
2021/09/13 23:31:10 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 155615232 for 20ms
2021/09/13 23:31:21 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 156663808 for 20ms
2021/09/13 23:32:05 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 161382400 for 20ms
2021/09/13 23:32:10 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 161644544 for 20ms
2021/09/13 23:32:30 DEBUG : embyexterno/Series Biblicas/Moisés y los 10 mandamientos/S01E226.mp4: waiting for in-sequence read to 163217408 for 20ms
2

I have tried with --vfs-cache-mode=writes and full. And I got the same result

3

IOWait is normal with cloud storage as you are download things remotely.

You didn't share the output of rclone version so I'm not sure what OS are you running.
You didn't share the rclone.conf so I don't know what your config looks like.

You only included a very small snippet of the log so it's hard to tell since it isn't a full log either.

4

My bad. Sorry, this is my first time posting.

This is my output:

rclone v1.56.0

  • os/version: debian 10.10 (64 bit)
  • os/kernel: 4.19.0-17-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.5
  • go/linking: static
  • go/tags: none

gcloud (1).service.txt (536 Bytes)

I used the --async-read=false flag, this reduce the iowait but the cpu consume is still high.

I have attached the log and the rclone.conf

Processing: gcloud (1).service...

gcloud.txt (875.0 KB)

5

Everything I see there looks very normal.

If you are using cloud storage, depending on your internet speeds, you get IOWait since it's not local storage.

What is high?

You can see mine jumps when I'm uploading and at times when streaming is happening:

image
image1622×348 22.6 KB

It just means the CPU is waiting for the IO, which normally happens very quickly but with cloud storage the times are much longer.

6

Thank you.

Do you recommend the --async-read=false flag?

About internet speed, Im testing this with Google Cloud, OVH and Kamatera vps. They have good internet connection. Do I have to limit the rclone transactions with --bwlimit or this high speed internet wont be a problem?

In your opinion, what is the better cloud for video streaming?

7

I have my own service file documented for me as it's tailored for gigabit FIOS and a home server.

I find for my use case, having a home server that I have control over is much better for my use so I wouldn't use a cloud based solution as I already have gigabit FIOS.

async false tends to be slower in most cases so I leave it as the default of on. Most recent kernels work great with it as that's how they operate.

We're almost on the same version as I'm running Debian 11 currently as I find it's more stable than Ubuntu for me.

rclone v1.56.0
- os/version: debian 11.0 (64 bit)
- os/kernel: 5.10.0-8-amd64 (x86_64)

That's been my recent testing anyway.

8

Thank you @Animosity022

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.