Rclone on Mac OSX: "caught waking the CPU" messages

durval · June 14, 2019, 5:38pm

Hello everyone,

As kindly requested by @Animosity022, here it comes:

Just got this kernel message here on dmesg:
process rclone[91967] caught waking the CPU 45001 times over ~253 seconds, averaging 177 wakes / second and violating a limit of 45000 wakes over 300 seconds.

As in the older topic where it was first mentioned, here it's also on Mac OSX High Sierra but with rclone serve restic instead of rclone mount .

When I check on the process with ps auwx , it has not been killed by the OS and indeed seems to be happily running along:
root 91967 0.0 0.1 4557216 86048 ?? S 9:17PM 2:11.65 rclone serve restic --stdio --b2-hard-delete --drive-use-trash=false --rc --rc-addr localhost:5582 --rc-no-auth REDACTED1:REDACTED2/REDACTED3

In fact, except for some slowness (which I attribute to restic and not to rclone ), the underlying restic backup operation seem to be progressing normally -- I just detected that message and wondered what the heck it's about, and then searched and found this topic.

Does anyone know what the heck it means? I'm far from being an expert on OSX... :-/

Cheers,
-- Durval.

blim5001 · June 15, 2019, 12:15pm

I know you pinged me in that old thread. But afraid I really cannot remember how or if I solved it. (and am not running High Sierra any more)

For me the problem only seemed to happen when copying files into the mount using OSX rather than rclone copy, which is what I use now for copying stuff from OSX

I have not used rclone serve so am not really familiar with it. Does it create a mount, so is it using osxfuse. if it is, is your osxfuse up to date?

I think back then I did solve some issues by building rclone from source on OSX, but not sure if that would make any difference in your case.

ncw · June 15, 2019, 1:33pm

I found that message in the darwin kernel source

github.com

opendtrace/xnu/blob/master/osfmk/kern/task.c#L5256


	/*
	 * Disable the exception notification so we don't overwhelm
	 * the listener with an endless stream of redundant exceptions.
	 * TODO: detect whether another thread is already reporting the violation.
	 */
	uint32_t flags = WAKEMON_DISABLE;
	task_wakeups_monitor_ctl(task, &flags, NULL);


	fatal = task->rusage_cpu_flags & TASK_RUSECPU_FLAGS_FATAL_WAKEUPSMON;
	trace_resource_violation(RMON_CPUWAKES_VIOLATED, &lei);
	printf("process %s[%d] caught waking the CPU %llu times "
	       "over ~%llu seconds, averaging %llu wakes / second and "
	       "violating a %slimit of %llu wakes over %llu seconds.\n",
	       procname, pid,
	       lei.lei_balance, lei.lei_last_refill / NSEC_PER_SEC,
		   lei.lei_last_refill == 0 ? 0 :
				(NSEC_PER_SEC * lei.lei_balance / lei.lei_last_refill),
	       fatal ? "FATAL " : "",
		   lei.lei_limit, lei.lei_refill_period / NSEC_PER_SEC);


	kr = send_resource_violation(send_cpu_wakes_violation, task, &lei,

It appears to be a resource violation notice.

Looking further down that function it can be fatal or not depending on whether TASK_RUSECPU_FLAGS_FATAL_WAKEUPSMON is set, but I don't know what that is!

The message seems not to be very well documented, but I think as long as the kernel didn't kill rclone you can ignore it.

durval · June 15, 2019, 5:59pm

Great job as usual, @ncw -- thank you very much!

I agree with your analysis: despite the message's ominous tone, the Darwin kernel source you dug up seems to indicate it's just a 'warning' and not an 'error' and so I'm proceeding by suitably ignoring it

BTW: I just re-checked and all the "power-saving" configs on the machine are turned off, so I don't think it has anything to do with PM or similar.

Cheers,
-- Durval.

durval · June 15, 2019, 6:12pm

Howdy @blim5001,

Not a problem, and thank you for coming back and replying.

rclone serve restic is a special mode where it acts as a "backend server" for restic, answering to commands from the latter up and down a pair of pipes to manipulate the backend storage accordingly to restic needs. This resulted from a great idea by @ncw, if you are curious you can read more about it here: https://restic.net/blog/2018-04-01/rclone-backend.

BTW, if you have lots of small files and is hitting your head against GDrive transactions/second limit, or is in any way getting slow backup speeds from rclone alone, I can't speak highly enough of the restic+rclone combo -- it literally made possible my backing up to GDrive of a 70M files / 33TB server which would have been unfeasible otherwise. And fd0, the restic developer, is a great guy and very willing to help -- restic is not as perfect as rclone is (there are a few issues regarding memory use, performance, detection of changed files etc) but it's getting there!

Cheers,
-- Durval.

system · September 13, 2019, 6:12pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.