Date: Sun, 20 Jul 2014 15:26:02 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Jeremy Chadwick" <jdc@koitsu.org>, <freebsd-stable@freebsd.org> Subject: Re: Consistently "high" CPU load on 10.0-STABLE Message-ID: <97EA8E571E634DBBAA70F7AA7F0DE97C@multiplay.co.uk> References: <20140720062413.GA56318@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
If you add -H -z to your top command does anything stand out?
Regards
Steve
----- Original Message -----
From: "Jeremy Chadwick" <jdc@koitsu.org>
To: <freebsd-stable@freebsd.org>
Sent: Sunday, July 20, 2014 7:24 AM
Subject: Consistently "high" CPU load on 10.0-STABLE
> (Please keep me CC'd as I'm not subscribed to freebsd-stable@)
>
> Today I took the liberty of upgrading my main home server from
> 9.3-STABLE (r268785) to 10.0-STABLE (r268894). The upgrade consisted of
> doing a fresh install of 10.0-STABLE on a brand new unused SSD. Most
> everything went as planned, barring a couple ports-related anomalies,
> and I seemed fairly impressed by the fact that buildworld times had
> dropped to 27 minutes and buildkernel to 4 minutes with clang (something
> I'd been avoiding like the plague for a long while). Kudos.
>
> But after an hour or so, I noticed a consistent (i.e. reproducible)
> trend: the system load average tends to hang around 0.10 to 0.15 all the
> time. There are times where the load drops to 0.03 or 0.04 but then
> something kicks it back up to 0.15 or 0.20 and then it slowly levels out
> again (over the course of a few minutes) then repeats.
>
> Obviously this is normal behaviour for a system when something is going
> on periodically. So I figured it might have been a userland process
> behaving differently under 10.x than 9.x. I let top -a -S -s 1 run and
> paid very very close attention to it for several minutes. Nothing. It
> doesn't appear to be something userland -- it appears to be something
> kernel-level, but nothing in top -S shows up as taking up any CPU time
> other than "[idle]" so I have no idea what might be doing it.
>
> The box isn't doing anything like routing network traffic/NAT, it's pure
> IPv4 (IPv6 disabled in world and kernel, and my home network does
> basically no IPv6) and sits idle most of the time fetching mail. It
> does use ZFS, but not for /, swap, /var, /tmp, or /usr.
>
> vmstat -i doesn't particularly show anything awful. All the cpuX:timer
> entries tend to fluctuate in rate, usually 120-200 or so; I'd expect an
> interrupt storm to be showing something in the 1000+ range.
>
> The only thing I can think of is the fact that the SSD being used has no
> 4K quirk entry in the kernel (and its ATA IDENTIFY responds with 512
> logical, 512 physical, even though we know it's 4K). The partitions are
> all 1MB-aligned regardless.
>
> This is all bare-metal, by the way -- no virtualisation involved.
>
> I do have DTrace enabled/built on this box but I have absolutely no clue
> how to go about profiling things. For example maybe output of this sort
> would be helpful (but I've no idea how to get it):
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2014-July/079276.html
>
> I'm certain I didn't see this behaviour in 9.x so I'd be happy to try
> and track it down if I had a little bit of hand-holding.
>
> I've put all the things I can think of that might be relevant to "system
> config/tuning bits" up here:
>
> http://jdc.koitsu.org/freebsd/releng10_perf_issue/
>
> I should note my kernel config is slightly inaccurate (I've removed some
> stuff from the file in attempt to rebuild, but building world prior to
> kernel failed due to r268896 breaking world, but anyone subscribed here
> has already seen the Jenkins job of that ;-) ).
>
> Thanks.
>
> --
> | Jeremy Chadwick jdc@koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?97EA8E571E634DBBAA70F7AA7F0DE97C>