From owner-freebsd-stable@FreeBSD.ORG Sun Jul 20 06:24:17 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 12135DA5 for ; Sun, 20 Jul 2014 06:24:17 +0000 (UTC) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:44:76:96:27:227]) by mx1.freebsd.org (Postfix) with ESMTP id E900F263C for ; Sun, 20 Jul 2014 06:24:16 +0000 (UTC) Received: from omta14.emeryville.ca.mail.comcast.net ([76.96.30.60]) by qmta12.emeryville.ca.mail.comcast.net with comcast id UWP11o0011HpZEs01WQEL1; Sun, 20 Jul 2014 06:24:14 +0000 Received: from jdc.koitsu.org ([69.181.136.108]) by omta14.emeryville.ca.mail.comcast.net with comcast id UWQD1o00D2LW5AV8aWQEFs; Sun, 20 Jul 2014 06:24:14 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 844C61744B1; Sat, 19 Jul 2014 23:24:13 -0700 (PDT) Date: Sat, 19 Jul 2014 23:24:13 -0700 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Subject: Consistently "high" CPU load on 10.0-STABLE Message-ID: <20140720062413.GA56318@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20140121; t=1405837454; bh=0WNquybfF878xTsYmes1tiSw3V7Ekm6DTPPv/0RQ43E=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=BMnmczl8zNYnl0DUQfszGE0yvX1EuHVgZqdRI4by6G0GQ6FGn3pLBDY3Fj5qn+hiV vQGuezE1BAeUfa8j9RozXBI/pIwoeeKgk+U6rAqw7rUidrC8MP7kNOP5XKuGLG+ddh n9j09/Narn0Z70HKvsXTDUfDnDg6W97tQGR6+aLBX+2NACef3TYvDELG/OLkc/Um+D MJeQUqRa9kmQR8AFKltLH1FJ9TkQb7Ca+JWqMVA+21xod31rI9KAXkwTg+MuLqGYsa IrfJ59B+pQJ5B42clmAksNJN2EsVCCttkVGd1WU7t8oHbbMCMZ9DPqWxfb0NusTfBC fBY4SUAYPb1+A== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 06:24:17 -0000 (Please keep me CC'd as I'm not subscribed to freebsd-stable@) Today I took the liberty of upgrading my main home server from 9.3-STABLE (r268785) to 10.0-STABLE (r268894). The upgrade consisted of doing a fresh install of 10.0-STABLE on a brand new unused SSD. Most everything went as planned, barring a couple ports-related anomalies, and I seemed fairly impressed by the fact that buildworld times had dropped to 27 minutes and buildkernel to 4 minutes with clang (something I'd been avoiding like the plague for a long while). Kudos. But after an hour or so, I noticed a consistent (i.e. reproducible) trend: the system load average tends to hang around 0.10 to 0.15 all the time. There are times where the load drops to 0.03 or 0.04 but then something kicks it back up to 0.15 or 0.20 and then it slowly levels out again (over the course of a few minutes) then repeats. Obviously this is normal behaviour for a system when something is going on periodically. So I figured it might have been a userland process behaving differently under 10.x than 9.x. I let top -a -S -s 1 run and paid very very close attention to it for several minutes. Nothing. It doesn't appear to be something userland -- it appears to be something kernel-level, but nothing in top -S shows up as taking up any CPU time other than "[idle]" so I have no idea what might be doing it. The box isn't doing anything like routing network traffic/NAT, it's pure IPv4 (IPv6 disabled in world and kernel, and my home network does basically no IPv6) and sits idle most of the time fetching mail. It does use ZFS, but not for /, swap, /var, /tmp, or /usr. vmstat -i doesn't particularly show anything awful. All the cpuX:timer entries tend to fluctuate in rate, usually 120-200 or so; I'd expect an interrupt storm to be showing something in the 1000+ range. The only thing I can think of is the fact that the SSD being used has no 4K quirk entry in the kernel (and its ATA IDENTIFY responds with 512 logical, 512 physical, even though we know it's 4K). The partitions are all 1MB-aligned regardless. This is all bare-metal, by the way -- no virtualisation involved. I do have DTrace enabled/built on this box but I have absolutely no clue how to go about profiling things. For example maybe output of this sort would be helpful (but I've no idea how to get it): http://lists.freebsd.org/pipermail/freebsd-stable/2014-July/079276.html I'm certain I didn't see this behaviour in 9.x so I'd be happy to try and track it down if I had a little bit of hand-holding. I've put all the things I can think of that might be relevant to "system config/tuning bits" up here: http://jdc.koitsu.org/freebsd/releng10_perf_issue/ I should note my kernel config is slightly inaccurate (I've removed some stuff from the file in attempt to rebuild, but building world prior to kernel failed due to r268896 breaking world, but anyone subscribed here has already seen the Jenkins job of that ;-) ). Thanks. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |