From owner-freebsd-stable@FreeBSD.ORG Sun Jul 20 07:13:06 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2FE64AF for ; Sun, 20 Jul 2014 07:13:06 +0000 (UTC) Received: from mail-qg0-x231.google.com (mail-qg0-x231.google.com [IPv6:2607:f8b0:400d:c04::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9513C2987 for ; Sun, 20 Jul 2014 07:13:06 +0000 (UTC) Received: by mail-qg0-f49.google.com with SMTP id j107so4368978qga.22 for ; Sun, 20 Jul 2014 00:13:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=gqWrNWBvnmmLe7qiQbepGB3yG0d2u6QRaLlrvnnIGJQ=; b=W6Y2r7oRTM0mp/u+ZcLQBBDYkBhC6lbSOgrELFzx5DpmwrH8aPE6HkEo5xVN8DypQk 2Y/JfUDz6inHP07OaFu/WfByUYSy+e53CETW9CRtMBWpefEl71JmRVAvBB/j0aPvDtYA M1ODfhkicNyRvveTatwW5XM6899kSOQl39sO/mEjAMXmP+MAfALnton9eE0SrBvy4/pZ Z/TJqyw3qFMjN5MXVOCxOQQB7QPwG68mpDQpbO7OX95TGNnCee1mpODZRDCh/t0c4AlM eIP04t9ElVEIzB4GLJETYOixvy12ejPELR2nvKHmzWh9Uf2H15x712Xc2tQnqg1GCeNQ 8a9Q== MIME-Version: 1.0 X-Received: by 10.224.223.135 with SMTP id ik7mr27769899qab.26.1405840385637; Sun, 20 Jul 2014 00:13:05 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Sun, 20 Jul 2014 00:13:05 -0700 (PDT) In-Reply-To: <20140720062413.GA56318@icarus.home.lan> References: <20140720062413.GA56318@icarus.home.lan> Date: Sun, 20 Jul 2014 00:13:05 -0700 X-Google-Sender-Auth: XBQOwhb0vILcwqJaP6uEi5ol-FI Message-ID: Subject: Re: Consistently "high" CPU load on 10.0-STABLE From: Adrian Chadd To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 07:13:06 -0000 Hi, I don't know how to do this with dtrace, but take a look at tools/sched/schedgraph.py and enable KTR to get some trace records. KTR logs the scheduler activity -and- the loadav with it. -a On 19 July 2014 23:24, Jeremy Chadwick wrote: > (Please keep me CC'd as I'm not subscribed to freebsd-stable@) > > Today I took the liberty of upgrading my main home server from > 9.3-STABLE (r268785) to 10.0-STABLE (r268894). The upgrade consisted of > doing a fresh install of 10.0-STABLE on a brand new unused SSD. Most > everything went as planned, barring a couple ports-related anomalies, > and I seemed fairly impressed by the fact that buildworld times had > dropped to 27 minutes and buildkernel to 4 minutes with clang (something > I'd been avoiding like the plague for a long while). Kudos. > > But after an hour or so, I noticed a consistent (i.e. reproducible) > trend: the system load average tends to hang around 0.10 to 0.15 all the > time. There are times where the load drops to 0.03 or 0.04 but then > something kicks it back up to 0.15 or 0.20 and then it slowly levels out > again (over the course of a few minutes) then repeats. > > Obviously this is normal behaviour for a system when something is going > on periodically. So I figured it might have been a userland process > behaving differently under 10.x than 9.x. I let top -a -S -s 1 run and > paid very very close attention to it for several minutes. Nothing. It > doesn't appear to be something userland -- it appears to be something > kernel-level, but nothing in top -S shows up as taking up any CPU time > other than "[idle]" so I have no idea what might be doing it. > > The box isn't doing anything like routing network traffic/NAT, it's pure > IPv4 (IPv6 disabled in world and kernel, and my home network does > basically no IPv6) and sits idle most of the time fetching mail. It > does use ZFS, but not for /, swap, /var, /tmp, or /usr. > > vmstat -i doesn't particularly show anything awful. All the cpuX:timer > entries tend to fluctuate in rate, usually 120-200 or so; I'd expect an > interrupt storm to be showing something in the 1000+ range. > > The only thing I can think of is the fact that the SSD being used has no > 4K quirk entry in the kernel (and its ATA IDENTIFY responds with 512 > logical, 512 physical, even though we know it's 4K). The partitions are > all 1MB-aligned regardless. > > This is all bare-metal, by the way -- no virtualisation involved. > > I do have DTrace enabled/built on this box but I have absolutely no clue > how to go about profiling things. For example maybe output of this sort > would be helpful (but I've no idea how to get it): > > http://lists.freebsd.org/pipermail/freebsd-stable/2014-July/079276.html > > I'm certain I didn't see this behaviour in 9.x so I'd be happy to try > and track it down if I had a little bit of hand-holding. > > I've put all the things I can think of that might be relevant to "system > config/tuning bits" up here: > > http://jdc.koitsu.org/freebsd/releng10_perf_issue/ > > I should note my kernel config is slightly inaccurate (I've removed some > stuff from the file in attempt to rebuild, but building world prior to > kernel failed due to r268896 breaking world, but anyone subscribed here > has already seen the Jenkins job of that ;-) ). > > Thanks. > > -- > | Jeremy Chadwick jdc@koitsu.org | > | UNIX Systems Administrator http://jdc.koitsu.org/ | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"