From owner-freebsd-questions@FreeBSD.ORG Mon Jun 30 14:48:27 2008 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87BAD1065683 for ; Mon, 30 Jun 2008 14:48:27 +0000 (UTC) (envelope-from alex@schnarff.com) Received: from mho-01-bos.mailhop.org (mho-01-bos.mailhop.org [63.208.196.178]) by mx1.freebsd.org (Postfix) with ESMTP id 583BA8FC20 for ; Mon, 30 Jun 2008 14:48:27 +0000 (UTC) (envelope-from alex@schnarff.com) Received: from c-76-114-208-110.hsd1.va.comcast.net ([76.114.208.110] helo=schnarff.com) by mho-01-bos.mailhop.org with esmtpa (Exim 4.68) (envelope-from ) id 1KDKg6-000Ane-DA for questions@freebsd.org; Mon, 30 Jun 2008 14:48:26 +0000 Received: (qmail 11178 invoked by uid 67); 30 Jun 2008 14:48:25 -0000 Received: from sf-nat.sourcefire.com (sf-nat.sourcefire.com [64.214.53.2]) by mail.schnarff.com (Horde) with HTTP for ; Mon, 30 Jun 2008 10:48:25 -0400 X-Mail-Handler: MailHop Outbound by DynDNS X-Originating-IP: 76.114.208.110 X-Report-Abuse-To: abuse@dyndns.com (see http://www.mailhop.org/outbound/abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+9PSDR8/dKcyduGADlgsWxKgh8YeAT7FI= Message-ID: <20080630104825.hm91ul0focsgg0s0@mail.schnarff.com> Date: Mon, 30 Jun 2008 10:48:25 -0400 From: alex@schnarff.com To: Kris Kennaway References: <20080629222123.n0rgkjka50c084k4@mail.schnarff.com> <4868B21D.9040105@FreeBSD.org> In-Reply-To: <4868B21D.9040105@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.0.4) Cc: questions@freebsd.org Subject: Re: Too Much Context Switching? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 14:48:27 -0000 First off, thanks for such a prompt response. :-) > alex@schnarff.com wrote: >> I'm the webmaster for www.marssociety.org, which is a FreeBSD >> 6.2-RELEASE box running on a dual-core AMD Opteron setup with 4GB of >> RAM. The box is reasonably busy, as it's the sole piece of hardware >> running web, database, and mail operations for the Mars Society, an >> international nonprofit group dedicated to space exploration. We >> regularly send out newsletters to ~10,000 members, and our web site >> is averaging ~50,000-100,000 hits/day. >> >> The main portion of the web site is run via the Zope/Plone CMS >> system (Plone 2.5, for anyone who may care). Recently, it's been >> slowing down dramatically, and our Plone guy (not me -- I inherited >> the system and can't stand it) can't figure out why. I've been >> diving into OS-related issues, and in so doing, I ran across what >> appears to be a very high number of context switches going on. >> Here's some sample output from "vmstat 2": > > A few hundred or thousand context switches per second is trivial load. > That is not your problem. Modern CPUs can do hundreds of thousands per > second before it starts to become a problem. OK, well that's good to know. > Note that your system is 50% idle and spending almost no time in the > kernel. This basically means that only one core is doing work, which > might be because you're not giving it enough work to do. There are only > 1-2 running tasks for most of your trace, one of which is probably > vmstat itself, so that means there is only one running server process > (which can obviously only saturate at most 1 CPU). Actually, I decided to run vmstat this morning for a little while after turning off Zope, and during the couple of minutes I had it going, the number of processes running (as indicated by the leftmost column of vmstat's output) was at 0 for all but one line worth of output, so I would guess that vmstat's not including itself in the number of processes there. Even so, though, your assessment about how saturated the CPU is is of course still valid, which leads me to a follow-up question: by default, can a multi-threaded app use both cores? Or would I need to have two instances of the process running (Zope is apparently able to handle multiple instances running reasonably well) in order to have it fully utilize the CPU? > The trace suggests that your performance problems are either in > userland, or elsewhere in your network or application stack, possibly > due to interactions between components. Try to look at why the system > is not being given enough work to keep it saturated. Any tips on tools I could use to check this out? I'll of course be looking at Zope profiling tools, to see if I can have them tell me where any bottlenecks are, but if there are any OS-level tools that I could use to profile a given process (or group thereof) for problems, I'd really appreciate hearing about them (simple links to man pages or the like would be fine, I don't mean to waste your time explaining how tools work when I can usually figure it out on my own). Alex Kirk