From owner-freebsd-questions@FreeBSD.ORG Tue Jul 1 00:04:58 2008 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E4FD1065672 for ; Tue, 1 Jul 2008 00:04:58 +0000 (UTC) (envelope-from alex@schnarff.com) Received: from mho-01-bos.mailhop.org (mho-01-bos.mailhop.org [63.208.196.178]) by mx1.freebsd.org (Postfix) with ESMTP id 4E66C8FC1E for ; Tue, 1 Jul 2008 00:04:58 +0000 (UTC) (envelope-from alex@schnarff.com) Received: from c-76-114-208-110.hsd1.va.comcast.net ([76.114.208.110] helo=schnarff.com) by mho-01-bos.mailhop.org with esmtpa (Exim 4.68) (envelope-from ) id 1KDTMf-000Pt1-MD for questions@freebsd.org; Tue, 01 Jul 2008 00:04:57 +0000 Received: (qmail 2867 invoked by uid 67); 1 Jul 2008 00:04:56 -0000 Received: from 192.168.2.80 ([192.168.2.80]) by mail.schnarff.com (Horde) with HTTP for ; Mon, 30 Jun 2008 20:04:56 -0400 X-Mail-Handler: MailHop Outbound by DynDNS X-Originating-IP: 76.114.208.110 X-Report-Abuse-To: abuse@dyndns.com (see http://www.mailhop.org/outbound/abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX18lwy9zTcKfPpRBhYvof4bhqoCitvPWN3s= Message-ID: <20080630200456.uf01ro1obms40cok@mail.schnarff.com> Date: Mon, 30 Jun 2008 20:04:56 -0400 From: alex@schnarff.com To: questions@freebsd.org References: <20080630165205.GA3033@lpthe.jussieu.fr> <48691D7C.2090804@FreeBSD.org> <20080630181755.GA3327@lpthe.jussieu.fr> <48692DE7.3020502@FreeBSD.org> <20080630192154.nj1sns26kg44w4w8@mail.schnarff.com> <48696EB0.6000906@FreeBSD.org> In-Reply-To: <48696EB0.6000906@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.0.4) Cc: Subject: Re: Too Much Context Switching? - FIXED X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 00:04:58 -0000 Quoting Kris Kennaway : > alex@schnarff.com wrote: >> Quoting Kris Kennaway : >> >>> Michel Talon wrote: >>>> On Mon, Jun 30, 2008 at 07:53:00PM +0200, Kris Kennaway wrote: >>>>> Yep, it could be that -- what confuses me though is that it is >>>>> claimed that performance suddenly regressed. If so then this >>>>> cannot be the underlying cause. >>>> >>>> It may be that the load has augmented to the point that contention >>>> imposes a rapid regression on throughput. >>> >>> Yes, it could be that. I don't know off-hand whether multiple threads >>> are counted separately by vmstat (at a guess I'd say no), but ps/top/etc >>> should show how many are active in the python process. >> >> Just ran ktrace, and a bit of Googling seems to confirm my initial >> suspicion that the results I'm seeing are abnormal. The first >> several screenfulls of output look like this: >> >> 52929 python2.4 1214867016.469416 CALL kse_wakeup(0x811740c) >> 52929 python2.4 0.000060 RET kse_wakeup 0 >> 52929 python2.4 0.000008 RET kse_release 0 >> 52929 python2.4 0.000040 CALL kse_release(0x811df4c) >> 52929 python2.4 0.000515 CALL kse_wakeup(0x811740c) >> 52929 python2.4 0.000012 RET kse_wakeup 0 >> 52929 python2.4 0.000004 RET kse_release 0 >> 52929 python2.4 0.000012 CALL kse_release(0x811df4c) >> 52929 python2.4 0.000365 CALL kse_wakeup(0x811740c) >> 52929 python2.4 0.000012 RET kse_wakeup 0 >> 52929 python2.4 0.000003 RET kse_release 0 >> 52929 python2.4 0.000010 CALL kse_release(0x811df4c) >> 52929 python2.4 0.000413 CALL kse_wakeup(0x811740c) >> 52929 python2.4 0.000011 RET kse_wakeup 0 >> 52929 python2.4 0.000004 RET kse_release 0 >> 52929 python2.4 0.000009 CALL kse_release(0x811df4c) >> 52929 python2.4 0.000393 CALL kse_wakeup(0x811740c) >> 52929 python2.4 0.000012 RET kse_wakeup 0 >> 52929 python2.4 0.000004 RET kse_release 0 >> 52929 python2.4 0.000009 CALL kse_release(0x811df4c) >> >> I may be mistaken, but it seems like that's a lot of unnecessary >> activity managing the threads; the confirmation I found came from >> http://arkiv.freebsd.se/?ml=freebsd-threads&a=2007-02&t=3178634. >> >> Am I correct that this is abnormal behavior? If so, any idea what I >> may need to do to fix the issue? > > Looks exactly like the python thread problem Michel described. > > You will get some improvement by switching to libthr and/or updating to > 7.0 as I discussed, but ultimately you're hitting limits of python, not > FreeBSD. WOW...it's *amazing* how much of a difference a single sysctl can make. I went ahead and set kern.threads.virtual_cpu=1, as suggested in the thread above, and the difference is ridiculous -- Zope is now faster than I've ever seen. More importantly, my ktracing shows that all of the kse_* garabage is now gone. I'll probably be upgrading to 7.0 in the next month or so, given that this is obviously a thread issue and that that release has much improved thread code. However, for the time being, the pressing issue is fixed, and for anyone in my position stuck on 6.2...this is night & day. Alex