From owner-freebsd-questions@FreeBSD.ORG Tue Jul 1 00:20:37 2008 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC37A106567A for ; Tue, 1 Jul 2008 00:20:36 +0000 (UTC) (envelope-from alex@schnarff.com) Received: from mho-01-bos.mailhop.org (mho-01-bos.mailhop.org [63.208.196.178]) by mx1.freebsd.org (Postfix) with ESMTP id AAFD18FC1D for ; Tue, 1 Jul 2008 00:20:36 +0000 (UTC) (envelope-from alex@schnarff.com) Received: from c-76-114-208-110.hsd1.va.comcast.net ([76.114.208.110] helo=schnarff.com) by mho-01-bos.mailhop.org with esmtpa (Exim 4.68) (envelope-from ) id 1KDTbn-0004rl-U8 for questions@freebsd.org; Tue, 01 Jul 2008 00:20:36 +0000 Received: (qmail 4130 invoked by uid 67); 1 Jul 2008 00:20:34 -0000 Received: from 192.168.2.80 ([192.168.2.80]) by mail.schnarff.com (Horde) with HTTP for ; Mon, 30 Jun 2008 20:20:34 -0400 X-Mail-Handler: MailHop Outbound by DynDNS X-Originating-IP: 76.114.208.110 X-Report-Abuse-To: abuse@dyndns.com (see http://www.mailhop.org/outbound/abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1/ckXZjdxkTUbXDloaWhSEAJFM4DlRgXMI= Message-ID: <20080630202034.dt6mqbf5css444gg@mail.schnarff.com> Date: Mon, 30 Jun 2008 20:20:34 -0400 From: alex@schnarff.com To: questions@freebsd.org References: <20080630165205.GA3033@lpthe.jussieu.fr> <48691D7C.2090804@FreeBSD.org> <20080630181755.GA3327@lpthe.jussieu.fr> <48692DE7.3020502@FreeBSD.org> <20080630192154.nj1sns26kg44w4w8@mail.schnarff.com> <48696EB0.6000906@FreeBSD.org> <20080630200456.uf01ro1obms40cok@mail.schnarff.com> <48697719.40101@FreeBSD.org> In-Reply-To: <48697719.40101@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.0.4) Cc: Subject: Re: Too Much Context Switching? - FIXED X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 00:20:37 -0000 Quoting Kris Kennaway : > alex@schnarff.com wrote: >> Quoting Kris Kennaway : >> >>> alex@schnarff.com wrote: >>>> Quoting Kris Kennaway : >>>> >>>>> Michel Talon wrote: >>>>>> On Mon, Jun 30, 2008 at 07:53:00PM +0200, Kris Kennaway wrote: >>>>>>> Yep, it could be that -- what confuses me though is that it is >>>>>>> claimed that performance suddenly regressed. If so then this >>>>>>> cannot be the underlying cause. >>>>>> >>>>>> It may be that the load has augmented to the point that contention >>>>>> imposes a rapid regression on throughput. >>>>> >>>>> Yes, it could be that. I don't know off-hand whether multiple threads >>>>> are counted separately by vmstat (at a guess I'd say no), but ps/top/etc >>>>> should show how many are active in the python process. >>>> >>>> Just ran ktrace, and a bit of Googling seems to confirm my initial >>>> suspicion that the results I'm seeing are abnormal. The first >>>> several screenfulls of output look like this: >>>> >>>> 52929 python2.4 1214867016.469416 CALL kse_wakeup(0x811740c) >>>> 52929 python2.4 0.000060 RET kse_wakeup 0 >>>> 52929 python2.4 0.000008 RET kse_release 0 >>>> 52929 python2.4 0.000040 CALL kse_release(0x811df4c) >>>> 52929 python2.4 0.000515 CALL kse_wakeup(0x811740c) >>>> 52929 python2.4 0.000012 RET kse_wakeup 0 >>>> 52929 python2.4 0.000004 RET kse_release 0 >>>> 52929 python2.4 0.000012 CALL kse_release(0x811df4c) >>>> 52929 python2.4 0.000365 CALL kse_wakeup(0x811740c) >>>> 52929 python2.4 0.000012 RET kse_wakeup 0 >>>> 52929 python2.4 0.000003 RET kse_release 0 >>>> 52929 python2.4 0.000010 CALL kse_release(0x811df4c) >>>> 52929 python2.4 0.000413 CALL kse_wakeup(0x811740c) >>>> 52929 python2.4 0.000011 RET kse_wakeup 0 >>>> 52929 python2.4 0.000004 RET kse_release 0 >>>> 52929 python2.4 0.000009 CALL kse_release(0x811df4c) >>>> 52929 python2.4 0.000393 CALL kse_wakeup(0x811740c) >>>> 52929 python2.4 0.000012 RET kse_wakeup 0 >>>> 52929 python2.4 0.000004 RET kse_release 0 >>>> 52929 python2.4 0.000009 CALL kse_release(0x811df4c) >>>> >>>> I may be mistaken, but it seems like that's a lot of unnecessary >>>> activity managing the threads; the confirmation I found came from >>>> http://arkiv.freebsd.se/?ml=freebsd-threads&a=2007-02&t=3178634. >>>> >>>> Am I correct that this is abnormal behavior? If so, any idea what >>>> I may need to do to fix the issue? >>> >>> Looks exactly like the python thread problem Michel described. >>> >>> You will get some improvement by switching to libthr and/or updating to >>> 7.0 as I discussed, but ultimately you're hitting limits of python, not >>> FreeBSD. >> >> WOW...it's *amazing* how much of a difference a single sysctl can make. >> >> I went ahead and set kern.threads.virtual_cpu=1, as suggested in the >> thread above, and the difference is ridiculous -- Zope is now faster >> than I've ever seen. More importantly, my ktracing shows that all of >> the kse_* garabage is now gone. >> >> I'll probably be upgrading to 7.0 in the next month or so, given >> that this is obviously a thread issue and that that release has much >> improved thread code. However, for the time being, the pressing >> issue is fixed, and for anyone in my position stuck on 6.2...this is >> night & day. > > Seriously, try libthr. No matter what you do to libkse it is going to > suck. That's why we removed it. I will, probably as part of upgrading to 7.0 (which I may accelerate, given this point). I'm just ecstatic at the difference I'm already seeing, and specifically wanted to make note of it in the archives. Point very much taken, though. :-) Alex