From owner-freebsd-net@FreeBSD.ORG Mon Nov 13 01:48:07 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5EE7416A415; Mon, 13 Nov 2006 01:48:07 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5679543D5E; Mon, 13 Nov 2006 01:48:05 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id kAD1lxBT071427; Sun, 12 Nov 2006 18:48:04 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <4557CECD.2000609@samsco.org> Date: Sun, 12 Nov 2006 18:47:57 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.0.7) Gecko/20060910 SeaMonkey/1.0.5 MIME-Version: 1.0 To: Mike Tancsa References: <2a41acea0611081719h31be096eu614d2f2325aff511@mail.gmail.com> <200611091536.kA9FaltD018819@lava.sentex.ca> <45534E76.6020906@samsco.org> <200611092200.kA9M0q1E020473@lava.sentex.ca> <200611102004.kAAK4iO9027778@lava.sentex.ca> <2a41acea0611101400w5b8cef40ob84ed6de181f3e2c@mail.gmail.com> <200611102221.kAAML6ol028630@lava.sentex.ca> <455570D8.6070000@samsco.org> <200611120412.kAC4CuIB035746@lava.sentex.ca> <45574ECA.4080207@samsco.org> <200611130040.kAD0etbp040637@lava.sentex.ca> In-Reply-To: <200611130040.kAD0etbp040637@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: freebsd-net , freebsd-stable@freebsd.org, Jack Vogel Subject: Re: Proposed 6.2 em RELEASE patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Nov 2006 01:48:07 -0000 Mike Tancsa wrote: > At 11:41 AM 11/12/2006, Scott Long wrote: >> Mike Tancsa wrote: >>> At 01:42 AM 11/11/2006, Scott Long wrote: >>> driver. What will help me is if you can hook up a serial console to >>>> your machine and see if it can be made to drop to the debugger while it >>>> is under load and otherwise unresponsive. If you can, getting a >>>> process >>>> dump might help confirm where each CPU is spending its time. >>> >>> ./netblast 192.168.88.218 500 110 1000 >>> I compiled in the various debugging options and on the serial console >>> I get a few >>> Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s >>> and the serial console >> >> One CPU seems to be stuck in softclock. My guess here is that there >> is significant lock contention. Please try the following: >> >> 1. Use addr2line or gdb on the kernel to find out what function is at >> 0xc0601e48 (the address that was printed above). This address will >> change every time you recompile the kernel, so you'll need to reacquire >> it from the console each time. > > # addr2line 0xc0601e48 -e kernel.debug -f > nfsrv_timer > /usr/src/sys/nfsserver/nfs_srvsock.c:809 > # > Can you try removing NFS from your kernel? > and > > # addr2line 0xc0561444 -e kernel.debug -f > sleepq_timeout > /usr/src/sys/kern/subr_sleepqueue.c:724 This is sched_lock contention. > > I dont have any nfs activity on the box > >> 2. Try compiling in WITNESS and running the test as before, then break >> into the debugger as before. Run 'show locks'. I'm not sure how >> fruitful this will be, WITNESS might make it unbearably slow. > > It was in that kernel already So you're seeing the livelock under load while also having WITNESS enabled? Have you tried your test without WITNESS? What about INVARIANTS? Scott