From owner-freebsd-net@FreeBSD.ORG Sun Nov 12 16:42:06 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39EDC16A403; Sun, 12 Nov 2006 16:42:06 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7F0DC43D45; Sun, 12 Nov 2006 16:41:54 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id kACGfl9i068088; Sun, 12 Nov 2006 09:41:52 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <45574ECA.4080207@samsco.org> Date: Sun, 12 Nov 2006 09:41:46 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.0.7) Gecko/20060910 SeaMonkey/1.0.5 MIME-Version: 1.0 To: Mike Tancsa References: <2a41acea0611081719h31be096eu614d2f2325aff511@mail.gmail.com> <200611091536.kA9FaltD018819@lava.sentex.ca> <45534E76.6020906@samsco.org> <200611092200.kA9M0q1E020473@lava.sentex.ca> <200611102004.kAAK4iO9027778@lava.sentex.ca> <2a41acea0611101400w5b8cef40ob84ed6de181f3e2c@mail.gmail.com> <200611102221.kAAML6ol028630@lava.sentex.ca> <455570D8.6070000@samsco.org> <200611120412.kAC4CuIB035746@lava.sentex.ca> In-Reply-To: <200611120412.kAC4CuIB035746@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: freebsd-net , freebsd-stable@freebsd.org, Jack Vogel Subject: Re: Proposed 6.2 em RELEASE patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Nov 2006 16:42:06 -0000 Mike Tancsa wrote: > At 01:42 AM 11/11/2006, Scott Long wrote: > driver. What will help me is if you can hook up a serial console to >> your machine and see if it can be made to drop to the debugger while it >> is under load and otherwise unresponsive. If you can, getting a process >> dump might help confirm where each CPU is spending its time. > > > ./netblast 192.168.88.218 500 110 1000 > > I compiled in the various debugging options and on the serial console I > get a few > > Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s > and the serial console > One CPU seems to be stuck in softclock. My guess here is that there is significant lock contention. Please try the following: 1. Use addr2line or gdb on the kernel to find out what function is at 0xc0601e48 (the address that was printed above). This address will change every time you recompile the kernel, so you'll need to reacquire it from the console each time. 2. Try compiling in WITNESS and running the test as before, then break into the debugger as before. Run 'show locks'. I'm not sure how fruitful this will be, WITNESS might make it unbearably slow. 3. Turn on MUTEX_PROFILING according to the text in /sys/conf/NOTES and the man page by the same name. 4. Remove ADAPTIVE_GIANT from your config and retest. If that doesn't show a difference, then try setting NO_ADAPTIVE_MUTEXES. Thanks, Scott