From owner-freebsd-arch@FreeBSD.ORG Fri Sep 19 15:59:23 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B59FB16A4B3 for ; Fri, 19 Sep 2003 15:59:23 -0700 (PDT) Received: from ebb.errno.com (ebb.errno.com [66.127.85.87]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED9AB43FDF for ; Fri, 19 Sep 2003 15:59:22 -0700 (PDT) (envelope-from sam@errno.com) Received: from melange.errno.com (melange.errno.com [66.127.85.82]) (authenticated bits=0) by ebb.errno.com (8.12.9/8.12.9) with ESMTP id h8JMxG0x011223 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Fri, 19 Sep 2003 15:59:20 -0700 (PDT) (envelope-from sam@errno.com) Date: Fri, 19 Sep 2003 15:59:15 -0700 From: Sam Leffler To: freebsd-arch@freebsd.org Message-ID: <606095639.1063987155@melange.errno.com> X-Mailer: Mulberry/3.0.3 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: interrupt latency and driver locking X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2003 22:59:23 -0000 I enabled MUTEX_PROFILING on a fast machine I've got setup as a firewall/router. The machine had an fxp device on one side and an em on the other. I then ran a bunch of netperf tests through the machine. Unfortunately for the moment I'm stuck with 100baseT on the fxp side of the machine so my tests were all limited by the media. But one thing stuck out: max total count avg name 282 786 3 262 kern/kern_exec.c:853 (vm page queue mute 775 7450 33 225 kern/vfs_subr.c:3218 (mntvnode) 1119 2789 15 185 kern/kern_fork.c:218 (Giant) 222 2344 14 167 vm/uma_core.c:1186 (UMA lock) 460 2972 19 156 kern/kern_exit.c:282 (vm page queue mute 508 2415 19 127 kern/kern_exit.c:101 (Giant) 194 1801 16 112 kern/kern_exec.c:261 (Giant) 332 31113 277 112 dev/fxp/if_fxp.c:1798 (fxp0) 2435 31848 366 87 kern/kern_sysctl.c:1210 (Giant) 901 5306 66 80 vm/vm_map.c:1256 (Giant) 77 77 1 77 kern/uipc_syscalls.c:472 (Giant) 182 2901 39 74 vm/vm_kern.c:328 (system map) 507 841090 12508 67 dev/random/yarrow.c:172 (random reseed) 7005 68335 1136 60 kern/kern_intr.c:533 (Giant) 66 964 16 60 vm/vm_map.c:2241 (system map) 88 7471 139 53 dev/em/if_em.c:1531 (em0) 881 8032 191 42 i386/i386/trap.c:997 (Giant) 591 10568722 271157 38 net/netisr.c:215 (netisr lock) 38 292 8 36 vm/vm_fault.c:997 (Giant) Note that the fxp driver holds its driver lock for an average 112 us in one spot. This turns out to be in fxp_tick and the likely culprit is the call to mii_tick which is done with the lock held. Since there's only one lock in the driver this means interrupts are blocked during this time which is likely to do bad things. (The em driver comparison has a similar bottleneck though it's <1/2 as long.) I'm not sure if it's simple to move MII operations outside of the driver lock but it might be worth investigating. Sam