From owner-freebsd-stable@FreeBSD.ORG Thu Aug 26 22:47:16 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB12B1065679 for ; Thu, 26 Aug 2010 22:47:16 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id B73788FC12 for ; Thu, 26 Aug 2010 22:47:16 +0000 (UTC) Received: by pwi8 with SMTP id 8so1059851pwi.13 for ; Thu, 26 Aug 2010 15:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=ZNhrC5HVOHU8ZqI2474OS3ILFwAE7r28YqjUcHtbzxk=; b=BV5BjzI+8ButQ+SU2LPPy1ohrjLLlOZcjQNiRZJ+4MEvJUP8y73/NJcdP1ZyUdAKhv nHJMpVpMgddd8C2TcVU2j8Cs3RXfoui3erV5VOZ+b2g6rBcOZOCVJYQ8/qGKQQZ7+cj4 xQcM3EghrFyILnX2W6yqTCRwcXQi4sL0Iy13E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=YqXcy7Ho2pe/SNoZ+EX2g2ziP18TjokTSbVWVLRQCR0SuLjIBIXf8vll3AQyWwMYiN MSwUg7ulLSSHdSZTmLXWJ4nB8rPR7g4kcutc4nhy2C1y5BGSP89swYjF3RyumenbZa0p FV70YcfBxO2y/xMvytk3GWqsrTEp20p8z2ny4= Received: by 10.142.173.13 with SMTP id v13mr550707wfe.43.1282862835965; Thu, 26 Aug 2010 15:47:15 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id z1sm3522560wfd.15.2010.08.26.15.47.13 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 26 Aug 2010 15:47:14 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 26 Aug 2010 15:47:13 -0700 From: Pyun YongHyeon Date: Thu, 26 Aug 2010 15:47:13 -0700 To: Philipp Wuensche Message-ID: <20100826224713.GG16395@michelle.cdnetworks.com> References: <4C744DC4.3070100@h3q.com> <201008250109.o7P19uEp046002@lava.sentex.ca> <4C76A226.5070302@h3q.com> <20100826212757.GA3391@icarus.home.lan> <4C76E320.9090008@h3q.com> <20100826221526.GA4760@icarus.home.lan> <4C76EA7E.4050205@h3q.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C76EA7E.4050205@h3q.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org, Jack Vogel , Jeremy Chadwick Subject: Re: Crashes on X7SPE-HF with em X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Aug 2010 22:47:17 -0000 On Fri, Aug 27, 2010 at 12:28:14AM +0200, Philipp Wuensche wrote: > Jeremy Chadwick wrote: > > On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote: > >> Jeremy Chadwick wrote: > >>> CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some > >>> ideas. OP's backtrace is here: > >>> > >>> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html > >>> > >>> Philipp, can you please provide the following output? > >>> > >>> * dmesg | egrep 'em[0-9]' > >> em0: port 0xdc00-0xdc1f mem > >> 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 16 at device 0.0 on pci2 > >> em0: Using MSI interrupt > >> em0: [FILTER] > >> em0: Ethernet address: 00:25:90:04:6e:fa > >> em1: port 0xec00-0xec1f mem > >> 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 17 at device 0.0 on pci3 > >> em1: Using MSI interrupt > >> em1: [FILTER] > >> em1: Ethernet address: 00:25:90:04:6e:fb > >> > >>> * uname -a (you can XXX out the machine name if need be) > >> FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST > >> 2010 root@XXX:/usr/obj/usr/src/sys/XXX amd64 > >> > >> Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE > >> too, I can go back to RELEASE or any SVN revision you would like, if it > >> is helping in any way. > >> > >> Kernel-config: > >> > >> include GENERIC > >> > >> ident XXX > >> > >> options IPSEC > >> > >> options DEVICE_POLLING > >> options ACCEPT_FILTER_HTTP > >> > >> options ALTQ > >> > >> options ALTQ_CBQ > >> options ALTQ_RED > >> options ALTQ_RIO > >> options ALTQ_HFSC > >> options ALTQ_PRIQ > >> > >> device crypto > >> device enc > >> > >> > >>> * pciconf -lvc (only include the em(4) items please) > >> em0@pci0:2:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 > >> hdr=0x00 > >> vendor = 'Intel Corporation' > >> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > >> class = network > >> subclass = ethernet > >> cap 01[c8] = powerspec 2 supports D0 D3 current D0 > >> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > >> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > >> cap 11[a0] = MSI-X supports 5 messages in map 0x1c > >> em1@pci0:3:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 > >> hdr=0x00 > >> vendor = 'Intel Corporation' > >> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > >> class = network > >> subclass = ethernet > >> cap 01[c8] = powerspec 2 supports D0 D3 current D0 > >> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > >> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > >> cap 11[a0] = MSI-X supports 5 messages in map 0x1c > >> > >>> * vmstat -i > >> interrupt total rate > >> irq1: atkbd0 9 0 > >> cpu0: timer 36544552 1994 > >> irq256: em0 3801 0 > >> irq257: em1 32963909 1799 > >> irq258: ahci0 175662 9 > >> cpu1: timer 36543525 1994 > >> cpu2: timer 36543525 1994 > >> cpu3: timer 36543525 1994 > >> Total 179318508 9786 > >> > >> There is an shared IPMI interface on em0, but the interface is not used > >> by FreeBSD. em1 is used by four VLANs. Polling is only in the > >> Kernelconfig, not activated on the devices. > > > > So much complexity here. Tracking this down might be difficult. > > > > One thing that does concern me is the interrupt rate for em1. Jack et > > al, is this normal? I don't see this behaviour on my 8.x systems with > > em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and don't > > have MSI-X support. > > This is with the current settings, RXCSUM,TXCSUM and TSO disabled. > Uptime is now only 5h:44m. It crashes about 2 to 3 times a day. I > haven't found a way to trigger this, so I can only wait for it happen again. > I'm not sure but it does not look like em(4) issue. Of course em(4) could free/poke mbuf which was already passed to upper stack but this may happen only when it has severe bug and I didn't see such code yet. It seems you're using ipsec, is it doable for you to disable ipsec to narrow down the issue?