From owner-freebsd-net@FreeBSD.ORG Sat Dec 25 02:19:29 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75CF7106566C; Sat, 25 Dec 2010 02:19:29 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 07F878FC0C; Sat, 25 Dec 2010 02:19:28 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:5570:ff31:c40c:8eac] ([IPv6:2607:f3e0:0:4:5570:ff31:c40c:8eac]) by smarthost1.sentex.ca (8.14.4/8.14.4) with ESMTP id oBP2JQmh036316 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 24 Dec 2010 21:19:26 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4D1554A7.5000507@sentex.net> Date: Fri, 24 Dec 2010 21:19:19 -0500 From: Mike Tancsa User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jan Koum References: <1290533941.3173.50.camel@home-yahoo> <4CEC0548.1080801@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on IPv6:2607:f3e0:0:1::12 Cc: "freebsd-net@freebsd.org" , Sean Bruno , Jack Vogel , Ivan Voras , "freebsd-hardware@freebsd.org" Subject: Re: em driver, 82574L chip, and possibly ASPM X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Dec 2010 02:19:29 -0000 On 12/24/2010 5:44 PM, Jan Koum wrote: > hi Ivan and Mike, > > wanted to follow up and see if you found a solid long-term solution to this > bug. we are still seeing this problem in our 8.2 environment with ASPM > already disabled. here is what we have: Hmmm, With the latest version of the driver in RELENG_8 (its the same as in HEAD) I havent seen the problem. However, I would only see it once per week prior to that. The odd thing is that it would happen during a slightly lower than normal backup load, but almost always at the same time (early sunday AM). Not sure what would trigger it exactly. If it happened again, I was going to enable port mirroring on the switchport and capture the traffic, hoping some "special" pattern would enable the issue. Do you have IPMI enabled on the NIC ? I tried to turn it off on my MB, but there is no clear way to do this. It 'seems' to be off, but not sure if it really is. One thing I noticed was that when the NIC was hung, it still was able to receive and process IPMI commands from an external host. ---Mike > > 1. motherboard is SuperMicro X8SIE-LN4F Intel Xeon: > > em0@pci0:3:0:0: class=0x020000 card=0x040d15d9 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > class = network > subclass = ethernet > em1@pci0:4:0:0: class=0x020000 card=0x040d15d9 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > class = network > subclass = ethernet > em2@pci0:5:0:0: class=0x020000 card=0x040d15d9 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > class = network > subclass = ethernet > em3@pci0:6:0:0: class=0x020000 card=0x040d15d9 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > class = network > subclass = ethernet > > 2. ASPM is already disabled in the BIOS > > 3. when em1 interface locks up, sysctl debug says: > > Interface is NOT RUNNING > and INACTIVE > em1: hw tdh = 0, hw tdt = 0 > em1: hw rdh = 0, hw rdt = 0 > em1: Tx Queue Status = 0 > em1: TX descriptors avail = 110 > em1: Tx Descriptors avail failure = 319 > em1: RX discarded packets = 0 > em1: RX Next to Check = 80 > em1: RX Next to Refresh = 80 > > 4. doing "ifconfig em1 down; sleep1; ifconfig em1 up" resolves the issue and > removes OACTIVE flag from em1. > > 5. we are running 8.2-PRERELEASE from December 19th: > % grep '$FreeBSD' /usr/src/sys/dev/e1000/if_em.c > /*$FreeBSD: src/sys/dev/e1000/if_em.c,v 1.21.2.18 2010/12/14 19:59:39 jfv > Exp $*/ > > dmesg output is: > > em1: port 0xcc00-0xcc1f mem > 0xfb4e0000-0xfb4fffff,0xfb4dc000-0xfb4dffff irq 17 at device 0.0 on pci4 > em1: Reserved 0x20000 bytes for rid 0x10 type 3 at 0xfb4e0000 > em1: Reserved 0x4000 bytes for rid 0x1c type 3 at 0xfb4dc000 > em1: attempting to allocate 3 MSI-X vectors (5 supported) > msi: routing MSI-X IRQ 259 to local APIC 0 vector 53 > msi: routing MSI-X IRQ 260 to local APIC 0 vector 54 > msi: routing MSI-X IRQ 261 to local APIC 0 vector 55 > em1: using IRQs 259-261 for MSI-X > em1: Using MSIX interrupts with 3 vectors > em1: [MPSAFE] > em1: [ITHREAD] > em1: [MPSAFE] > em1: [ITHREAD] > em1: [MPSAFE] > em1: [ITHREAD] > em1: bpf attached > em1: Ethernet address: 00:25:90:0e:25:e9 > > aside from running cronjob every minute to check for dead interface and > reset it, is there anything else we can try? > > thanks. > > > On Tue, Nov 23, 2010 at 10:36 AM, Jack Vogel wrote: > >> 82574 is supposed to be em, not igb :) Its always had this kind of >> 'in-between' >> status, it was targeted as a 'client' or consumer part, but it has MSIX >> which >> make it almost like 8257[56]. >> >> Mike, there are some further 82574 changes to shared code that I'm looking >> into today. >> >> Jack >> >> >> On Tue, Nov 23, 2010 at 10:17 AM, Mike Tancsa wrote: >> >>> On 11/23/2010 12:39 PM, Sean Bruno wrote: >>>> On Tue, 2010-11-23 at 04:47 -0800, Ivan Voras wrote: >>>>> It looks like I'm unfortunate enough to have to deploy on a machine >>>>> which has the 82574L Intel NIC chip on a Supermicro X8SIE-F board, >> which >>>> igb0@pci0:5:0:0: class=0x020000 card=0x8975152d chip=0x10c98086 >>> >>> Strange, the 82574 attaches as em for me, not igb >>> >>> em1@pci0:10:0:0: class=0x020000 card=0x34ec8086 chip=0x10d38086 >>> rev=0x00 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' >>> class = network >>> subclass = ethernet >>> cap 01[c8] = powerspec 2 supports D0 D3 current D0 >>> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message >>> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) >>> cap 11[a0] = MSI-X supports 5 messages in map 0x1c >>> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected >>> ecap 0003[140] = Serial 1 001517ffffed68a4 >>> >>> Normally, its msix, but I had disabled that hoping it would fix the >> problem >>> >>> em1: port 0x2000-0x201f mem >>> 0xb4100000-0xb411ffff,0xb4120000-0xb4123fff irq 16 at dev >>> ice 0.0 on pci10 >>> em1: Using an MSI interrupt >>> em1: [FILTER] >>> em1: Ethernet address: 00:15:17:ed:68:a4 >>> >>> >>> ---Mike >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >> _______________________________________________ >> freebsd-hardware@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hardware >> To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@freebsd.org >> " >> >