From owner-freebsd-net@FreeBSD.ORG Fri Oct 7 15:06:51 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8F1F1065674 for ; Fri, 7 Oct 2011 15:06:51 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 88DD88FC14 for ; Fri, 7 Oct 2011 15:06:51 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id p97F6kIJ009463; Fri, 7 Oct 2011 11:06:46 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <4E8F157A.40702@sentex.net> Date: Fri, 07 Oct 2011 11:06:34 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jason Wolfe References: In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Oct 2011 15:06:51 -0000 On 10/6/2011 7:15 PM, Jason Wolfe wrote: > I'm seeing the interface wedge on a good number of systems with Intel 82574L > chips under FBSD8.2 _only when MSI-X is enabled_, running either 7.1.9 from > 8.2-RELEASE or 7.2.3 from 8.2-STABLE. I have em0 and em1 in a lagg, but > only one side would fail, and a few systems that didn't have a lagg also saw > the issue. Higher traffic did seem to increase the likely hood of it > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.1.9 Hi, This sure sounds like the issue I was seeing with the 7.1.9 driver... However, it has been fixed for me by going to 7.2.3, which is in RELENG_8. Is it possible you have a couple of issues going on since you are using lagg as well ? Another problem some folks have reported is that in the BIOS, if you have an option for ASPM, make sure its disabled. Google around for ASPM and 82574L for a discussion about it. If I recall correctly, disabling MSI-X just reduces the chance of the problem happening, but its been a while since I ran into this issue. But for sure you want to be running 7.2.3 from stable This server used to see this issue dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.1.%driver: em dev.em.1.%location: slot=0 function=0 handle=\_SB_.PCI0.PEX4.HART dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0x34ec class=0x020000 dev.em.1.%parent: pci11 dev.em.1.nvm: -1 dev.em.1.debug: -1 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 dev.em.1.flow_control: 3 dev.em.1.eee_control: 0 dev.em.1.link_irq: 0 dev.em.1.mbuf_alloc_fail: 0 dev.em.1.cluster_alloc_fail: 0 dev.em.1.dropped: 0 dev.em.1.tx_dma_fail: 0 dev.em.1.rx_overruns: 0 dev.em.1.watchdog_timeouts: 0 dev.em.1.device_control: 1209008712 dev.em.1.rx_control: 67141634 dev.em.1.fc_high_water: 18432 dev.em.1.fc_low_water: 16932 dev.em.1.queue0.txd_head: 754 dev.em.1.queue0.txd_tail: 754 dev.em.1.queue0.tx_irq: 251430977 dev.em.1.queue0.no_desc_avail: 0 dev.em.1.queue0.rxd_head: 304 dev.em.1.queue0.rxd_tail: 303 dev.em.1.queue0.rx_irq: 295670362 dev.em.1.mac_stats.excess_coll: 0 dev.em.1.mac_stats.single_coll: 0 dev.em.1.mac_stats.multiple_coll: 0 dev.em.1.mac_stats.late_coll: 0 dev.em.1.mac_stats.collision_count: 0 dev.em.1.mac_stats.symbol_errors: 0 dev.em.1.mac_stats.sequence_errors: 0 dev.em.1.mac_stats.defer_count: 0 dev.em.1.mac_stats.missed_packets: 0 dev.em.1.mac_stats.recv_no_buff: 0 dev.em.1.mac_stats.recv_undersize: 0 dev.em.1.mac_stats.recv_fragmented: 0 dev.em.1.mac_stats.recv_oversize: 0 dev.em.1.mac_stats.recv_jabber: 0 dev.em.1.mac_stats.recv_errs: 0 dev.em.1.mac_stats.crc_errs: 0 dev.em.1.mac_stats.alignment_errs: 0 dev.em.1.mac_stats.coll_ext_errs: 0 dev.em.1.mac_stats.xon_recvd: 0 dev.em.1.mac_stats.xon_txd: 0 dev.em.1.mac_stats.xoff_recvd: 0 dev.em.1.mac_stats.xoff_txd: 0 dev.em.1.mac_stats.total_pkts_recvd: 712410384 dev.em.1.mac_stats.good_pkts_recvd: 712410384 dev.em.1.mac_stats.bcast_pkts_recvd: 52263 dev.em.1.mac_stats.mcast_pkts_recvd: 24921 dev.em.1.mac_stats.rx_frames_64: 170050 dev.em.1.mac_stats.rx_frames_65_127: 32571360 dev.em.1.mac_stats.rx_frames_128_255: 19796510 dev.em.1.mac_stats.rx_frames_256_511: 6283830 dev.em.1.mac_stats.rx_frames_512_1023: 7922330 dev.em.1.mac_stats.rx_frames_1024_1522: 645666304 dev.em.1.mac_stats.good_octets_recvd: 988128549661 dev.em.1.mac_stats.good_octets_txd: 48849605092 dev.em.1.mac_stats.total_pkts_txd: 501680484 dev.em.1.mac_stats.good_pkts_txd: 501680484 dev.em.1.mac_stats.bcast_pkts_txd: 4266 dev.em.1.mac_stats.mcast_pkts_txd: 8 dev.em.1.mac_stats.tx_frames_64: 134256137 dev.em.1.mac_stats.tx_frames_65_127: 291152180 dev.em.1.mac_stats.tx_frames_128_255: 67219002 dev.em.1.mac_stats.tx_frames_256_511: 5935140 dev.em.1.mac_stats.tx_frames_512_1023: 812920 dev.em.1.mac_stats.tx_frames_1024_1522: 2305105 dev.em.1.mac_stats.tso_txd: 366978 dev.em.1.mac_stats.tso_ctx_fail: 0 dev.em.1.interrupts.asserts: 2 dev.em.1.interrupts.rx_pkt_timer: 0 dev.em.1.interrupts.rx_abs_timer: 0 dev.em.1.interrupts.tx_pkt_timer: 0 dev.em.1.interrupts.tx_abs_timer: 0 dev.em.1.interrupts.tx_queue_empty: 0 dev.em.1.interrupts.tx_queue_min_thresh: 0 dev.em.1.interrupts.rx_desc_min_thresh: 0 dev.em.1.interrupts.rx_overrun: 0 interrupt total rate irq4: uart0 44896 0 irq16: bge0 19753077 32 irq18: arcmsr0 37518694 62 irq19: twa0 556664 0 irq21: ehci0 2149928 3 irq23: ehci1 1209435 2 cpu0: timer 1209274084 2000 irq256: siis0 65793731 108 irq257: em0 504313285 834 irq258: em1:rx 0 295681170 489 irq259: em1:tx 0 251430780 415 irq261: ahci0 71285304 117 cpu1: timer 1209264969 2000 cpu3: timer 1209266038 2000 cpu2: timer 1209265460 2000 Total 6086807515 10067 vendor = 'Intel Corporation' device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xb4100000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled bar [1c] = type Memory, range 32, base 0xb4120000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 0003[140] = Serial 1 001517ffffed68a4 ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/