From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 13:16:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D2601065675 for ; Wed, 26 Oct 2011 13:16:20 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id A5C578FC14 for ; Wed, 26 Oct 2011 13:16:19 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id p9QDGFbj089388; Wed, 26 Oct 2011 09:16:15 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <4EA80818.3030504@sentex.net> Date: Wed, 26 Oct 2011 09:16:08 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jason Wolfe References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jack Vogel Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 13:16:20 -0000 On 10/26/2011 7:33 AM, Jason Wolfe wrote: > Hooman, > > I have run with dev.em.X.flow_control=0, which should have the same result > as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the > remaining options would be able to produce the scenario I'm seeing, but I'm > open to giving it a try with no options on the interfaces. I've also added > ifconfig output to the collection. > > options=219b > ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol > options=88 > > It's always TX, but these servers push ~12x what they receive, so I'm > guessing it could happen to either buffer given the right traffic patterns. > While looking through commits I also found a patch to add a couple sysctls > for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 Can you provide some more details as to how traffic flows on these servers ? Are they going through them, or are they generating the traffic ? I wonder given a smaller CPU, it would be easier to trigger the condition somehow. We recently got a Soekris 6501 which has onboard 4 such em nics, but has just a 1Ghz Atom. em0@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa1000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa1020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled em1@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa2000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa2020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled em2@pci0:10:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa3000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x4000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa3020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled em3@pci0:11:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa4000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x5000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa4020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled soekris6501# vmstat -i interrupt total rate irq4: uart0 4993 0 cpu0:timer 84947808 71 irq256: ahci0 59195 0 irq257: em0:rx 0 1946546 1 irq258: em0:tx 0 629707 0 irq259: em0:link 2 0 Total 87588251 73 soekris6501# -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/