From owner-freebsd-net@FreeBSD.ORG Wed Dec 2 14:33:59 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF256106566B for ; Wed, 2 Dec 2009 14:33:59 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63906.mail.re1.yahoo.com (web63906.mail.re1.yahoo.com [69.147.97.121]) by mx1.freebsd.org (Postfix) with SMTP id 88FD58FC14 for ; Wed, 2 Dec 2009 14:33:59 +0000 (UTC) Received: (qmail 28365 invoked by uid 60001); 2 Dec 2009 14:33:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1259764438; bh=SRnDvwBKzGDNDK2/V3bfMRHQh3GyBbek9z5Mp/9gWpA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=0NldB2g4E/LBwVC9cC23Z+N+xmKunvNQATG7c6uURANdEdAohvA27Lnwcx+8xu/Ez9y1pCKcZ6Jf6nz6gSCQNYkTe9sUbCs+97iPwPHKXR/jg1yz73Ldhvvk8X0+ybxTkXQC9V9/vN3yfToIa1zHCPcmkQEgE70T0q1+m8ro+ic= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=KiluLuUuyNsvbc4lT3iNtygmgwP3jIXhyK6J+aBSKzN+3IAhSl2i456vluKbXpVv/nFBuAQ5xG5O/y05K72LNTPrzNtJT8kDpnIOOXBHzUVTCCXDa0UPQptYVcS4gsHdOcBD0oa5tNO9OAcYx6AuIwAttCy1VPQ0PrBEZc5/BU8=; Message-ID: <913529.24196.qm@web63906.mail.re1.yahoo.com> X-YMail-OSG: fRJ2OlcVM1n7xLG9csELfEBwDBdT6NbM7NWhwr_ZCPOnGByLsEJTF7tgwvJpwhT1TIU1TSgs6BMi5M92T1zwk_UCPK5YXmafdVlPoGawiFZLD7XwhLMAk.qEW3ov1Lc2OYPhxko5lZQeR82Au2IrX62u8tBPzDXwzMTDgg0VtvxO.LMvslQPKuvYFmMcByp2TW78RPkaAPyF3f0zy6ROyXFD0dACyHT9aYe0PBrV1UY9PKzIGz5b6qY.8mbt2vavcBwpjf7FR2khzymhUCFNndpJeHJoRaQNb5Txvzk- Received: from [98.203.21.152] by web63906.mail.re1.yahoo.com via HTTP; Wed, 02 Dec 2009 06:33:58 PST X-Mailer: YahooMailClassic/8.1.6 YahooMailWebService/0.8.100.260964 Date: Wed, 2 Dec 2009 06:33:58 -0800 (PST) From: Barney Cordoba To: "Yuriy A. Korobko" In-Reply-To: <1258998140.3015.34.camel@stormi-desktop> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Reducing number of interrupts from intel pro 1000 et adapter X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Dec 2009 14:33:59 -0000 =0A=0A--- On Mon, 11/23/09, Yuriy A. Korobko wro= te:=0A=0A> From: Yuriy A. Korobko =0A> Subject: R= educing number of interrupts from intel pro 1000 et adapter=0A> To: freebsd= -net@freebsd.org=0A> Date: Monday, November 23, 2009, 12:42 PM=0A> Hi,=0A> = =0A> I'd like to know a way to control tx interrupts on intel=0A> pro 1000 = et=0A> adapter with igb driver. Just installed one in the router=0A> and sy= stat=0A> shows 8-9k rx interrupts and 20k tx interrupts from igb0=0A> and i= gb1=0A> adapters. Box is a router running freebsd 7.2 release, I've=0A> tri= ed=0A> default driver from kernel source and latest from intel=0A> site, ef= fect is=0A> the same with automatic interrupt moderation enabled and=0A> di= sabled. I=0A> have the same box with intel pro 1000 pt adapter which=0A> ha= ve=0A> tx(rx)_int_delay sysctls in em driver, I was able to reduce=0A> numb= er of=0A> tx/rx interrupts to 7-8k per interface and got much more=0A> cpu = idle=0A> because of less context switches with same pps.=0A> =0A> Interfaca= e load=0A> =0A> border# netstat -I igb0 -w 1=0A> =A0 =A0 =A0 =A0 =A0 =A0 in= put=A0=0A> =A0 =A0 =A0=A0=A0(igb0)=A0 =A0 =A0=0A> =A0 =A0=A0=A0output=0A> = =A0=A0=A0packets=A0 errs=A0 =A0 =A0=0A> bytes=A0 =A0 packets=A0 errs=A0 =A0= =A0=0A> bytes colls=0A> =A0 =A0=A0=A041438=A0=0A> =A0=A0=A00=A0=A0=A037923= 274=A0 =A0=0A> =A0 51173=A0=0A> =A0=A0=A00=A0=A0=A024539512=A0=0A> =A0=A0= =A00=0A> =A0 =A0=A0=A044827=A0=0A> =A0=A0=A00=A0=A0=A041626876=A0 =A0=0A> = =A0 53408=A0=0A> =A0=A0=A00=A0=A0=A024595412=A0=0A> =A0=A0=A00=0A> =A0 =A0= =A0=A043300=A0=0A> =A0=A0=A00=A0=A0=A039736056=A0 =A0=0A> =A0 53118=A0=0A> = =A0=A0=A00=A0=A0=A024574219=A0=0A> =A0=A0=A00=0A> =A0 =A0=A0=A043146=A0=0A>= =A0=A0=A00=A0=A0=A040399285=A0 =A0=0A> =A0 53455=A0=0A> =A0=A0=A00=A0=A0= =A024368290=A0=0A> =A0=A0=A00=0A> =A0 =A0=A0=A044827=A0=0A> =A0=A0=A00=A0= =A0=A042463307=A0 =A0=0A> =A0 53921=A0=0A> =A0=A0=A00=A0=A0=A023959752=A0= =0A> =A0=A0=A00=0A> =0A> Here is sysctls=0A> =0A> dev.igb.0.%desc: Intel(R)= PRO/1000 Network Connection=0A> version - 1.7.4=0A> dev.igb.0.%driver: igb= =0A> dev.igb.0.%location: slot=3D0 function=3D0=0A> dev.igb.0.%pnpinfo: ven= dor=3D0x8086 device=3D0x10c9=0A> subvendor=3D0x8086=0A> subdevice=3D0xa03c = class=3D0x020000=0A> dev.igb.0.%parent: pci1=0A> dev.igb.0.debug: -1=0A> de= v.igb.0.stats: -1=0A> dev.igb.0.flow_control: 0=0A> dev.igb.0.enable_aim: 1= =0A> dev.igb.0.low_latency: 1000=0A> dev.igb.0.ave_latency: 4000=0A> dev.ig= b.0.bulk_latency: 8000=0A> dev.igb.0.rx_processing_limit: 1000=0A> =0A> dev= .igb.1.%desc: Intel(R) PRO/1000 Network Connection=0A> version - 1.7.4=0A> = dev.igb.1.%driver: igb=0A> dev.igb.1.%location: slot=3D0 function=3D1=0A> d= ev.igb.1.%pnpinfo: vendor=3D0x8086 device=3D0x10c9=0A> subvendor=3D0x8086= =0A> subdevice=3D0xa03c class=3D0x020000=0A> dev.igb.1.%parent: pci1=0A> de= v.igb.1.debug: -1=0A> dev.igb.1.stats: -1=0A> dev.igb.1.flow_control: 0=0A>= dev.igb.1.enable_aim: 1=0A> dev.igb.1.low_latency: 1000=0A> dev.igb.1.ave_= latency: 4000=0A> dev.igb.1.bulk_latency: 8000=0A> dev.igb.1.rx_processing_= limit: 1000=0A> =0A> And debug =0A> =0A> kernel: igb0: Adapter hardware ad= dress =3D 0xc796ec1c =0A> kernel: igb0: CTRL =3D 0x40c00241 RCTL =3D 0x480= 02 =0A> kernel: igb0: Packet buffer =3D Tx=3D0k Rx=3D0k =0A> kernel: igb0= : Flow control watermarks high =3D 63488 low =3D=0A> 61988=0A> kernel: igb= 0: Queue(0) tdh =3D 3023, tdt =3D 3025=0A> kernel: igb0: TX(0) no descript= ors avail event =3D 0=0A> kernel: igb0: TX(0) MSIX IRQ Handled =3D 3754097= 484=0A> kernel: igb0: TX(0) Packets sent =3D 4815628967=0A> kernel: igb0:= Queue(0) rdh =3D 3658, rdt =3D 3645=0A> kernel: igb0: RX(0) Packets recei= ved =3D 7611879022=0A> kernel: igb0: RX(0) Split Packets =3D 0=0A> kernel= : igb0: RX(0) Byte count =3D 7013625984942=0A> kernel: igb0: RX(0) MSIX IR= Q Handled =3D 3232986641=0A> kernel: igb0: RX(0) LRO Queued=3D 0=0A> kern= el: igb0: RX(0) LRO Flushed=3D 0=0A> kernel: igb0: LINK MSIX IRQ Handled = =3D 3=0A> kernel: igb0: Mbuf defrag failed =3D 0=0A> kernel: igb0: Std mb= uf header failed =3D 0=0A> kernel: igb0: Std mbuf packet failed =3D 0=0A> = kernel: igb0: Driver dropped packets =3D 0=0A> kernel: igb0: Driver tx dm= a failure in xmit =3D 0=0A> =0A> kernel: igb1: Adapter hardware address = =3D 0xc796dc1c =0A> kernel: igb1: CTRL =3D 0x40c00241 RCTL =3D 0x48002 =0A= > kernel: igb1: Packet buffer =3D Tx=3D0k Rx=3D0k =0A> kernel: igb1: Flow= control watermarks high =3D 63488 low =3D=0A> 61988=0A> kernel: igb1: Que= ue(0) tdh =3D 4093, tdt =3D 4093=0A> kernel: igb1: TX(0) no descriptors av= ail event =3D 0=0A> kernel: igb1: TX(0) MSIX IRQ Handled =3D 10882048108= =0A> kernel: igb1: TX(0) Packets sent =3D 31169311987=0A> kernel: igb1: Q= ueue(0) rdh =3D 2515, rdt =3D 2513=0A> kernel: igb1: RX(0) Packets receive= d =3D 30747961847=0A> kernel: igb1: RX(0) Split Packets =3D 0=0A> kernel:= igb1: RX(0) Byte count =3D 26511993282060=0A> kernel: igb1: RX(0) MSIX IR= Q Handled =3D 4834518320=0A> kernel: igb1: RX(0) LRO Queued=3D 0=0A> kern= el: igb1: RX(0) LRO Flushed=3D 0=0A> kernel: igb1: LINK MSIX IRQ Handled = =3D 5=0A> kernel: igb1: Mbuf defrag failed =3D 0=0A> kernel: igb1: Std mb= uf header failed =3D 0=0A> kernel: igb1: Std mbuf packet failed =3D 0=0A> = kernel: igb1: Driver dropped packets =3D 0=0A> kernel: igb1: Driver tx dm= a failure in xmit =3D 0=0A> =0A> =0A> =0A=0AI'm curious as to why you are d= oing a load test on a single core system=0Awith a part that is clearly desi= gned to be used on a multicore system?=0A=0ABarney=0A=0A=0A