From owner-freebsd-net@FreeBSD.ORG Thu Dec 3 00:21:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 962EC1065696 for ; Thu, 3 Dec 2009 00:21:40 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63906.mail.re1.yahoo.com (web63906.mail.re1.yahoo.com [69.147.97.121]) by mx1.freebsd.org (Postfix) with SMTP id 37D248FC22 for ; Thu, 3 Dec 2009 00:21:39 +0000 (UTC) Received: (qmail 47054 invoked by uid 60001); 3 Dec 2009 00:21:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1259799698; bh=I9sTsREev3D8+LVzHxmw/lRFGuwx/eyJWpSp3vlAKCs=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=NTBj4+T2AX9cRNfCpM+BBh+7QeweZE3XWMgQ5ayXzgt3CxSqSuUQCttSnX/V++f4gmSJH5j9dzUV3UZbPkacKW3w4puy9QiTBUji1GOMgSJH+Bep2Bn2UyTBQaNeOu3f77YbjT4cEVj5G3fAhBt95xsfxOzpxSZHCUtgAja4ngE= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=KHnhAoR/upSejpF9nfl783M/wA6DlgDPwUJmtFi1s75THf4akZCW3i3z2rBaY1CJpHPsZBvHgnmo1P5zqrSlJ39peMihvLNUjLSU3vhk614BM8MqDbXhg7gDHHhnRCFo5kRhrEOW4g7C7yiE1A+J8z+WxV9HMmtmEDWjqa562/g=; Message-ID: <501706.46449.qm@web63906.mail.re1.yahoo.com> X-YMail-OSG: 96JUrEUVM1nH.R9Oh5gwF3AgYMn7.fB0RnVBiSdSvkxwv6kH0NOWpOiDbNI_7qv38PoOiOWy48es_4wPKZXkgQseVHYq7R.X_zJ6Tg5UCCIk82uWPdWfn_ae6J9K0NAb_IZYodFw1zDblygJGVpSEuWN.fn6QAOfFXqIOW.vqwvg4Ynroia_n6dHZR8S3glh6IZP7dINoiu3QOFSkVReK9iLwSwAcsbn3Wh_AeKxCjh8xonuJdcaRQ14xD5bfKRLGy7himymfgM2Ye1RNugtA9fEuXK_A4suKPNQ Received: from [98.203.21.152] by web63906.mail.re1.yahoo.com via HTTP; Wed, 02 Dec 2009 16:21:38 PST X-Mailer: YahooMailClassic/8.1.6 YahooMailWebService/0.8.100.260964 Date: Wed, 2 Dec 2009 16:21:38 -0800 (PST) From: Barney Cordoba To: "Yuriy A. Korobko" In-Reply-To: <1259769942.6739.33.camel@stormi-desktop> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Reducing number of interrupts from intel pro 1000 et adapter X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Dec 2009 00:21:40 -0000 =0A=0A--- On Wed, 12/2/09, Yuriy A. Korobko wrot= e:=0A=0A> From: Yuriy A. Korobko =0A> Subject: Re= : Reducing number of interrupts from intel pro 1000 et adapter=0A> To: "Bar= ney Cordoba" =0A> Cc: freebsd-net@freebsd.org=0A>= Date: Wednesday, December 2, 2009, 11:05 AM=0A> > > =0A> > > I'd like to k= now a way to control tx interrupts=0A> on intel=0A> > > pro 1000 et=0A> > >= adapter with igb driver. Just installed one in=0A> the router=0A> > > and = systat=0A> > > shows 8-9k rx interrupts and 20k tx interrupts=0A> from igb0= =0A> > > and igb1=0A> > > adapters. Box is a router running freebsd 7.2=0A>= release, I've=0A> > > tried=0A> > > default driver from kernel source and = latest from=0A> intel=0A> > > site, effect is=0A> > > the same with automat= ic interrupt moderation=0A> enabled and=0A> > > disabled. I=0A> > > have th= e same box with intel pro 1000 pt adapter=0A> which=0A> > > have=0A> > > tx= (rx)_int_delay sysctls in em driver, I was able=0A> to reduce=0A> > > numbe= r of=0A> > > tx/rx interrupts to 7-8k per interface and got=0A> much more= =0A> > > cpu idle=0A> > > because of less context switches with same pps.= =0A> > > =0A> =0A> =0A> =0A> > I'm curious as to why you are doing a load t= est on a=0A> single core system=0A> > with a part that is clearly designed = to be used on a=0A> multicore system?=0A> > =0A> > Barney=0A> =0A> Box have= core quad cpu:=0A> =0A> 87 processes:=A0 7 running, 58 sleeping, 22 waitin= g=0A> CPU 0:=A0 0.0% user,=A0 0.0% nice,=A0 3.1% system,=0A> 36.4% interrup= t, 60.5% idle=0A> CPU 1:=A0 0.0% user,=A0 0.0% nice, 15.5% system,=0A> 41.1= % interrupt, 43.4% idle=0A> CPU 2:=A0 0.0% user,=A0 0.0% nice, 11.6% system= ,=0A> 27.1% interrupt, 61.2% idle=0A> CPU 3:=A0 0.0% user,=A0 0.0% nice,=A0= 2.3% system,=0A> 41.5% interrupt, 56.2% idle=0A> Mem: 192M Active, 206M In= act, 246M Wired, 716K Cache, 112M=0A> Buf, 1356M =0A> =0A> Anyway, high int= errupt rate fixed by=0A> =0A> border# cat /boot/loader.conf | grep igb=0A> = if_igb_load=3D"YES"=0A> hw.igb.rxd=3D4096=0A> hw.igb.txd=3D4096=0A> hw.igb.= num_queues=3D1=0A> hw.igb.enable_aim=3D1=0A> hw.igb.low_latency=3D1000=0A> = hw.igb.ave_latency=3D2000=0A> hw.igb.bulk_latency=3D4000=0A> hw.igb.rx_proc= ess_limit=3D200=0A> hw.igb.fc_setting=3D0=0A> hw.igb.lro=3D0=0A> border# = =0A> =0A> Have no idea why, but it seems like setting dev.igb.=0A> sysctls = at runtime=0A> just ignored by the driver. Now I have 1000-1200 rx and=0A> = around 2000 tx=0A> interrupts per second from interface with 100 kpps flow = in=0A> each=0A> direction, main cpu eater on the router is dummynet right= =0A> now, it will=0A> be awesome if one day we can have it multithreaded. = =0A> =0A> I'm using latest igb driver from intel site, default driver=0A> f= rom kernel=0A> may not have this issue.=0A> =0A> Thanks.=0A> =0A> =0A> ____= ___________________________________________=0A> freebsd-net@freebsd.org=0A>= mailing list=0A> http://lists.freebsd.org/mailman/listinfo/freebsd-net=0A>= To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A>= =0A=0AAh, dummynet. Explains much. I've never seen livelock on a quad core= =0Asystem.=0A=0AThe igb driver in FreeBSD needs a complete and total overha= ul. Don't =0Awaste alot of time tweeking it. Turn off AIM and hard code som= ething=0Athat suits your network. Intel's aim algorithm is ill-conceived IM= O.=0A=0A=0ABarney=0A=0A=0A