Date: Wed, 2 Dec 2009 16:21:38 -0800 (PST) From: Barney Cordoba <barney_cordoba@yahoo.com> To: "Yuriy A. Korobko" <administrator@shtorm.com> Cc: freebsd-net@freebsd.org Subject: Re: Reducing number of interrupts from intel pro 1000 et adapter Message-ID: <501706.46449.qm@web63906.mail.re1.yahoo.com> In-Reply-To: <1259769942.6739.33.camel@stormi-desktop>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Wed, 12/2/09, Yuriy A. Korobko <administrator@shtorm.com> wrot= e:=0A=0A> From: Yuriy A. Korobko <administrator@shtorm.com>=0A> Subject: Re= : Reducing number of interrupts from intel pro 1000 et adapter=0A> To: "Bar= ney Cordoba" <barney_cordoba@yahoo.com>=0A> Cc: freebsd-net@freebsd.org=0A>= Date: Wednesday, December 2, 2009, 11:05 AM=0A> > > =0A> > > I'd like to k= now a way to control tx interrupts=0A> on intel=0A> > > pro 1000 et=0A> > >= adapter with igb driver. Just installed one in=0A> the router=0A> > > and = systat=0A> > > shows 8-9k rx interrupts and 20k tx interrupts=0A> from igb0= =0A> > > and igb1=0A> > > adapters. Box is a router running freebsd 7.2=0A>= release, I've=0A> > > tried=0A> > > default driver from kernel source and = latest from=0A> intel=0A> > > site, effect is=0A> > > the same with automat= ic interrupt moderation=0A> enabled and=0A> > > disabled. I=0A> > > have th= e same box with intel pro 1000 pt adapter=0A> which=0A> > > have=0A> > > tx= (rx)_int_delay sysctls in em driver, I was able=0A> to reduce=0A> > > numbe= r of=0A> > > tx/rx interrupts to 7-8k per interface and got=0A> much more= =0A> > > cpu idle=0A> > > because of less context switches with same pps.= =0A> > > =0A> =0A> =0A> =0A> > I'm curious as to why you are doing a load t= est on a=0A> single core system=0A> > with a part that is clearly designed = to be used on a=0A> multicore system?=0A> > =0A> > Barney=0A> =0A> Box have= core quad cpu:=0A> =0A> 87 processes:=A0 7 running, 58 sleeping, 22 waitin= g=0A> CPU 0:=A0 0.0% user,=A0 0.0% nice,=A0 3.1% system,=0A> 36.4% interrup= t, 60.5% idle=0A> CPU 1:=A0 0.0% user,=A0 0.0% nice, 15.5% system,=0A> 41.1= % interrupt, 43.4% idle=0A> CPU 2:=A0 0.0% user,=A0 0.0% nice, 11.6% system= ,=0A> 27.1% interrupt, 61.2% idle=0A> CPU 3:=A0 0.0% user,=A0 0.0% nice,=A0= 2.3% system,=0A> 41.5% interrupt, 56.2% idle=0A> Mem: 192M Active, 206M In= act, 246M Wired, 716K Cache, 112M=0A> Buf, 1356M =0A> =0A> Anyway, high int= errupt rate fixed by=0A> =0A> border# cat /boot/loader.conf | grep igb=0A> = if_igb_load=3D"YES"=0A> hw.igb.rxd=3D4096=0A> hw.igb.txd=3D4096=0A> hw.igb.= num_queues=3D1=0A> hw.igb.enable_aim=3D1=0A> hw.igb.low_latency=3D1000=0A> = hw.igb.ave_latency=3D2000=0A> hw.igb.bulk_latency=3D4000=0A> hw.igb.rx_proc= ess_limit=3D200=0A> hw.igb.fc_setting=3D0=0A> hw.igb.lro=3D0=0A> border# = =0A> =0A> Have no idea why, but it seems like setting dev.igb.=0A> sysctls = at runtime=0A> just ignored by the driver. Now I have 1000-1200 rx and=0A> = around 2000 tx=0A> interrupts per second from interface with 100 kpps flow = in=0A> each=0A> direction, main cpu eater on the router is dummynet right= =0A> now, it will=0A> be awesome if one day we can have it multithreaded. = =0A> =0A> I'm using latest igb driver from intel site, default driver=0A> f= rom kernel=0A> may not have this issue.=0A> =0A> Thanks.=0A> =0A> =0A> ____= ___________________________________________=0A> freebsd-net@freebsd.org=0A>= mailing list=0A> http://lists.freebsd.org/mailman/listinfo/freebsd-net=0A>= To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A>= =0A=0AAh, dummynet. Explains much. I've never seen livelock on a quad core= =0Asystem.=0A=0AThe igb driver in FreeBSD needs a complete and total overha= ul. Don't =0Awaste alot of time tweeking it. Turn off AIM and hard code som= ething=0Athat suits your network. Intel's aim algorithm is ill-conceived IM= O.=0A=0A=0ABarney=0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?501706.46449.qm>