From owner-freebsd-net@FreeBSD.ORG Sat Jan 19 15:49:30 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BC00AD8C for ; Sat, 19 Jan 2013 15:49:30 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm14-vm1.bullet.mail.ne1.yahoo.com (nm14-vm1.bullet.mail.ne1.yahoo.com [98.138.91.38]) by mx1.freebsd.org (Postfix) with ESMTP id 5A61818F for ; Sat, 19 Jan 2013 15:49:30 +0000 (UTC) Received: from [98.138.226.180] by nm14.bullet.mail.ne1.yahoo.com with NNFMP; 19 Jan 2013 15:47:32 -0000 Received: from [98.138.89.252] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 19 Jan 2013 15:47:31 -0000 Received: from [127.0.0.1] by omp1044.mail.ne1.yahoo.com with NNFMP; 19 Jan 2013 15:47:31 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 898773.18503.bm@omp1044.mail.ne1.yahoo.com Received: (qmail 77235 invoked by uid 60001); 19 Jan 2013 15:47:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1358610451; bh=GrYf6rngHl5NjMVEGaHxyw3tBr2QqbjGIVyHFSJVEM4=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=sYFfHDQdhAfkxi32c13SoqES12liZweqIB7kFVgxSuUFUjyxHoqv6OTFGS/0SZaYZP7VGfWDbRJbsrmFN+eISFEgOjdlS99XxlWY3G0DjGcS0IAjrpfYBFrvY11H5QrYybzFbU6UU5kAqPHdkhaXWvW4ojbAKnK/p19sJgj9yLY= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=u5Ua//7/lUR0sdxFHhxn1TC1Pk1vRPQl5Plhgp6Ytec2AXP0w5Mo4zcS7KWhxPXYr7l302crY4HOezzoozOZn9I4+SLTo3jnHQYWgZerzLWvZJHwUGbUMthF0mqob3Itiuzg3eSnhvep5AJVN/63XiC1MZsVSEO6i8azY+TUoc8=; X-YMail-OSG: LSH_C8MVM1kHrjvKOA1vwwK9WIHpu8Z9WTAkBJYjPmv7hWf 5SFPVXY9WpNanC3pBVqecbd1IXraDBjR04AWT9Ke44FXXzK5oE85JWjCwwpP NDiR3HrI11J0CZkumFvn1_KS8Et1oZWk3eK7SE8faBDWMC6a.MH9vv1GfkJg e_NljK1c1c62tw8B3QME5voFwlamHckAn82fRvpgjbLH.RFbs6D.BHysMgLS AoNHqASARqI6PUWcFd0mULWHO6bRtprq60sDsWh0mX_aHIFQ.bujuTf.uVDt 4Jyujc2PrmabpkQPt0_i3_7NvUCXpCp_yhT5LKGPsCgEVnbGCWYizZFEDMaL q7FwWVXa1Kawpw85OTDf8Qt4GKrIYbTvxXLSZgWI_HT4njMlMPGaS2hdbhtF UIavX15fkVfK6b7jxt7F6.LhfIUnmdHwANiZPaGhykgqVpy.egpLWzkcNTQV By7kbzVcfqsHWmimEk2IXqRwcnwmkqE_WTv_8PG0a0e4qcZQqo2eppCxRfFK lS7Dewpbcty0OgOYgpBKakmhpbf1b Received: from [174.48.128.27] by web121604.mail.ne1.yahoo.com via HTTP; Sat, 19 Jan 2013 07:47:30 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gRnJpLCAxLzE4LzEzLCBKb2huIEJhbGR3aW4gPGpoYkBmcmVlYnNkLm9yZz4gd3JvdGU6Cgo.IEZyb206IEpvaG4gQmFsZHdpbiA8amhiQGZyZWVic2Qub3JnPgo.IFN1YmplY3Q6IFJlOiB0d28gcHJvYmxlbXMgaW4gZGV2L2UxMDAwL2lmX2xlbS5jOjpsZW1faGFuZGxlX3J4dHgoKQo.IFRvOiBmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZwo.IENjOiAiQmFybmV5IENvcmRvYmEiIDxiYXJuZXlfY29yZG9iYUB5YWhvby5jb20.LCAiQWRyaWFuIENoYWRkIiA8YWRyaWFuQGZyZWVic2Qub3JnPiwgIkwBMAEBAQE- X-Mailer: YahooMailClassic/15.1.2 YahooMailWebService/0.8.130.494 Message-ID: <1358610450.75691.YahooMailClassic@web121604.mail.ne1.yahoo.com> Date: Sat, 19 Jan 2013 07:47:30 -0800 (PST) From: Barney Cordoba Subject: Re: two problems in dev/e1000/if_lem.c::lem_handle_rxtx() To: freebsd-net@freebsd.org, John Baldwin In-Reply-To: <201301181149.42277.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Adrian Chadd , Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jan 2013 15:49:30 -0000 =0A=0A--- On Fri, 1/18/13, John Baldwin wrote:=0A=0A> Fro= m: John Baldwin =0A> Subject: Re: two problems in dev/e100= 0/if_lem.c::lem_handle_rxtx()=0A> To: freebsd-net@freebsd.org=0A> Cc: "Barn= ey Cordoba" , "Adrian Chadd" = , "Luigi Rizzo" =0A> Date: Friday, January 18, 2013, 11= :49 AM=0A> On Friday, January 18, 2013 9:30:40=0A> am Barney Cordoba wrote:= =0A> > =0A> > --- On Thu, 1/17/13, Adrian Chadd =0A> wr= ote:=0A> > =0A> > > From: Adrian Chadd =0A> > > Subject= : Re: two problems in=0A> dev/e1000/if_lem.c::lem_handle_rxtx()=0A> > > To:= "Barney Cordoba" =0A> > > Cc: "Luigi Rizzo" ,=0A> freebsd-net@freebsd.org=0A> > > Date: Thursday, Janua= ry 17, 2013, 11:48 AM=0A> > > There's also the subtle race=0A> > > conditio= n in TX and RX handling that=0A> > > re-queuing the taskqueue gets around.= =0A> > > =0A> > > Which is:=0A> > > =0A> > > * The hardware is constantly r= eceiving frames ,=0A> right until=0A> > > you blow=0A> > > the FIFO away by= filling it up;=0A> > > * The RX thread receives a bunch of frames;=0A> > >= * .. and processes them;=0A> > > * .. once it's done processing, the hardw= are may=0A> have read=0A> > > some more=0A> > > frames in the meantime;=0A>= > > * .. and the hardware may have generated a=0A> mitigated=0A> > > inter= rupt which=0A> > > you're ignoring, since you're processing frames;=0A> > >= * So if your architecture isn't 100% paranoid, you=0A> may end=0A> > > up = having=0A> > > to wait for the next interrupt to handle what's=0A> currentl= y in=0A> > > the=0A> > > queue.=0A> > > =0A> > > Now if things are done cor= rect:=0A> > > =0A> > > * The hardware generates a mitigated interrupt=0A> >= > * The mask register has that bit disabled, so you=0A> don't end=0A> > > = up receiving it;=0A> > > * You finish your RX queue processing, and there's= =0A> more=0A> > > stuff that's=0A> > > appeared in the FIFO (hence why the = hardware has=0A> generated=0A> > > another=0A> > > mitigated interrupt);=0A= > > > * You unmask the interrupt;=0A> > > * .. and the hardware immediately= sends you the=0A> MSI or=0A> > > signals an interrupt;=0A> > > * .. thus y= ou re-enter the RX processing thread=0A> almost(!)=0A> > > immediately.=0A>= > > =0A> > > However as the poster(s) have said, the interrupt=0A> > > mas= k/unmask in the=0A> > > intel driver(s) may not be 100% correct, so you're= =0A> going to=0A> > > end up=0A> > > with situations where interrupts are m= issed.=0A> > > =0A> > > The reason why this wasn't a big deal in the=0A> de= ep/distant=0A> > > past is=0A> > > because we didn't used to have kernel pr= eemption,=0A> or=0A> > > multiple kernel=0A> > > threads running, or an ove= rly aggressive scheduler=0A> trying=0A> > > to=0A> > > parallelise things a= s much as possible. A lot of=0A> > > net80211/ath bugs=0A> > > have popped = out of the woodwork specifically=0A> because of the=0A> > > above=0A> > > c= hanges to the kernel. They were bugs before, but=0A> people=0A> > > didn't = hit=0A> > > them.=0A> > > =0A> > =0A> > I don't see the distinction between= the rx thread=0A> getting re-scheduled=0A> > "immediately" vs introducing = another thread. In fact=0A> you increase missed=0A> > interrupts by this me= thod. The entire point of=0A> interrupt moderation is=0A> > to tune the int= ervals where a driver is processed.=0A> > =0A> > You might as well just not= have a work limit and=0A> process until your done.=0A> > The idea that "ge= e, I've been taking up too much cpu,=0A> I'd better yield"=0A> > to just qu= eue a task and continue soon after doesn't=0A> make much sense to =0A> > me= .=0A> =0A> If there are multiple threads with the same priority then=0A> ba= tching the work up =0A> into chunks allows the scheduler to round-robin amo= ng=0A> them.=A0 However, when a =0A> task requeues itself that doesn't actu= ally work since the=0A> taskqueue thread =0A> will see the requeued task be= fore it yields the CPU.=A0=0A> Alternatively, if you =0A> force all the rel= evant interrupt handlers to use the same=0A> thread pool and =0A> instead o= f requeueing a separate task you requeue your=0A> handler in the ithread = =0A> pool then you can get the desired round-robin=0A> behavior.=A0 (I have= changes to =0A> the ithread stuff that get us part of the way there in tha= t=0A> handlers can =0A> reschedule themselves and much of the plumbing is i= n place=0A> for shared thread =0A> pools among different interrupts.)=0A=0A= I dont see any "round robin" effect here. You have:=0A=0ARepeat:=0A- Proces= s 100 frames=0Aif (more)=0A- Queue a Task=0A=0Athere's only 1 task at a tim= e. All its really doing is yielding and=0Arescheduling itself to resume the= loop.=0A=0ABC