Date: Tue, 8 Sep 2009 14:42:39 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: alexpalias-bsdnet@yahoo.com, Manish Vachharajani <manishv@lineratesystems.com> Cc: freebsd-net@freebsd.org, Artis Caune <artis.caune@gmail.com> Subject: Re: em driver input errors Message-ID: <324031.44935.qm@web63904.mail.re1.yahoo.com> In-Reply-To: <5bc218350909041041x49ec9765k81346e90bbfe891a@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Fri, 9/4/09, Manish Vachharajani <manishv@lineratesystems.com>= wrote:=0A=0A> From: Manish Vachharajani <manishv@lineratesystems.com>=0A> = Subject: Re: em driver input errors=0A> To: alexpalias-bsdnet@yahoo.com=0A>= Cc: freebsd-net@freebsd.org, "Artis Caune" <artis.caune@gmail.com>=0A> Dat= e: Friday, September 4, 2009, 1:41 PM=0A> Just decided to follow this threa= d as=0A> it seems to be related to some=0A> issues we are seeing as well.= =0A> =0A> It appears that under heavy packet loads, the kernel cannot=0A> p= ull=0A> packets off the NIC fast enough and thus is slow to free=0A> up=0A>= descriptors into which the NIC can DMA packets.=A0 This=0A> causes the NIC= =0A> to drop the packet after it's internal queue fills up (and=0A> record = the=0A> packet as missed) because the hardware does not have=0A> enough=0A>= descriptors to write the packets into.=A0 We ahve this=0A> issue with the= =0A> ixgbe 10 Gb/s card though the absolute packet rates at=0A> which we se= e a=0A> problem are higher than those reported here.=0A> =0A> In our test s= cenario the problem gets worse with many=0A> simultaneous TCP=0A> connectio= ns, but the issue is the same.=A0 Under high=0A> packet rates, the=0A> driv= er cannot keep up and the NIC reports missed=0A> packets.=A0 The issue=0A> = is not related to data throughput though as turning on=0A> jumbo frames=0A>= solves our issue for a fixed number of connections, and it=0A> seems here= =0A> that reducing the packet rate makes the misses go=0A> away.=A0 More=0A= > importantly, in our tests, only the receiver sees a=0A> problem, the=0A> = transmitter is fine.=0A> =0A> There was also another thread about problems = with UDP=0A> throughput that=0A> I suspect are caused by the same type of p= acket rate=0A> spikes.=0A> =0A> The question is, why is the kernel stack sl= ow to handle=0A> these packet=0A> rates, doing some back of the envelope ca= lculations, they=0A> don't seem=0A> too bad?=A0 Where is the time going?=A0= And, are our=0A> problem, the UDP=0A> issue, and this problem all caused b= y the same source of=0A> slowness or=0A> are they three unrelated issues.= =0A> =0A> Manish=0A=0AWhat specific kinds of input errors are you getting?= =0A=0AHow many PPS are you doing, what is the size of the ring, and=0Athe i= nterrupt modulation rate?=0A=0AAre the NICs PCIe or PCIx?=0A=0ABarney=0A=0A= =0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?324031.44935.qm>