From owner-freebsd-net@FreeBSD.ORG Tue Sep 8 21:42:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D50B1065670 for ; Tue, 8 Sep 2009 21:42:40 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63904.mail.re1.yahoo.com (web63904.mail.re1.yahoo.com [69.147.97.119]) by mx1.freebsd.org (Postfix) with SMTP id D754F8FC20 for ; Tue, 8 Sep 2009 21:42:39 +0000 (UTC) Received: (qmail 44988 invoked by uid 60001); 8 Sep 2009 21:42:39 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252446159; bh=oXv/q1uxwPOClHHdgTilGBS/T6I+ursjoPhQK0sZhck=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=OQrPVI6yoMZ2FB+gTfmquW6h0iTCu0/RIFb0QPz56cd4T9CieCCXVWWYFDUJSPq0nFWvr5kYr3aJPu1DFOjFP7XoAOR7zjboyouW8iGjSqU/T7ekxmhpckaB3ZpE0QkydN7O3kXXgDWApDuOfLaEM3y9qM9CQiasUazAB/Oq7II= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=J3WKpG6kHJvdoGURK25q9KzCFL4dPTkXswCCboVDN4mjOB+bpHH9IaQG6RhmN0IAwvZ7BoamUSjL6HOtdyLkirdMEhwUAuLXiEmgAX1jCEOPAB7QzzJ5UbwyVsa8N5OF3AdSzlH5SUbVk1eFXELcBkBpnjFugT1aRRmasycK/Ik=; Message-ID: <324031.44935.qm@web63904.mail.re1.yahoo.com> X-YMail-OSG: UszZVoYVM1m3RegbJlCsbRs2gA_R_nnAt5jHRsY5CLIfkeAmx6dcin1bWAKcrmrQ5Z4pz5uNYrixvnL8joJTjegmwAqueOxE_nF.m9XWx5Z8_EFYFarz0eZ8CIETqwOgbrCD8TO7KsrwuVMlyzHscn2i8pvFixw8JXq1vEsvU4cMoC36075vM3okuyaXRw.1n4MBCcuDOjO.4XfdCu0r4ihMDCVDUlw.YFiV87GRlc8YejZ2pCyY.V0bcxaN1WQGIc6wLEjPWFN_A3KOoVqkJ0M- Received: from [98.203.21.152] by web63904.mail.re1.yahoo.com via HTTP; Tue, 08 Sep 2009 14:42:39 PDT X-Mailer: YahooMailClassic/6.1.2 YahooMailWebService/0.7.338.2 Date: Tue, 8 Sep 2009 14:42:39 -0700 (PDT) From: Barney Cordoba To: alexpalias-bsdnet@yahoo.com, Manish Vachharajani In-Reply-To: <5bc218350909041041x49ec9765k81346e90bbfe891a@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, Artis Caune Subject: Re: em driver input errors X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2009 21:42:40 -0000 =0A=0A--- On Fri, 9/4/09, Manish Vachharajani = wrote:=0A=0A> From: Manish Vachharajani =0A> = Subject: Re: em driver input errors=0A> To: alexpalias-bsdnet@yahoo.com=0A>= Cc: freebsd-net@freebsd.org, "Artis Caune" =0A> Dat= e: Friday, September 4, 2009, 1:41 PM=0A> Just decided to follow this threa= d as=0A> it seems to be related to some=0A> issues we are seeing as well.= =0A> =0A> It appears that under heavy packet loads, the kernel cannot=0A> p= ull=0A> packets off the NIC fast enough and thus is slow to free=0A> up=0A>= descriptors into which the NIC can DMA packets.=A0 This=0A> causes the NIC= =0A> to drop the packet after it's internal queue fills up (and=0A> record = the=0A> packet as missed) because the hardware does not have=0A> enough=0A>= descriptors to write the packets into.=A0 We ahve this=0A> issue with the= =0A> ixgbe 10 Gb/s card though the absolute packet rates at=0A> which we se= e a=0A> problem are higher than those reported here.=0A> =0A> In our test s= cenario the problem gets worse with many=0A> simultaneous TCP=0A> connectio= ns, but the issue is the same.=A0 Under high=0A> packet rates, the=0A> driv= er cannot keep up and the NIC reports missed=0A> packets.=A0 The issue=0A> = is not related to data throughput though as turning on=0A> jumbo frames=0A>= solves our issue for a fixed number of connections, and it=0A> seems here= =0A> that reducing the packet rate makes the misses go=0A> away.=A0 More=0A= > importantly, in our tests, only the receiver sees a=0A> problem, the=0A> = transmitter is fine.=0A> =0A> There was also another thread about problems = with UDP=0A> throughput that=0A> I suspect are caused by the same type of p= acket rate=0A> spikes.=0A> =0A> The question is, why is the kernel stack sl= ow to handle=0A> these packet=0A> rates, doing some back of the envelope ca= lculations, they=0A> don't seem=0A> too bad?=A0 Where is the time going?=A0= And, are our=0A> problem, the UDP=0A> issue, and this problem all caused b= y the same source of=0A> slowness or=0A> are they three unrelated issues.= =0A> =0A> Manish=0A=0AWhat specific kinds of input errors are you getting?= =0A=0AHow many PPS are you doing, what is the size of the ring, and=0Athe i= nterrupt modulation rate?=0A=0AAre the NICs PCIe or PCIx?=0A=0ABarney=0A=0A= =0A=0A=0A