From owner-freebsd-net@FreeBSD.ORG Wed Aug 19 18:43:06 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86B96106568F for ; Wed, 19 Aug 2009 18:43:06 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from n72.bullet.mail.sp1.yahoo.com (n72.bullet.mail.sp1.yahoo.com [98.136.44.34]) by mx1.freebsd.org (Postfix) with SMTP id 5B64C8FC3D for ; Wed, 19 Aug 2009 18:43:06 +0000 (UTC) Received: from [69.147.84.144] by n72.bullet.mail.sp1.yahoo.com with NNFMP; 19 Aug 2009 18:29:31 -0000 Received: from [68.142.200.227] by t6.bullet.mail.sp1.yahoo.com with NNFMP; 19 Aug 2009 18:29:31 -0000 Received: from [76.13.13.26] by t8.bullet.mud.yahoo.com with NNFMP; 19 Aug 2009 18:29:31 -0000 Received: from [76.13.10.172] by t3.bullet.mail.ac4.yahoo.com with NNFMP; 19 Aug 2009 18:29:31 -0000 Received: from [127.0.0.1] by omp113.mail.ac4.yahoo.com with NNFMP; 19 Aug 2009 18:29:31 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 91495.56173.bm@omp113.mail.ac4.yahoo.com Received: (qmail 18673 invoked by uid 60001); 19 Aug 2009 18:29:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1250706570; bh=QHn+CP2H9zm0O4TzcunGKCRV4eXa0t0nBTGZZ698X8Q=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=6ZOkiEGMBAtCRsf6mJKNigTRQy8G9mnFrv7Aajbwj8Hw78+kEGYSK3ezDQVElTaNm0SFU6nDC27vpjrceLNrjt/+PBzX6/wrKxFKmCUqJBC7yfefunbPzEsedmM4e92brqaX9cRZ6pGmOBnm5ELJtTcZY6/48KxQ6cjbUFYgB00= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=56y1BQ1s2fyBi0Z3iDS5QyvI5VD6xQZC6yJ9W6ROkgI6Z8zSdpkjRX3cQfXdaAWo/gZvoeMi7W+Hel3QRQkWiPYNHpIxE5Y+PFoX6sFePKq7zpjzASRGENoFJoOhxZGIs6dJ+A4zi6YNawt/O7MB70Yg4YytRNp0a5RyokL4mqQ=; Message-ID: <822688.18516.qm@web63903.mail.re1.yahoo.com> X-YMail-OSG: Fa_qbmwVM1m8N.ByR4nyFvFXdxtgrwYUoUJxjRsq54aDfN38DaPrbB8kQVMrmpc9bndHAxn93Ez350UifizXuN0X.89HTZ5Lxjrlp_rTnQCYZp_MF_jVrzTYOx98rxgi6I0Tb0bgkdRTNXZ1Kx74DWm6e8FOXduhyQC.P23MwbFZwP5.OQwgmUve5iwhriU8ZyI.4XrnEe5XRcOWp2s3z0PouccUWxfis8qFBbGUnK1YtzRA_JyD2LsSZlydsA_JYGz63R50OPCd5mf0Dk0ha3HVC5L3lvFysiRFUOrw2aD8OUHyMDIeKVSWpR94L9_DDliXiLsYDhaPngwze_CB_2gRCUHvSu_rlEVi9g2c Received: from [66.176.162.245] by web63903.mail.re1.yahoo.com via HTTP; Wed, 19 Aug 2009 11:29:30 PDT X-Mailer: YahooMailClassic/6.1.2 YahooMailWebService/0.7.338.1 Date: Wed, 19 Aug 2009 11:29:30 -0700 (PDT) From: Barney Cordoba To: Manish Vachharajani In-Reply-To: <5bc218350908181535o7c5275dfn2f6647454cfac804@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Dropped vs. missed packets in the ixgbe driver X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2009 18:43:06 -0000 =0A=0A--- On Tue, 8/18/09, Manish Vachharajani wrote:=0A=0A> From: Manish Vachharajani =0A>= Subject: Re: Dropped vs. missed packets in the ixgbe driver=0A> To: "Barne= y Cordoba" =0A> Cc: freebsd-net@freebsd.org=0A> D= ate: Tuesday, August 18, 2009, 6:35 PM=0A> Indeed the debugging info is als= o=0A> interesting.=A0 However, I'd like to=0A> get some data from netstat w= hen the driver drops=0A> frames.=A0 When looking=0A> at the bge driver sour= ce, it appears that all input drops=0A> at the NIC=0A> are reported as inpu= t errors.=A0 It appears that the=0A> intel drivers=0A> (e1000, ixgb, and ix= gbe) don't report that number outside=0A> of the debug=0A> and stats printf= s at all, and this seems broken.=A0 What=0A> I want to know=0A> is if I hav= e just missed where these are reported.=0A> =0A> So, in a nutshell, the que= stion is:=A0 should these=0A> drivers be reporting=0A> miss events as input= errors in the ifnet struct as the bge=0A> driver=0A> does, or as drops in = the ifnet struct, was there some=0A> conscious=0A> decision not to report m= iss events anywhere outside the=0A> debug and=0A> stats info, or am I just = being silly and not seeing where=0A> the numbers=0A> are reported?=0A> =0A>= Also, don't worry on the performance front, we are also=0A> looking at the= =0A> driver in FreeBSD 8.0 :) which supports RSS to help=0A> performance=0A= > scaling,=A0 though we have some interesting data there=0A> that I'll post= =0A> about once I confirm that the numbers are indeed correct=0A> and not a= =0A> tuning or setup problem.=0A> =0A> Manish=0A> =0A> > --- On Mon, 8/17/0= 9, Manish Vachharajani =0A> wrote:=0A> >=0A> >= > From: Manish Vachharajani =0A> >> Subject: D= ropped vs. missed packets in the ixgbe=0A> driver=0A> >> To: freebsd-net@fr= eebsd.org=0A> >> Date: Monday, August 17, 2009, 6:24 PM=0A> >> I've been do= ing some performance=0A> >> testing on freebsd 7.2 and noticed=0A> >> that = the ixgbe driver does not report missed=0A> packets as=0A> >> dropped when= =0A> >> queried via netstat -id (the ixgbe driver in the=0A> 8.0=0A> >> bet= a has a=0A> >> similar issue).=A0 A missed packet is a packet that=0A> was= =0A> >> correctly=0A> >> received by the NIC but because it was out of=0A> = descriptors=0A> >> and internal=0A> >> memory, the packet had to be dropped= by the NIC=0A> itself -- a=0A> >> hardware=0A> >> register counts such eve= nts.=A0 Instead the driver=0A> only=0A> >> reports drops=0A> >> that are du= e to the driver itself, i.e., the NIC=0A> DMAed the=0A> >> packet to=0A> >>= memory but the driver had to drop something=0A> because it was=0A> >> out = of=0A> >> mbufs.=A0 Is the miss count reported elsewhere=0A> (besides=0A> >= > via a kernel=0A> >> printf from the driver)?=A0 At the end of the email= =0A> I=0A> >> give a stats dump=0A> >> from the driver and from netstat, no= te the number=0A> of=0A> >> packets dropped=0A> >> is 0 as reported by nets= tat but the Missed field=0A> in the=0A> >> dmesg output=0A> >> shows many m= issed packets.=0A> >>=0A> >> >From my perspective, it is disconcerting to= =0A> see=0A> >> performance=0A> >> degradation on the link, along with TCP = ack=0A> retransmits,=0A> >> packet=0A> >> reordering, etc. (on a point-to-p= oint link with no=0A> switch=0A> >> in between)=0A> >> but then see no drop= s reported by netstat because=0A> the=0A> >> driver didn't=0A> >> drop the = packet, the NIC did.=A0 The fix should be=0A> >> straight-forward and=0A> >= > I'll gladly make a patch assuming that it is=0A> indeed a bug=0A> >> and = not a=0A> >> conscious design choice.=0A> >>=0A> >> Here is the relevant ne= tstat output=0A> >>=0A> >> Name=A0 =A0 Mtu Network=0A> >> =A0=A0=A0Address= =0A> >> =A0 =A0 Ipkts Ierrs=A0 =A0 Opkts=0A> >> Oerrs=A0 Coll Drop=0A> >> i= x0=A0 =A0 1500 =0A> >> 00:30:48:94:60:ec=A0 =A0 =A0 =A0 0=0A> >> = =A0=A0=A00=A0 =A0 =A0 =A0 1=0A> >> =A00=A0 =A0=A0=A00=A0 =A0 0=0A> >> ix0= =A0 =A0 1500 192.168.105.0 192.168.105.2=0A> >> =A0 =A0 =A0 =A0 =A0 0=0A> >= > =A0=A0=A0-=A0 =A0 =A0 =A0 0=0A> >> =A0-=A0 =A0=A0=A0-=A0 =A0 -=0A=0A=0Aif= you look in ixgbe_update_stats_counters at the bottom:=0A=0A ifp->i= f_ierrors =3D missed_rx + adapter->stats.crcerrs +=0A adapte= r->stats.rlec;=0A=0Athe errors are added in. =0A=0ABC=0A> >> 00:30:48:94:60= :ed=A0 =A0 =A0 11M=0A> >> =A0=A0=A00=A0 =A0=A0=A06.1M=0A> >> =A00=A0 =A0=A0= =A00=A0 =A0 0=0A> >> ix1=A0 =A0 1500=0A> >> 192.168.5.0=A0=A0=A0192.168.5.2= =0A> >> =A0 =A0 =A0 10M=A0 =A0=A0=A0-=0A> >> =A0=A0=A06.1M=0A> >> =A0-=A0 = =A0=A0=A0-=A0 =A0 -=0A> >>=0A> >> And here is the dmesg output after doing = a sysctl=0A> >> dev.ix.1.stats=3D1=0A> >>=0A> >> ix1: Std Mbuf Failed =3D 0= =0A> >> ix1: Missed Packets =3D 413872=0A> >> ix1: Receive length errors = =3D 0=0A> >> ix1: Crc errors =3D 0=0A> >> ix1: Driver dropped packets =3D 0= =0A> >> ix1: watchdog timeouts =3D 0=0A> >> ix1: XON Rcvd =3D 616428212235= =0A> >> ix1: XON Xmtd =3D 0=0A> >> ix1: XOFF Rcvd =3D 616428212235=0A> >> i= x1: XOFF Xmtd =3D 0=0A> >> ix1: Total Packets Rcvd =3D 12424533=0A> >> ix1:= Good Packets Rcvd =3D 12010661=0A> >> ix1: Good Packets Xmtd =3D 6419128= =0A> >> ix1: TSO Transmissions =3D 0=0A> >>=0A> >> Manish=0A> >=0A> > the d= ebug sysctl show more interesting info. Don't get=0A> too excited=0A> > abo= ut doing performance testing with that driver. Its=0A> not designed=0A> > t= o be any higher in performance than any of the other=0A> intel drivers.=0A>= >=0A> > Barney=0A> >=0A> >=0A> >=0A> >=0A> =0A> =0A> =0A> -- =0A> Manish V= achharajani=0A> Founder=0A> LineRate Systems=0A> manishv@lineratesystems.co= m=0A> (609)635-9531 M=0A> _______________________________________________= =0A> freebsd-net@freebsd.org=0A> mailing list=0A> http://lists.freebsd.org/= mailman/listinfo/freebsd-net=0A> To unsubscribe, send any mail to "freebsd-= net-unsubscribe@freebsd.org"=0A> =0A=0A=0A