From owner-freebsd-net@FreeBSD.ORG Tue Aug 18 22:35:08 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F1EC106568D for ; Tue, 18 Aug 2009 22:35:08 +0000 (UTC) (envelope-from manishv@lineratesystems.com) Received: from mail-vw0-f180.google.com (mail-vw0-f180.google.com [209.85.212.180]) by mx1.freebsd.org (Postfix) with ESMTP id 260768FC59 for ; Tue, 18 Aug 2009 22:35:07 +0000 (UTC) Received: by vws10 with SMTP id 10so3427004vws.7 for ; Tue, 18 Aug 2009 15:35:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.82.2 with SMTP id z2mr7903159vck.0.1250634904901; Tue, 18 Aug 2009 15:35:04 -0700 (PDT) In-Reply-To: <373149.52091.qm@web63907.mail.re1.yahoo.com> References: <5bc218350908171524m5a46c3dbm3e6af625c51370d0@mail.gmail.com> <373149.52091.qm@web63907.mail.re1.yahoo.com> Date: Tue, 18 Aug 2009 16:35:04 -0600 Message-ID: <5bc218350908181535o7c5275dfn2f6647454cfac804@mail.gmail.com> From: Manish Vachharajani To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Dropped vs. missed packets in the ixgbe driver X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2009 22:35:08 -0000 Indeed the debugging info is also interesting. However, I'd like to get some data from netstat when the driver drops frames. When looking at the bge driver source, it appears that all input drops at the NIC are reported as input errors. It appears that the intel drivers (e1000, ixgb, and ixgbe) don't report that number outside of the debug and stats printfs at all, and this seems broken. What I want to know is if I have just missed where these are reported. So, in a nutshell, the question is: should these drivers be reporting miss events as input errors in the ifnet struct as the bge driver does, or as drops in the ifnet struct, was there some conscious decision not to report miss events anywhere outside the debug and stats info, or am I just being silly and not seeing where the numbers are reported? Also, don't worry on the performance front, we are also looking at the driver in FreeBSD 8.0 :) which supports RSS to help performance scaling, though we have some interesting data there that I'll post about once I confirm that the numbers are indeed correct and not a tuning or setup problem. Manish > --- On Mon, 8/17/09, Manish Vachharajani wr= ote: > >> From: Manish Vachharajani >> Subject: Dropped vs. missed packets in the ixgbe driver >> To: freebsd-net@freebsd.org >> Date: Monday, August 17, 2009, 6:24 PM >> I've been doing some performance >> testing on freebsd 7.2 and noticed >> that the ixgbe driver does not report missed packets as >> dropped when >> queried via netstat -id (the ixgbe driver in the 8.0 >> beta has a >> similar issue).=A0 A missed packet is a packet that was >> correctly >> received by the NIC but because it was out of descriptors >> and internal >> memory, the packet had to be dropped by the NIC itself -- a >> hardware >> register counts such events.=A0 Instead the driver only >> reports drops >> that are due to the driver itself, i.e., the NIC DMAed the >> packet to >> memory but the driver had to drop something because it was >> out of >> mbufs.=A0 Is the miss count reported elsewhere (besides >> via a kernel >> printf from the driver)?=A0 At the end of the email I >> give a stats dump >> from the driver and from netstat, note the number of >> packets dropped >> is 0 as reported by netstat but the Missed field in the >> dmesg output >> shows many missed packets. >> >> >From my perspective, it is disconcerting to see >> performance >> degradation on the link, along with TCP ack retransmits, >> packet >> reordering, etc. (on a point-to-point link with no switch >> in between) >> but then see no drops reported by netstat because the >> driver didn't >> drop the packet, the NIC did.=A0 The fix should be >> straight-forward and >> I'll gladly make a patch assuming that it is indeed a bug >> and not a >> conscious design choice. >> >> Here is the relevant netstat output >> >> Name=A0 =A0 Mtu Network >> =A0=A0=A0Address >> =A0 =A0 Ipkts Ierrs=A0 =A0 Opkts >> Oerrs=A0 Coll Drop >> ix0=A0 =A0 1500 >> 00:30:48:94:60:ec=A0 =A0 =A0 =A0 0 >> =A0=A0=A00=A0 =A0 =A0 =A0 1 >> =A00=A0 =A0=A0=A00=A0 =A0 0 >> ix0=A0 =A0 1500 192.168.105.0 192.168.105.2 >> =A0 =A0 =A0 =A0 =A0 0 >> =A0=A0=A0-=A0 =A0 =A0 =A0 0 >> =A0-=A0 =A0=A0=A0-=A0 =A0 - >> ix1=A0 =A0 1500 >> 00:30:48:94:60:ed=A0 =A0 =A0 11M >> =A0=A0=A00=A0 =A0=A0=A06.1M >> =A00=A0 =A0=A0=A00=A0 =A0 0 >> ix1=A0 =A0 1500 >> 192.168.5.0=A0=A0=A0192.168.5.2 >> =A0 =A0 =A0 10M=A0 =A0=A0=A0- >> =A0=A0=A06.1M >> =A0-=A0 =A0=A0=A0-=A0 =A0 - >> >> And here is the dmesg output after doing a sysctl >> dev.ix.1.stats=3D1 >> >> ix1: Std Mbuf Failed =3D 0 >> ix1: Missed Packets =3D 413872 >> ix1: Receive length errors =3D 0 >> ix1: Crc errors =3D 0 >> ix1: Driver dropped packets =3D 0 >> ix1: watchdog timeouts =3D 0 >> ix1: XON Rcvd =3D 616428212235 >> ix1: XON Xmtd =3D 0 >> ix1: XOFF Rcvd =3D 616428212235 >> ix1: XOFF Xmtd =3D 0 >> ix1: Total Packets Rcvd =3D 12424533 >> ix1: Good Packets Rcvd =3D 12010661 >> ix1: Good Packets Xmtd =3D 6419128 >> ix1: TSO Transmissions =3D 0 >> >> Manish > > the debug sysctl show more interesting info. Don't get too excited > about doing performance testing with that driver. Its not designed > to be any higher in performance than any of the other intel drivers. > > Barney > > > > --=20 Manish Vachharajani Founder LineRate Systems manishv@lineratesystems.com (609)635-9531 M