From owner-freebsd-current@FreeBSD.ORG Tue Mar 9 22:07:28 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 029ED106566B; Tue, 9 Mar 2010 22:07:28 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from mms1.broadcom.com (mms1.broadcom.com [216.31.210.17]) by mx1.freebsd.org (Postfix) with ESMTP id C5DA48FC18; Tue, 9 Mar 2010 22:07:27 +0000 (UTC) Received: from [10.9.200.131] by mms1.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Tue, 09 Mar 2010 14:04:58 -0800 X-Server-Uuid: 02CED230-5797-4B57-9875-D5D2FEE4708A Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.30]) by IRVEXCHHUB01.corp.ad.broadcom.com ([10.9.200.131]) with mapi; Tue, 9 Mar 2010 14:04:57 -0800 From: "David Christensen" To: "pyunyh@gmail.com" Date: Tue, 9 Mar 2010 14:04:57 -0800 Thread-Topic: dev.bce.X.com_no_buffers increasing and packet loss Thread-Index: Acq/0SUjKGwie/fPR2uZKx1IeUMCyQAAD6Xg Message-ID: <5D267A3F22FD854F8F48B3D2B52381933AF90EEDA7@IRVEXCHCCR01.corp.ad.broadcom.com> References: <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> In-Reply-To: <20100309214012.GQ1311@michelle.cdnetworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-WSS-ID: 6788198020S53434302-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Cc: Ian FREISLICH , "current@freebsd.org" , David Christensen Subject: RE: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 22:07:28 -0000 > > > > It's been running for about 1:23 on the patched driver.=20 > I'm still=20 > > > > seeing the com_no_buffers increase: > > > >=20 > > > > [firewall2.jnb1] ~ # sysctl dev.bce |grep com_no_buffers > > > > dev.bce.0.com_no_buffers: 5642 > > > > dev.bce.1.com_no_buffers: 497 > > > > dev.bce.2.com_no_buffers: 6260612 > > > > dev.bce.3.com_no_buffers: 4871338 > > > >=20 > > >=20 > > > Still have no idea why these counters are increasing here. > > > Actually the counter is read from scratch pad of completion=20 > > > processor. The datasheet does not even document the counter. > > > Maybe david know better what's happening here(CCed). > > >=20 > > > > Interupt rate is down now, at about 3500 per second per=20 > interface. > > > >=20 > > > > Interestingly setting net.inet.ip.fastforwarding=3D0 reduces CPU=20 > > > > consumption from 25% to 9% and less packet loss. > >=20 > > The com_no_buffers statistic comes from firmware and indicates how=20 > > many times a valid frame was received but could not be=20 > placed into the=20 > > receive chain because there were no available RX buffers. The=20 > > firmware will then drop the frame but that dropped frame won't be=20 > > reflected in any of the hardware based statistics. > >=20 >=20 > Yeah, but the question is why bce(4) has no available RX buffers. > The system has a lot of available mbufs so I don't see the=20 > root cause here. What's the traffic look like? Jumbo, standard, short frames? Any=20 good ideas on profiling the code? I haven't figured out how to use the CPU TSC but there is a free running timer on the device that might be usable to calculate where the driver's time is spent. Dave=