From owner-freebsd-current@FreeBSD.ORG Wed Mar 10 19:11:22 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BE7F106564A for ; Wed, 10 Mar 2010 19:11:22 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from MMS3.broadcom.com (mms3.broadcom.com [216.31.210.19]) by mx1.freebsd.org (Postfix) with ESMTP id 503038FC29 for ; Wed, 10 Mar 2010 19:11:22 +0000 (UTC) Received: from [10.9.200.131] by MMS3.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Wed, 10 Mar 2010 11:11:14 -0800 X-Server-Uuid: B55A25B1-5D7D-41F8-BC53-C57E7AD3C201 Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.30]) by IRVEXCHHUB01.corp.ad.broadcom.com ([10.9.200.131]) with mapi; Wed, 10 Mar 2010 11:11:14 -0800 From: "David Christensen" To: "Ian FREISLICH" Date: Wed, 10 Mar 2010 11:11:13 -0800 Thread-Topic: dev.bce.X.com_no_buffers increasing and packet loss Thread-Index: AcrAMPUTon1v+XjlSqCBqmDnLEAN2wAUgwOQ Message-ID: <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> References: <5D267A3F22FD854F8F48B3D2B52381933AF90EEDA7@IRVEXCHCCR01.corp.ad.broadcom.com> <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-WSS-ID: 6789315831G55615605-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Cc: "pyunyh@gmail.com" , "current@freebsd.org" Subject: RE: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 19:11:22 -0000 > > What's the traffic look like? Jumbo, standard, short=20 > frames? Any=3D20=20 > > good ideas on profiling the code? I haven't figured out how to use=20 > > the CPU TSC but there is a free running timer on the device=20 > that might=20 > > be usable to calculate where the driver's time is spent. >=20 > It looks like the traffic that provoked it was this: >=20 > 10:18:42.319370 IP X.4569 > X.4569: UDP, length 12 > 10:18:42.319402 IP X.4569 > X.4569: UDP, length 12 > 10:18:42.319438 IP X.4569 > X.4569: UDP, length 12 > 10:18:42.319484 IP X.4569 > X.4569: UDP, length 12 > 10:18:42.319517 IP X.4569 > X.4569: UDP, length 12 >=20 > A flurry of UDP tinygrams on an IAX2 trunk. The packet rate=20 > isn't spectacular at about 30kpps which on top of the base=20 > load of 60kpps still isn't a fantastic packet rate. The=20 > interesting thing is that while this storm was inprogress, it=20 > almost entirely excluded other traffic on the network. Ok, small packet performance is involved, this narrows down the range of problems. The current design of bce_rx_intr() attempts to process all RX frames in the receive ring. After all available frames have been processed then the function=20 will attempt to refill the ring with new buffers. It's likely that there's a long gap between the time the last receive buffer is consumed and the time the RX ring is=20 refilled and the buffers are posted to the hardware, causing a burst of dropped frames and the com_no_buffers firmware counter to increment. Changing the high level design of bce_rx_intr() and bce_rx_fill_chain() slightly to post a new buffer as each frame is passed to the OS would likely avoid these gaps=20 during bursts of small frames but I'm not sure whether=20 they'll have a negative impact on the more common case of streams of MTU sized frames. I've considered this in the past but never coded the change and tested the resulting performance. Does anyone have some experience with one method over the other? Dave=