From owner-freebsd-current@FreeBSD.ORG Wed Mar 10 22:47:07 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C81FB106566B for ; Wed, 10 Mar 2010 22:47:07 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from mms2.broadcom.com (mms2.broadcom.com [216.31.210.18]) by mx1.freebsd.org (Postfix) with ESMTP id 9CDC08FC24 for ; Wed, 10 Mar 2010 22:47:07 +0000 (UTC) Received: from [10.9.200.133] by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Wed, 10 Mar 2010 14:45:48 -0800 X-Server-Uuid: D3C04415-6FA8-4F2C-93C1-920E106A2031 Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.30]) by IRVEXCHHUB02.corp.ad.broadcom.com ([10.9.200.133]) with mapi; Wed, 10 Mar 2010 14:47:12 -0800 From: "David Christensen" To: "pyunyh@gmail.com" Date: Wed, 10 Mar 2010 14:45:47 -0800 Thread-Topic: dev.bce.X.com_no_buffers increasing and packet loss Thread-Index: AcrAizRkYmKmu4SaS3KppsIoJoOY8AAFrpRQ Message-ID: <5D267A3F22FD854F8F48B3D2B52381933AF90EF25A@IRVEXCHCCR01.corp.ad.broadcom.com> References: <20100305175639.GB14818@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> <20100310195206.GB10657@michelle.cdnetworks.com> In-Reply-To: <20100310195206.GB10657@michelle.cdnetworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-WSS-ID: 6786FE9638O137986979-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Cc: Ian FREISLICH , "current@freebsd.org" Subject: RE: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 22:47:07 -0000 > I successfully reproduced the issue with netperf on BCM5709.=20 > You can use UDP frame size 1 to trigger the issue. >=20 > > Changing the high level design of bce_rx_intr() and > > bce_rx_fill_chain() slightly to post a new buffer as each frame is=20 > > passed to the OS would likely avoid these gaps during=20 > bursts of small=20 > > frames but I'm not sure whether they'll have a negative=20 > impact on the=20 > > more common case of streams of MTU sized frames. I've=20 > considered this=20 > > in the past but never coded the change and tested the resulting=20 > > performance. > >=20 >=20 > I guess this may slightly increase performance with additional > bus_dma(9) overheads but I think one of reason of dropping=20 > frames under heavy UDP frames may come from lack of free RX=20 > descriptors. > Because bce(4) just uses a single RX ring so the number of=20 > available RX buffers would be 512. However it seems it's not=20 > possible to increase the number of RX buffers per RX ring so=20 > the next possible approach would be switching to use multiple=20 > RX rings with RSS. Even though FreeBSD does not dynamically=20 > adjust loads among CPUs I guess using RSS would be the way to go. The bce(4) hardware supports a linked list of pages for RX=20 buffer descriptors. The stock build supports 2 pages (RX_PAGES) with a total of 511 BD's per page. The hardware can support a maximum of 64K BD's but that would be an unnecessarily large amount of mbufs for an infrequent problem. The middle road would probably involve changing RX_PAGES from a #define to a sysctl variable to allow tuning for specific environments along with a change in bce_rx_intr() to fill the ring after all frames have been processed or when more than 256 BDs have been consumed, whichever comes first. RSS would be great as well though it wouldn't make a dent in this case since RSS is only supported for TCP, not UDP. Dave=