From owner-freebsd-current@FreeBSD.ORG Wed Mar 10 19:52:12 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3032D106564A for ; Wed, 10 Mar 2010 19:52:12 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id AC9248FC13 for ; Wed, 10 Mar 2010 19:52:11 +0000 (UTC) Received: by fxm23 with SMTP id 23so7546494fxm.3 for ; Wed, 10 Mar 2010 11:52:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=eefxUWtx02CNhv5UP0NWcOeXTipobIs9tlBZIlQ64zQ=; b=Cmu5QfQVaC7Jt5CuejTH+rH7slYqDgxLg6donltYkMHd1kWlH4jSJsOjtp3ZL5TKoU VJeQsXwJxY+jtMzKc1d1TgO0j8oHixEr4R7KyZGOyb7LQDWIstmkPglTsuPgbKDMIxK7 juwd1sdZZ3eBYK19iL5vPXwf/3dbZMUwSofK0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=lEOPjfanIElkmbzjp5IoXfYUf+DuxfIdUGIEk70KO2fg38CvuO3WkPIDQBkFOFK9bb fZfmZvduRVBK7v4EnaBhYjLmXh+6Kdxo3QVNRbT3aj3b7ZAqnPcxU+R9gRIdN0EcTz9j wpGY5DHHOVqBQQoJ1pUTpe4B/x1DRi/tdYHw4= Received: by 10.87.67.10 with SMTP id u10mr3671895fgk.28.1268250730590; Wed, 10 Mar 2010 11:52:10 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id d8sm10337213fga.4.2010.03.10.11.52.07 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 10 Mar 2010 11:52:09 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 10 Mar 2010 11:52:06 -0800 From: Pyun YongHyeon Date: Wed, 10 Mar 2010 11:52:06 -0800 To: David Christensen Message-ID: <20100310195206.GB10657@michelle.cdnetworks.com> References: <20100305175639.GB14818@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> User-Agent: Mutt/1.4.2.3i Cc: Ian FREISLICH , "current@freebsd.org" Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 19:52:12 -0000 On Wed, Mar 10, 2010 at 11:11:13AM -0800, David Christensen wrote: > > > What's the traffic look like? Jumbo, standard, short > > frames? Any=20 > > > good ideas on profiling the code? I haven't figured out how to use > > > the CPU TSC but there is a free running timer on the device > > that might > > > be usable to calculate where the driver's time is spent. > > > > It looks like the traffic that provoked it was this: > > > > 10:18:42.319370 IP X.4569 > X.4569: UDP, length 12 > > 10:18:42.319402 IP X.4569 > X.4569: UDP, length 12 > > 10:18:42.319438 IP X.4569 > X.4569: UDP, length 12 > > 10:18:42.319484 IP X.4569 > X.4569: UDP, length 12 > > 10:18:42.319517 IP X.4569 > X.4569: UDP, length 12 > > > > A flurry of UDP tinygrams on an IAX2 trunk. The packet rate > > isn't spectacular at about 30kpps which on top of the base > > load of 60kpps still isn't a fantastic packet rate. The > > interesting thing is that while this storm was inprogress, it > > almost entirely excluded other traffic on the network. > > Ok, small packet performance is involved, this narrows down > the range of problems. The current design of bce_rx_intr() > attempts to process all RX frames in the receive ring. After > all available frames have been processed then the function > will attempt to refill the ring with new buffers. It's > likely that there's a long gap between the time the last > receive buffer is consumed and the time the RX ring is > refilled and the buffers are posted to the hardware, causing > a burst of dropped frames and the com_no_buffers firmware > counter to increment. > I successfully reproduced the issue with netperf on BCM5709. You can use UDP frame size 1 to trigger the issue. > Changing the high level design of bce_rx_intr() and > bce_rx_fill_chain() slightly to post a new buffer as each > frame is passed to the OS would likely avoid these gaps > during bursts of small frames but I'm not sure whether > they'll have a negative impact on the more common case of > streams of MTU sized frames. I've considered this in the > past but never coded the change and tested the resulting > performance. > I guess this may slightly increase performance with additional bus_dma(9) overheads but I think one of reason of dropping frames under heavy UDP frames may come from lack of free RX descriptors. Because bce(4) just uses a single RX ring so the number of available RX buffers would be 512. However it seems it's not possible to increase the number of RX buffers per RX ring so the next possible approach would be switching to use multiple RX rings with RSS. Even though FreeBSD does not dynamically adjust loads among CPUs I guess using RSS would be the way to go. > Does anyone have some experience with one method over > the other? > > Dave