From owner-freebsd-current@FreeBSD.ORG Tue Mar 9 22:12:45 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD0E91065670 for ; Tue, 9 Mar 2010 22:12:45 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.154]) by mx1.freebsd.org (Postfix) with ESMTP id 454C48FC1D for ; Tue, 9 Mar 2010 22:12:44 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 22so2414583fge.13 for ; Tue, 09 Mar 2010 14:12:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=qRInxI3uBxuYToIsvhp5NsFaabm64s4LODWNIYm81W8=; b=MaqHlKH0oakWoYLL2oX6Z9mt/8sZ8jSAgbKJt5fqohiia73hHdQxM9kaqpMV9elPrW 0h9aObEv0ejrHMknDNUAGMNSsI/ye1L+hnxBX7sdNbtq3X4SrMIJD9RP5nDoQ176DGJn Yl39j/MLn+3uosAW78A6f8FAYX4N1nV+Lp33A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=LuLcUq9O4fBmrhpm2/kZFJglogiI2ppT40kKX46g72HzYgCNlwmcDW2KuQlODkXQKa xu5Y9MgKB4ywBXC6aatHUTCZIKD86w4VOD1KaNvhCJf2GIhhZxioU+E4kx03Xf5iJCmq 7GRC/Ru8RcEg5/ZuAvAK9CVhGWDPoTpfPrKDw= Received: by 10.87.40.24 with SMTP id s24mr1242290fgj.48.1268172764110; Tue, 09 Mar 2010 14:12:44 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id e20sm8882829fga.20.2010.03.09.14.12.41 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 09 Mar 2010 14:12:42 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Tue, 9 Mar 2010 14:12:40 -0800 From: Pyun YongHyeon Date: Tue, 9 Mar 2010 14:12:40 -0800 To: Ian FREISLICH Message-ID: <20100309221240.GR1311@michelle.cdnetworks.com> References: <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 22:12:45 -0000 On Tue, Mar 09, 2010 at 11:55:30PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Tue, Mar 09, 2010 at 03:31:55PM +0200, Ian FREISLICH wrote: > > > Can you explain the tunables please - I'm guessing it's these: > > I think I asked the wrong question. What is a "Quick BD Chain"? I don't know why Broadocom uses 'Quick'. It's a just buffer descriptor chain for TX/RX. > What relation should this number have to traffic rate. Is there a > maximum and what are reasonable numbers for setting this to? > The maximum BD chain would be number of configured TX/RX descriptors. And 1 would be minimum value as you have to want to get status block for each TX/RX. Finding best value may depend on specific load. > I set the RX as high as 512 in 64 quanta but it made little difference > to the interrupt rate. At times where we experience the packet > loss and com_no_buffers increases, the interrupt rate on between 1 > and 3 of the 4 bce interfaces fell from about 3200/s to 130/s. > BD chain is just one of parameters. bce(4) controllers also provide more advanced features that fine control interrupt moderation(TX/RX ticks). It's hard to explain all the details so you may want to read public data sheet of bce(4). http://www.broadcom.com/collateral/pg/NetXtremeII-PG203-R.pdf See host coalescing registers(page 484). > We're wondering if the switches we're using could be causing this > problem - they're Dell PowerConnect 5448. I've seen complaints of > random packet loss caused by these switches on the Internet. We > have some new H3C 5100 series switches which we're planning on > swapping for the Dells tomorrow to see if it makes a difference. > Not sure but it does not explain increasing dev.bce.X.com_no_buffers counter. > Ian > > -- > Ian Freislich