From owner-freebsd-current@FreeBSD.ORG Tue Mar 9 21:40:19 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78E7E10656BC; Tue, 9 Mar 2010 21:40:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id C9C9F8FC08; Tue, 9 Mar 2010 21:40:18 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id d23so458404fga.13 for ; Tue, 09 Mar 2010 13:40:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=CLnR4t5yHelh7sLyp37niy9S+u1I81d++Lp+eK/7v4s=; b=XppaxkADOA0u++EQLdTFdw6QekJqoAwpQ6D9J3AaqdOZQGGnHdWzkATeR7Nm5KEDyj T/zEdrtEL5pzukoajvMpldvu29X2l8q+mxx/Q2J8hj00wDtnqOiM2FCe1IwWhWafUgWK HL5768vi30REuMXAF90f4Hmu4RkAUiY9S/VjY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=UQnBfP9CsIEP1m4nXsG7qJso7doXKNpKP+SEz5WIhpAyjtLmMDkyU31J5bdGoin9/E kWuvEsRoATvIjY2I7pPtgn3w8NtKNVVtPfXEjl3iRSfH8jO0IcCwyyB6RYjOhp87cSgK ejK0M7nIHLtuEUdQdjK3TGgi2UncpmWl+fljY= Received: by 10.87.40.12 with SMTP id s12mr1203590fgj.38.1268170817061; Tue, 09 Mar 2010 13:40:17 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id d8sm8792326fga.19.2010.03.09.13.40.14 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 09 Mar 2010 13:40:16 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Tue, 9 Mar 2010 13:40:13 -0800 From: Pyun YongHyeon Date: Tue, 9 Mar 2010 13:40:13 -0800 To: David Christensen Message-ID: <20100309214012.GQ1311@michelle.cdnetworks.com> References: <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> User-Agent: Mutt/1.4.2.3i Cc: Ian FREISLICH , "current@freebsd.org" , David Christensen Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 21:40:19 -0000 On Tue, Mar 09, 2010 at 01:31:55PM -0800, David Christensen wrote: > > > > patch can fix the RX issue you're suffering from. Anyway, > > would you > > > > give it try the patch at the following URL? > > > > http://people.freebsd.org/~yongari/bce/bce.20100305.diff > > > > The patch was generated against CURRENT and you may see a message > > > > like "Disabling COAL_NOW timedout!" during interface up. You can > > > > ignore that message. > > > > > > It's been running for about 1:23 on the patched driver. I'm still > > > seeing the com_no_buffers increase: > > > > > > [firewall2.jnb1] ~ # sysctl dev.bce |grep com_no_buffers > > > dev.bce.0.com_no_buffers: 5642 > > > dev.bce.1.com_no_buffers: 497 > > > dev.bce.2.com_no_buffers: 6260612 > > > dev.bce.3.com_no_buffers: 4871338 > > > > > > > Still have no idea why these counters are increasing here. > > Actually the counter is read from scratch pad of completion > > processor. The datasheet does not even document the counter. > > Maybe david know better what's happening here(CCed). > > > > > Interupt rate is down now, at about 3500 per second per interface. > > > > > > Interestingly setting net.inet.ip.fastforwarding=0 reduces CPU > > > consumption from 25% to 9% and less packet loss. > > The com_no_buffers statistic comes from firmware and indicates how > many times a valid frame was received but could not be placed into > the receive chain because there were no available RX buffers. The > firmware will then drop the frame but that dropped frame won't be > reflected in any of the hardware based statistics. > Yeah, but the question is why bce(4) has no available RX buffers. The system has a lot of available mbufs so I don't see the root cause here. > Dave