From owner-freebsd-current@FreeBSD.ORG Fri Mar 5 17:57:15 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58269106564A for ; Fri, 5 Mar 2010 17:57:15 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-bw0-f224.google.com (mail-bw0-f224.google.com [209.85.218.224]) by mx1.freebsd.org (Postfix) with ESMTP id D85798FC1B for ; Fri, 5 Mar 2010 17:57:14 +0000 (UTC) Received: by bwz24 with SMTP id 24so2621067bwz.13 for ; Fri, 05 Mar 2010 09:57:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=GAm+hc+a6/PqPwWYhwFCYw05z5iY110s634wOw1mRk8=; b=JI7uDOtnUCBlVloVJ3LV94NtH48z9BJX4uIyA0BFOuCnO1XZzxeFoYw7+XHKB/y1OC P2pfid3Jiysp5oiXLcm6Z3TrVA1vL6aL7Na6hnRtTvm16/y2TWa83g8vmLUQ1k/p1pI3 u4/Dbx1fN6+1BKzC8taxjdJwBe/OiNgw0PcOM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=jdBwHU5g9FFscyJ2yuh4qTMa/a9HkDfSZmhu9Egivh1fnqAV+vZmZ9bnS99knOTwDx wf8tZlPXTotVTumGcS7wL8dQuqQgphhDLFsG3GVfQEQ5sThN03s+peb+RTUZX1bUUfg+ qSN9hodK99fOGfA+R+A/YaqyRhes0mZga5mvo= Received: by 10.102.127.20 with SMTP id z20mr507019muc.74.1267811820740; Fri, 05 Mar 2010 09:57:00 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id s10sm9060731mue.27.2010.03.05.09.56.57 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 05 Mar 2010 09:56:59 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 5 Mar 2010 09:56:39 -0800 From: Pyun YongHyeon Date: Fri, 5 Mar 2010 09:56:39 -0800 To: Ian FREISLICH Message-ID: <20100305175639.GB14818@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 17:57:15 -0000 On Fri, Mar 05, 2010 at 01:20:57PM +0200, Ian FREISLICH wrote: > Hi > > I have a system that is experiencing mild to severe packet loss. > The interfaces are configured as follows: > > lagg0: bce0, bce1, bce2, bce3 lagproto lacp > > lagg0 then is used as the hwdev for the vlan interfaces. > > I have pf with a few queues for bandwidth management. > > There isn't that much traffic on it (200-500Mbit/s). > > I see only the following suspect for packet loss: > > dev.bce.0.com_no_buffers: 140151466 > dev.bce.1.com_no_buffers: 514723247 > dev.bce.2.com_no_buffers: 10454050 > dev.bce.3.com_no_buffers: 369371 > > Most of the time, these numbers are static, but every once in a > while they increase massively by several thousand, but only on 2 > interfaces. The 1 minute average rate on those interfaces is 266/s > and 123/s. > > Does anyone think this is related to the packet loss or are these > counters just a red herring? Is there anything that can be done > to reduce this count? > I think this sysctl node indicates number of dropped frames in completion processor of NetXtreme II. The counter is incremented when the processor received a frame successfully but it couldn't pass the frame to system as there are no available RX buffers so completion processor dopped the received frame. If you see mbuf shortage from netstat that would be normal. But if system has a lot of free mbuf resources it may indicate other issue. bce(4) may not be able to replenish controller with RX buffer if system is suffering from high load.