From owner-freebsd-net@FreeBSD.ORG Wed Jul 6 20:16:43 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC8061065670; Wed, 6 Jul 2011 20:16:43 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 88AFD8FC1B; Wed, 6 Jul 2011 20:16:43 +0000 (UTC) Received: by iyb11 with SMTP id 11so349972iyb.13 for ; Wed, 06 Jul 2011 13:16:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Dzp54JO664nLD0sSLDS5RGMQgjXC4lYgQcZCLYIkDnE=; b=W5X4wzcpDaHLCCB7vpQlt8Z/gQfwlnL5jSJ17ejhLgr9vsYSJEw+9AqQ9X+YFUAANX fm9VwkAJ0VDQiwyt2qO11xogKFnTBLv3cAHnNGTS4iNKdKt0KvYdD8O9jVfdDUC6H/fb QVLWH4MaMx1vu5mbD2kKJ+WzSbESEpmsfTJAA= Received: by 10.42.178.137 with SMTP id bm9mr8467197icb.313.1309983402759; Wed, 06 Jul 2011 13:16:42 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id s2sm9087876icw.17.2011.07.06.13.16.40 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 06 Jul 2011 13:16:41 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 06 Jul 2011 13:15:09 -0700 From: YongHyeon PYUN Date: Wed, 6 Jul 2011 13:15:09 -0700 To: Charles Sprickman Message-ID: <20110706201509.GA5559@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, David Christensen Subject: Re: bce packet loss X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 20:16:44 -0000 On Mon, Jul 04, 2011 at 09:32:11PM -0400, Charles Sprickman wrote: > Hello, > > We're running a few 8.1-R servers with Broadcom bce interfaces (Dell R510) > and I'm seeing occasional packet loss on them (enough that it trips nagios > now and then). Cabling seems fine as neither the switch nor the sysctl > info for the device show any errors/collisions/etc, however there is one > odd one, which is "dev.bce.1.stat_IfHCInBadOctets: 539369". See [1] below > for full sysctl output. The switch shows no errors but for "Dropped > packets 683868". > > pciconf output is also below. [2] > > By default, the switch had flow control set to "on". I also let it run > with "auto". In both cases, the drops continued to increment. I'm now > running with flow control off to see if that changes anything. > > I do see some correlation between cpu usage and drops - I have cpu usage > graphed in nagios and cacti is graphing the drops on the dell switch. > There's no signs of running out of mbufs or similar. > > So given that limited info, is there anything I can look at to track this > down? Anything stand out in the stats sysctl exposes? Two things are > standing out for me - the number of changes in bce regarding flow control > that are not in 8.1, and the correlation between cpu load and the drops. > > What other information can I provide? > You had 282 RX buffer shortages and these frames were dropped. This may explain why you see occasional packet loss. 'netstat -m' will show which size of cluster allocation were failed. However it seems you have 0 com_no_buffers which indicates controller was able to receive all packets destined for this host. You may host lost some packets(i.e. non-zero mbuf_alloc_failed_count) but your controller and system was still responsive to the network traffic. Data sheet says IfHCInBadOctets indicates number of octets received on the interface, including framing characters for packets that were dropped in the MAC for any reason. I'm not sure this counter includes packets IfInFramesL2FilterDiscards which indicates number of good frames that have been dropped due to the L2 perfect match, broadcast, multicast or MAC control frame filters. If your switch runs STP it would periodically sends BPDU packets to destination address of STP multicast address 01:80:C2:00:00:00. Not sure this is the reason though. Probably David can explain more details on IfHCInBadOctets counter(CCed).