From owner-freebsd-net@FreeBSD.ORG Tue Jul 5 01:32:13 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B47B0106564A for ; Tue, 5 Jul 2011 01:32:13 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 668C68FC0C for ; Tue, 5 Jul 2011 01:32:13 +0000 (UTC) Received: (qmail 81944 invoked by uid 0); 5 Jul 2011 01:32:12 -0000 Received: from smtp.bway.net (216.220.96.25) by xena.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 5 Jul 2011 01:32:12 -0000 Received: (qmail 81940 invoked by uid 90); 5 Jul 2011 01:32:12 -0000 Received: from unknown (HELO ?10.3.2.40?) (spork@bway.net@96.57.144.66) by smtp.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 5 Jul 2011 01:32:12 -0000 Date: Mon, 4 Jul 2011 21:32:11 -0400 (EDT) From: Charles Sprickman X-X-Sender: spork@freemac To: freebsd-net@freebsd.org Message-ID: User-Agent: Alpine 2.00 (OSX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: bce packet loss X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 01:32:13 -0000 Hello, We're running a few 8.1-R servers with Broadcom bce interfaces (Dell R510) and I'm seeing occasional packet loss on them (enough that it trips nagios now and then). Cabling seems fine as neither the switch nor the sysctl info for the device show any errors/collisions/etc, however there is one odd one, which is "dev.bce.1.stat_IfHCInBadOctets: 539369". See [1] below for full sysctl output. The switch shows no errors but for "Dropped packets 683868". pciconf output is also below. [2] By default, the switch had flow control set to "on". I also let it run with "auto". In both cases, the drops continued to increment. I'm now running with flow control off to see if that changes anything. I do see some correlation between cpu usage and drops - I have cpu usage graphed in nagios and cacti is graphing the drops on the dell switch. There's no signs of running out of mbufs or similar. So given that limited info, is there anything I can look at to track this down? Anything stand out in the stats sysctl exposes? Two things are standing out for me - the number of changes in bce regarding flow control that are not in 8.1, and the correlation between cpu load and the drops. What other information can I provide? Thanks, Charles [1] [root@h23 /home/spork]# sysctl -a |grep bce.1 dev.bce.1.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0) dev.bce.1.%driver: bce dev.bce.1.%location: slot=0 function=1 dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x163b subvendor=0x1028 subdevice=0x02f1 class=0x020000 dev.bce.1.%parent: pci1 dev.bce.1.l2fhdr_error_count: 0 dev.bce.1.mbuf_alloc_failed_count: 282 dev.bce.1.fragmented_mbuf_count: 2748 dev.bce.1.dma_map_addr_rx_failed_count: 0 dev.bce.1.dma_map_addr_tx_failed_count: 5 dev.bce.1.unexpected_attention_count: 0 dev.bce.1.stat_IfHcInOctets: 62708651108 dev.bce.1.stat_IfHCInBadOctets: 539369 dev.bce.1.stat_IfHCOutOctets: 434264587173 dev.bce.1.stat_IfHCOutBadOctets: 0 dev.bce.1.stat_IfHCInUcastPkts: 533441918 dev.bce.1.stat_IfHCInMulticastPkts: 3108746 dev.bce.1.stat_IfHCInBroadcastPkts: 1314905 dev.bce.1.stat_IfHCOutUcastPkts: 640961970 dev.bce.1.stat_IfHCOutMulticastPkts: 26 dev.bce.1.stat_IfHCOutBroadcastPkts: 8909 dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0 dev.bce.1.stat_Dot3StatsFCSErrors: 0 dev.bce.1.stat_Dot3StatsAlignmentErrors: 0 dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0 dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0 dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0 dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0 dev.bce.1.stat_Dot3StatsLateCollisions: 0 dev.bce.1.stat_EtherStatsCollisions: 0 dev.bce.1.stat_EtherStatsFragments: 0 dev.bce.1.stat_EtherStatsJabbers: 0 dev.bce.1.stat_EtherStatsUndersizePkts: 0 dev.bce.1.stat_EtherStatsOversizePkts: 0 dev.bce.1.stat_EtherStatsPktsRx64Octets: 34048797 dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 431844366 dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 25946173 dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 39936369 dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 2296565 dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 3931392 dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0 dev.bce.1.stat_EtherStatsPktsTx64Octets: 60122571 dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 221041349 dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 40177071 dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 24099944 dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 44493532 dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 251036438 dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0 dev.bce.1.stat_XonPauseFramesReceived: 61778 dev.bce.1.stat_XoffPauseFramesReceived: 76315 dev.bce.1.stat_OutXonSent: 0 dev.bce.1.stat_OutXoffSent: 0 dev.bce.1.stat_FlowControlDone: 0 dev.bce.1.stat_MacControlFramesReceived: 0 dev.bce.1.stat_XoffStateEntered: 0 dev.bce.1.stat_IfInFramesL2FilterDiscards: 145832 dev.bce.1.stat_IfInRuleCheckerDiscards: 0 dev.bce.1.stat_IfInFTQDiscards: 0 dev.bce.1.stat_IfInMBUFDiscards: 0 dev.bce.1.stat_IfInRuleCheckerP4Hit: 4448215 dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0 dev.bce.1.stat_CatchupInFTQDiscards: 0 dev.bce.1.stat_CatchupInMBUFDiscards: 0 dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0 dev.bce.1.com_no_buffers: 0 [2] pciconf -lvb bce1@pci0:1:0:1: class=0x020000 card=0x02f11028 chip=0x163b14e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' class = network subclass = ethernet bar [10] = type Memory, range 64, base 0xdc000000, size 33554432, enabled