From owner-freebsd-net@FreeBSD.ORG Thu Sep 9 20:58:41 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A818410656E7 for ; Thu, 9 Sep 2010 20:58:41 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from eu1sys200aog101.obsmtp.com (eu1sys200aog101.obsmtp.com [207.126.144.111]) by mx1.freebsd.org (Postfix) with SMTP id 2B7298FC23 for ; Thu, 9 Sep 2010 20:58:39 +0000 (UTC) Received: from source ([63.174.175.251]) by eu1sys200aob101.postini.com ([207.126.147.11]) with SMTP ID DSNKTIlKfVEStNdnNCC9vyVjrS5lAiE1adxg@postini.com; Thu, 09 Sep 2010 20:58:40 UTC Received: from [172.17.10.53] (unknown [172.17.10.53]) by bbbx3.usdmm.com (Postfix) with ESMTP id EE458FD01A; Thu, 9 Sep 2010 20:58:36 +0000 (UTC) Message-ID: <4C894A76.5040200@tomjudge.com> Date: Thu, 09 Sep 2010 15:58:30 -0500 From: Tom Judge User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.12) Gecko/20100826 Lightning/1.0b1 Thunderbird/3.0.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@FreeBSD.org X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 20:58:41 -0000 Hi, I am just following up on the thread from March (I think) about this issue. We are seeing this issue on a number of systems running 7.1. The systems in question are all Dell: * R710 R610 R410 * PE2950 The latter do not show the issue as much as the R series systems. The cards in one of the R610's that I am testing with are: bce0@pci0:1:0:0: class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet They are connected to Dell PowerConnect 5424 switches. uname -a: FreeBSD bandor.chi-dc.mintel.ad 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #3: Wed Sep 8 08:19:03 UTC 2010 tj@dev-tj-7-1-amd64.chicago.mintel.ad:/usr/obj/usr/src/sys/MINTELv10 amd64 We are also using 8192 byte jumbo frames, if_lagg and if_vlan in the configuration (the nics are in promisc as we are currently capturing netflow data on another vlan for diagnostic purposes. ): tj@bandor '20:51:17' '~' > $ ifconfig bce0 bce0: flags=8943 metric 0 mtu 8192 options=400bb ether 00:21:9b:95:7a:b8 media: Ethernet autoselect (1000baseTX ) status: active lagg: laggdev lagg0 tj@bandor '20:51:22' '~' > $ ifconfig bce1 bce1: flags=8943 metric 0 mtu 8192 options=400bb ether 00:21:9b:95:7a:b8 media: Ethernet autoselect (1000baseTX ) status: active lagg: laggdev lagg0 tj@bandor '20:51:35' '~' > $ ifconfig lagg0 lagg0: flags=8943 metric 0 mtu 8192 options=400bb ether 00:21:9b:95:7a:b8 media: Ethernet autoselect status: active laggproto failover laggport: bce1 flags=0<> laggport: bce0 flags=5 tj@bandor '20:51:40' '~' > $ ifconfig vlan2 vlan2: flags=8943 metric 0 mtu 8192 options=3 ether 00:21:9b:95:7a:b8 inet 172.30.XX.XX netmask 0xfffffe00 broadcast 172.30.XX.XX media: Ethernet autoselect status: active vlan: 2 parent interface: lagg0 I have updated the bce driver and the Broadcomm MII driver to the version from stable/7 and am still seeing the issue. This morning I did a test with increasing the RX_PAGES to 8 but the system just hung starting the network. The route command got stuck in a zone state (Sorry can't remember exactly which). The real question is, how do we go about increasing the number of RX BDs? I guess we have to bump more that just RX_PAGES... The cause for us, from what we can see, is the openldap server sending large group search results back to nss_ldap or pam_ldap. When it does this it seems to send each of the 600 results in its own TCP segment creating a small packet storm (600*~100byte PDU's) at the destination host. The kernel then retransmits 2 blocks of 100 results each after SACK kicks in for the data that was dropped by the NIC. Thanks in advance Tom tj@bandor '20:57:33' '~' > $ sysctl -a dev.bce.0 dev.bce.0.%desc: Broadcom NetXtreme II BCM5709 1000Base-T (C0) dev.bce.0.%driver: bce dev.bce.0.%location: slot=0 function=0 dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028 subdevice=0x0236 class=0x020000 dev.bce.0.%parent: pci1 dev.bce.0.l2fhdr_error_count: 0 dev.bce.0.mbuf_alloc_failed_count: 0 dev.bce.0.mbuf_frag_count: 0 dev.bce.0.dma_map_addr_rx_failed_count: 0 dev.bce.0.dma_map_addr_tx_failed_count: 0 dev.bce.0.unexpected_attention_count: 0 dev.bce.0.stat_IfHcInOctets: 439779802 dev.bce.0.stat_IfHCInBadOctets: 0 dev.bce.0.stat_IfHCOutOctets: 108341440 dev.bce.0.stat_IfHCOutBadOctets: 0 dev.bce.0.stat_IfHCInUcastPkts: 2341369 dev.bce.0.stat_IfHCInMulticastPkts: 26065 dev.bce.0.stat_IfHCInBroadcastPkts: 9191 dev.bce.0.stat_IfHCOutUcastPkts: 1230052 dev.bce.0.stat_IfHCOutMulticastPkts: 2870 dev.bce.0.stat_IfHCOutBroadcastPkts: 45 dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0 dev.bce.0.stat_Dot3StatsFCSErrors: 0 dev.bce.0.stat_Dot3StatsAlignmentErrors: 0 dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0 dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0 dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0 dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0 dev.bce.0.stat_Dot3StatsLateCollisions: 0 dev.bce.0.stat_EtherStatsCollisions: 0 dev.bce.0.stat_EtherStatsFragments: 0 dev.bce.0.stat_EtherStatsJabbers: 0 dev.bce.0.stat_EtherStatsUndersizePkts: 0 dev.bce.0.stat_EtherStatsOversizePkts: 0 dev.bce.0.stat_EtherStatsPktsRx64Octets: 3381 dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 98883 dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 2255959 dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 12508 dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 4247 dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 522 dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 1125 dev.bce.0.stat_EtherStatsPktsTx64Octets: 496 dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 1176041 dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 29079 dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 2933 dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 23898 dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 234 dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 286 dev.bce.0.stat_XonPauseFramesReceived: 0 dev.bce.0.stat_XoffPauseFramesReceived: 0 dev.bce.0.stat_OutXonSent: 0 dev.bce.0.stat_OutXoffSent: 0 dev.bce.0.stat_FlowControlDone: 0 dev.bce.0.stat_MacControlFramesReceived: 0 dev.bce.0.stat_XoffStateEntered: 0 dev.bce.0.stat_IfInFramesL2FilterDiscards: 0 dev.bce.0.stat_IfInRuleCheckerDiscards: 0 dev.bce.0.stat_IfInFTQDiscards: 0 dev.bce.0.stat_IfInMBUFDiscards: 0 dev.bce.0.stat_IfInRuleCheckerP4Hit: 35256 dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0 dev.bce.0.stat_CatchupInFTQDiscards: 0 dev.bce.0.stat_CatchupInMBUFDiscards: 0 dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0 dev.bce.0.com_no_buffers: 13021 -- TJU13-ARIN