Date: Thu, 09 Sep 2010 15:58:30 -0500 From: Tom Judge <tom@tomjudge.com> To: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@FreeBSD.org Subject: bce(4) - com_no_buffers (Again) Message-ID: <4C894A76.5040200@tomjudge.com>
next in thread | raw e-mail | index | archive | help
Hi, I am just following up on the thread from March (I think) about this issue. We are seeing this issue on a number of systems running 7.1. The systems in question are all Dell: * R710 R610 R410 * PE2950 The latter do not show the issue as much as the R series systems. The cards in one of the R610's that I am testing with are: bce0@pci0:1:0:0: class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet They are connected to Dell PowerConnect 5424 switches. uname -a: FreeBSD bandor.chi-dc.mintel.ad 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #3: Wed Sep 8 08:19:03 UTC 2010 tj@dev-tj-7-1-amd64.chicago.mintel.ad:/usr/obj/usr/src/sys/MINTELv10 amd64 We are also using 8192 byte jumbo frames, if_lagg and if_vlan in the configuration (the nics are in promisc as we are currently capturing netflow data on another vlan for diagnostic purposes. ): tj@bandor '20:51:17' '~' > $ ifconfig bce0 bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 8192 options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> ether 00:21:9b:95:7a:b8 media: Ethernet autoselect (1000baseTX <full-duplex>) status: active lagg: laggdev lagg0 tj@bandor '20:51:22' '~' > $ ifconfig bce1 bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 8192 options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> ether 00:21:9b:95:7a:b8 media: Ethernet autoselect (1000baseTX <full-duplex>) status: active lagg: laggdev lagg0 tj@bandor '20:51:35' '~' > $ ifconfig lagg0 lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 8192 options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> ether 00:21:9b:95:7a:b8 media: Ethernet autoselect status: active laggproto failover laggport: bce1 flags=0<> laggport: bce0 flags=5<MASTER,ACTIVE> tj@bandor '20:51:40' '~' > $ ifconfig vlan2 vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 8192 options=3<RXCSUM,TXCSUM> ether 00:21:9b:95:7a:b8 inet 172.30.XX.XX netmask 0xfffffe00 broadcast 172.30.XX.XX media: Ethernet autoselect status: active vlan: 2 parent interface: lagg0 I have updated the bce driver and the Broadcomm MII driver to the version from stable/7 and am still seeing the issue. This morning I did a test with increasing the RX_PAGES to 8 but the system just hung starting the network. The route command got stuck in a zone state (Sorry can't remember exactly which). The real question is, how do we go about increasing the number of RX BDs? I guess we have to bump more that just RX_PAGES... The cause for us, from what we can see, is the openldap server sending large group search results back to nss_ldap or pam_ldap. When it does this it seems to send each of the 600 results in its own TCP segment creating a small packet storm (600*~100byte PDU's) at the destination host. The kernel then retransmits 2 blocks of 100 results each after SACK kicks in for the data that was dropped by the NIC. Thanks in advance Tom tj@bandor '20:57:33' '~' > $ sysctl -a dev.bce.0 dev.bce.0.%desc: Broadcom NetXtreme II BCM5709 1000Base-T (C0) dev.bce.0.%driver: bce dev.bce.0.%location: slot=0 function=0 dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028 subdevice=0x0236 class=0x020000 dev.bce.0.%parent: pci1 dev.bce.0.l2fhdr_error_count: 0 dev.bce.0.mbuf_alloc_failed_count: 0 dev.bce.0.mbuf_frag_count: 0 dev.bce.0.dma_map_addr_rx_failed_count: 0 dev.bce.0.dma_map_addr_tx_failed_count: 0 dev.bce.0.unexpected_attention_count: 0 dev.bce.0.stat_IfHcInOctets: 439779802 dev.bce.0.stat_IfHCInBadOctets: 0 dev.bce.0.stat_IfHCOutOctets: 108341440 dev.bce.0.stat_IfHCOutBadOctets: 0 dev.bce.0.stat_IfHCInUcastPkts: 2341369 dev.bce.0.stat_IfHCInMulticastPkts: 26065 dev.bce.0.stat_IfHCInBroadcastPkts: 9191 dev.bce.0.stat_IfHCOutUcastPkts: 1230052 dev.bce.0.stat_IfHCOutMulticastPkts: 2870 dev.bce.0.stat_IfHCOutBroadcastPkts: 45 dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0 dev.bce.0.stat_Dot3StatsFCSErrors: 0 dev.bce.0.stat_Dot3StatsAlignmentErrors: 0 dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0 dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0 dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0 dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0 dev.bce.0.stat_Dot3StatsLateCollisions: 0 dev.bce.0.stat_EtherStatsCollisions: 0 dev.bce.0.stat_EtherStatsFragments: 0 dev.bce.0.stat_EtherStatsJabbers: 0 dev.bce.0.stat_EtherStatsUndersizePkts: 0 dev.bce.0.stat_EtherStatsOversizePkts: 0 dev.bce.0.stat_EtherStatsPktsRx64Octets: 3381 dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 98883 dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 2255959 dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 12508 dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 4247 dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 522 dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 1125 dev.bce.0.stat_EtherStatsPktsTx64Octets: 496 dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 1176041 dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 29079 dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 2933 dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 23898 dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 234 dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 286 dev.bce.0.stat_XonPauseFramesReceived: 0 dev.bce.0.stat_XoffPauseFramesReceived: 0 dev.bce.0.stat_OutXonSent: 0 dev.bce.0.stat_OutXoffSent: 0 dev.bce.0.stat_FlowControlDone: 0 dev.bce.0.stat_MacControlFramesReceived: 0 dev.bce.0.stat_XoffStateEntered: 0 dev.bce.0.stat_IfInFramesL2FilterDiscards: 0 dev.bce.0.stat_IfInRuleCheckerDiscards: 0 dev.bce.0.stat_IfInFTQDiscards: 0 dev.bce.0.stat_IfInMBUFDiscards: 0 dev.bce.0.stat_IfInRuleCheckerP4Hit: 35256 dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0 dev.bce.0.stat_CatchupInFTQDiscards: 0 dev.bce.0.stat_CatchupInMBUFDiscards: 0 dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0 dev.bce.0.com_no_buffers: 13021 -- TJU13-ARIN
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C894A76.5040200>