From owner-freebsd-net@FreeBSD.ORG Sat Mar 12 20:18:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20DEA106566B for ; Sat, 12 Mar 2011 20:18:09 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id E26608FC14 for ; Sat, 12 Mar 2011 20:18:08 +0000 (UTC) Received: by iyj12 with SMTP id 12so4446545iyj.13 for ; Sat, 12 Mar 2011 12:18:08 -0800 (PST) Received: by 10.42.169.197 with SMTP id c5mr1585669icz.344.1299961088243; Sat, 12 Mar 2011 12:18:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.42.241.3 with HTTP; Sat, 12 Mar 2011 12:17:28 -0800 (PST) In-Reply-To: References: From: Vlad Galu Date: Sat, 12 Mar 2011 21:17:28 +0100 Message-ID: To: Arnaud Lacombe Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: bge(4) on RELENG_8 mbuf cluster starvation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Mar 2011 20:18:09 -0000 On Sat, Mar 12, 2011 at 8:53 PM, Arnaud Lacombe wrote: > Hi, > > On Sat, Mar 12, 2011 at 4:03 AM, Vlad Galu wrote: > > Hi folks, > > > > On a fairly busy recent (r219010) RELENG_8 machine I keep getting > > -- cut here -- > > 1096/1454/2550 mbufs in use (current/cache/total) > > 1035/731/1766/262144 mbuf clusters in use (current/cache/total/max) > > 1035/202 mbuf+clusters out of packet secondary zone in use > (current/cache) > > 0/117/117/12800 4k (page size) jumbo clusters in use > > (current/cache/total/max) > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 2344K/2293K/4637K bytes allocated to network (current/cache/total) > > 0/70128196/37726935 requests for mbufs denied > (mbufs/clusters/mbuf+clusters) > > ^^^^^^^^^^^^^^^^^^^^^ > > -- and here -- > > > > kern.ipc.nmbclusters is set to 131072. Other settings: > no, netstat(8) says 262144. > > Heh, you're right, I forgot I'd doubled it a while ago. Wrote that from the top of my head. > Maybe can you include $(sysctl dev.bge) ? Might be useful. > > - Arnaud > Sure: -- cut here -- dev.bge.0.%desc: Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x004101 dev.bge.0.%driver: bge dev.bge.0.%location: slot=0 function=0 dev.bge.0.%pnpinfo: vendor=0x14e4 device=0x1659 subvendor=0x1014 subdevice=0x02c6 class=0x020000 dev.bge.0.%parent: pci4 dev.bge.0.forced_collapse: 2 dev.bge.0.forced_udpcsum: 0 dev.bge.0.stats.FramesDroppedDueToFilters: 0 dev.bge.0.stats.DmaWriteQueueFull: 0 dev.bge.0.stats.DmaWriteHighPriQueueFull: 0 dev.bge.0.stats.NoMoreRxBDs: 0 dev.bge.0.stats.InputDiscards: 0 dev.bge.0.stats.InputErrors: 0 dev.bge.0.stats.RecvThresholdHit: 0 dev.bge.0.stats.rx.ifHCInOctets: 68037590571 dev.bge.0.stats.rx.Fragments: 0 dev.bge.0.stats.rx.UnicastPkts: 85983136 dev.bge.0.stats.rx.MulticastPkts: 0 dev.bge.0.stats.rx.BroadcastPkts: 1 dev.bge.0.stats.rx.FCSErrors: 0 dev.bge.0.stats.rx.AlignmentErrors: 0 dev.bge.0.stats.rx.xonPauseFramesReceived: 0 dev.bge.0.stats.rx.xoffPauseFramesReceived: 0 dev.bge.0.stats.rx.ControlFramesReceived: 0 dev.bge.0.stats.rx.xoffStateEntered: 0 dev.bge.0.stats.rx.FramesTooLong: 0 dev.bge.0.stats.rx.Jabbers: 0 dev.bge.0.stats.rx.UndersizePkts: 0 dev.bge.0.stats.tx.ifHCOutOctets: 91957314255 dev.bge.0.stats.tx.Collisions: 0 dev.bge.0.stats.tx.XonSent: 0 dev.bge.0.stats.tx.XoffSent: 0 dev.bge.0.stats.tx.InternalMacTransmitErrors: 0 dev.bge.0.stats.tx.SingleCollisionFrames: 0 dev.bge.0.stats.tx.MultipleCollisionFrames: 0 dev.bge.0.stats.tx.DeferredTransmissions: 0 dev.bge.0.stats.tx.ExcessiveCollisions: 0 dev.bge.0.stats.tx.LateCollisions: 0 dev.bge.0.stats.tx.UnicastPkts: 624045221 dev.bge.0.stats.tx.MulticastPkts: 10841 dev.bge.0.stats.tx.BroadcastPkts: 1121 dev.bge.1.%desc: Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x004101 dev.bge.1.%driver: bge dev.bge.1.%location: slot=0 function=0 dev.bge.1.%pnpinfo: vendor=0x14e4 device=0x1659 subvendor=0x1014 subdevice=0x02c6 class=0x020000 dev.bge.1.%parent: pci5 dev.bge.1.forced_collapse: 2 dev.bge.1.forced_udpcsum: 0 dev.bge.1.stats.FramesDroppedDueToFilters: 0 dev.bge.1.stats.DmaWriteQueueFull: 0 dev.bge.1.stats.DmaWriteHighPriQueueFull: 0 dev.bge.1.stats.NoMoreRxBDs: 680050 dev.bge.1.stats.InputDiscards: 228755931 dev.bge.1.stats.InputErrors: 49080818 dev.bge.1.stats.RecvThresholdHit: 0 dev.bge.1.stats.rx.ifHCInOctets: 2095148839247 dev.bge.1.stats.rx.Fragments: 47887706 dev.bge.1.stats.rx.UnicastPkts: 32672557601 dev.bge.1.stats.rx.MulticastPkts: 1218 dev.bge.1.stats.rx.BroadcastPkts: 2 dev.bge.1.stats.rx.FCSErrors: 2822217 dev.bge.1.stats.rx.AlignmentErrors: 0 dev.bge.1.stats.rx.xonPauseFramesReceived: 0 dev.bge.1.stats.rx.xoffPauseFramesReceived: 0 dev.bge.1.stats.rx.ControlFramesReceived: 0 dev.bge.1.stats.rx.xoffStateEntered: 0 dev.bge.1.stats.rx.FramesTooLong: 0 dev.bge.1.stats.rx.Jabbers: 0 dev.bge.1.stats.rx.UndersizePkts: 0 dev.bge.1.stats.tx.ifHCOutOctets: 48751515826 dev.bge.1.stats.tx.Collisions: 0 dev.bge.1.stats.tx.XonSent: 0 dev.bge.1.stats.tx.XoffSent: 0 dev.bge.1.stats.tx.InternalMacTransmitErrors: 0 dev.bge.1.stats.tx.SingleCollisionFrames: 0 dev.bge.1.stats.tx.MultipleCollisionFrames: 0 dev.bge.1.stats.tx.DeferredTransmissions: 0 dev.bge.1.stats.tx.ExcessiveCollisions: 0 dev.bge.1.stats.tx.LateCollisions: 0 dev.bge.1.stats.tx.UnicastPkts: 281039183 dev.bge.1.stats.tx.MulticastPkts: 0 dev.bge.1.stats.tx.BroadcastPkts: 1153 -- and here -- And now, that I remembered about this as well: -- cut here -- Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll bge1 1500 00:11:25:22:0d:ed 32321767025 278517070 37726837 281068216 0 0 -- and here -- The colo provider changed my cable a couple of times so I'd not blame it on that. Unfortunately, I don't have access to the port statistics on the switch. Running netstat with -w1 yields between 0 and 4 errors/second. -- Good, fast & cheap. Pick any two.