From owner-freebsd-stable Mon Jun 17 17:26:50 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mail.westbend.net (ns1.westbend.net [216.47.253.3]) by hub.freebsd.org (Postfix) with ESMTP id E176137B40A for ; Mon, 17 Jun 2002 17:26:32 -0700 (PDT) Received: from ADMIN00 (bnet.westbend.net [216.47.253.17]) by mail.westbend.net (8.12.3/8.12.3) with SMTP id g5I0QUqD074388 for ; Mon, 17 Jun 2002 19:26:30 -0500 (CDT) (envelope-from hetzels@westbend.net) Message-ID: <011101c2165e$cbc5f400$11fd2fd8@ADMIN00> From: "Scot W. Hetzel" To: "FreeBSD-Stable" Subject: System running out of mbufs Date: Mon, 17 Jun 2002 19:26:32 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 X-Virus-Scanned: by amavisd-milter (http://amavis.org/) Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG We are currently running: FreeBSD ns0.westbend.net 4.6-RC FreeBSD 4.6-RC #1: Wed Jun 12 11:39:27 CDT 2002 root@mail.westbend.net:/usr/obj/usr/src/src4/sys/GENERIC-ET i386 The kernel configuration is: # cvs diff -u GENERIC Index: GENERIC =================================================================== RCS file: /home/ncvs/src/sys/i386/conf/GENERIC,v retrieving revision 1.246.2.43 diff -u -r1.246.2.43 GENERIC --- GENERIC 23 May 2002 17:04:01 -0000 1.246.2.43 +++ GENERIC 17 Jun 2002 23:56:02 -0000 @@ -56,6 +56,23 @@ options ICMP_BANDLIM #Rate limit bad replies options KBD_INSTALL_CDEV # install a CDEV entry in /dev +options IPFIREWALL #firewall +options IPFIREWALL_VERBOSE #enable logging to syslogd(8) +options IPFIREWALL_FORWARD #enable transparent proxy support +options IPFIREWALL_VERBOSE_LIMIT=100 #limit verbosity +#options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default +options IPV6FIREWALL #firewall for IPv6 +options IPV6FIREWALL_VERBOSE +options IPV6FIREWALL_VERBOSE_LIMIT=100 +#options IPV6FIREWALL_DEFAULT_TO_ACCEPT + +# RANDOM_IP_ID causes the ID field in IP packets to be randomized +# instead of incremented by 1 with each packet generated. This +# option closes a minor information leak which allows remote +# observers to determine the rate of packet generation on the +# machine by watching the counter. +options RANDOM_IP_ID + # To make an SMP kernel, the next two are needed #options SMP # Symmetric MultiProcessor Kernel #options APIC_IO # Symmetric (APIC) I/O @@ -150,7 +167,7 @@ device npx0 at nexus? port IO_NPX irq 13 # Power management support (see LINT for more options) -device apm0 at nexus? disable flags 0x20 # Advanced Power Management +#device apm0 at nexus? disable flags 0x20 # Advanced Power Management # PCCARD (PCMCIA) support device card @@ -163,6 +180,11 @@ device sio2 at isa? disable port IO_COM3 irq 5 device sio3 at isa? disable port IO_COM4 irq 9 +options BREAK_TO_DEBUGGER # a BREAK on a comconsole goes to + # DDB, if available. +options CONSPEED=115200 # speed for serial console + # (default 9600) + # Parallel port device ppc0 at isa? irq 7 device ppbus # Parallel port bus (required) @@ -177,6 +199,10 @@ device em # Intel PRO/1000 adapter Gigabit Ethernet Card (``Wiseman'') device txp # 3Com 3cR990 (``Typhoon'') device vx # 3Com 3c590, 3c595 (``Vortex'') + +# ETinc +device eth0 +#device bw0 at isa ? # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! The problem we are experiencling is that an unusual high number of mbufs are being used by the system, this problem started about 2 weeks ago with an April 25th 4.5-STABLE kernel, we have tried, upgrading to 4.6-RC, and downgrading to 4.4-RELEASE, without any success in solving this problem. I had posted about this problem before (see "Run away mbufs"), which I received only one response too. That response had pointed me to a utility called minfo (by Ian Dowse). The minfo utility had shown quite a few "Free mbuf not on free list" and "refs missing" in it's output. We have been using a crontab script to reboot the server when ever the peak mbufs in use is > 75% (checked at 5 min intervals), currently the server seems to stay up for only 1Hr 45min, before we need to reboot it. Mon Jun 17 16:45:00 CDT 2002 0% mbufs in use 1717/1936/240000 mbufs in use (current/peak/max): 1717 mbufs allocated to data 220/278/60000 mbuf clusters in use (current/peak/max) 1040 Kbytes allocated to network (0% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines : : Mon Jun 17 18:15:00 CDT 2002 62% mbufs in use 150769/150960/240000 mbufs in use (current/peak/max): 150762 mbufs allocated to data 7 mbufs allocated to packet headers 212/318/60000 mbuf clusters in use (current/peak/max) 38376 Kbytes allocated to network (21% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Mon Jun 17 18:20:00 CDT 2002 66% mbufs in use 160437/160496/240000 mbufs in use (current/peak/max): 160427 mbufs allocated to data 10 mbufs allocated to packet headers 223/318/60000 mbuf clusters in use (current/peak/max) 40760 Kbytes allocated to network (22% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Mon Jun 17 18:25:01 CDT 2002 72% mbufs in use 172748/172864/240000 mbufs in use (current/peak/max): 172739 mbufs allocated to data 9 mbufs allocated to packet headers 216/318/60000 mbuf clusters in use (current/peak/max) 43852 Kbytes allocated to network (1% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Mon Jun 17 18:30:00 CDT 2002 76% mbufs in use 182638/182848/240000 mbufs in use (current/peak/max): 182638 mbufs allocated to data 212/318/60000 mbuf clusters in use (current/peak/max) 46348 Kbytes allocated to network (2% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines We have also sent messages to ET Inc, and they have advised that the problem is not with their driver, since we are seeing mbufs starvation, and not mbuf cluster starvation occuring. We have tried disabling sendmail, amavisd, named, and haven't been able to figure out what could be causing the mbuf not to be freed properly. Any ideals as to what else we should do to trouble shoot this problem? Scot To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message