Date: Mon, 17 Jun 2002 19:26:32 -0500 From: "Scot W. Hetzel" <hetzels@westbend.net> To: "FreeBSD-Stable" <FreeBSD-Stable@freebsd.org> Subject: System running out of mbufs Message-ID: <011101c2165e$cbc5f400$11fd2fd8@ADMIN00>
next in thread | raw e-mail | index | archive | help
We are currently running:
FreeBSD ns0.westbend.net 4.6-RC FreeBSD 4.6-RC #1: Wed Jun 12 11:39:27 CDT
2002 root@mail.westbend.net:/usr/obj/usr/src/src4/sys/GENERIC-ET i386
The kernel configuration is:
# cvs diff -u GENERIC
Index: GENERIC
===================================================================
RCS file: /home/ncvs/src/sys/i386/conf/GENERIC,v
retrieving revision 1.246.2.43
diff -u -r1.246.2.43 GENERIC
--- GENERIC 23 May 2002 17:04:01 -0000 1.246.2.43
+++ GENERIC 17 Jun 2002 23:56:02 -0000
@@ -56,6 +56,23 @@
options ICMP_BANDLIM #Rate limit bad replies
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
+options IPFIREWALL #firewall
+options IPFIREWALL_VERBOSE #enable logging to syslogd(8)
+options IPFIREWALL_FORWARD #enable transparent proxy support
+options IPFIREWALL_VERBOSE_LIMIT=100 #limit verbosity
+#options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by
default
+options IPV6FIREWALL #firewall for IPv6
+options IPV6FIREWALL_VERBOSE
+options IPV6FIREWALL_VERBOSE_LIMIT=100
+#options IPV6FIREWALL_DEFAULT_TO_ACCEPT
+
+# RANDOM_IP_ID causes the ID field in IP packets to be randomized
+# instead of incremented by 1 with each packet generated. This
+# option closes a minor information leak which allows remote
+# observers to determine the rate of packet generation on the
+# machine by watching the counter.
+options RANDOM_IP_ID
+
# To make an SMP kernel, the next two are needed
#options SMP # Symmetric MultiProcessor Kernel
#options APIC_IO # Symmetric (APIC) I/O
@@ -150,7 +167,7 @@
device npx0 at nexus? port IO_NPX irq 13
# Power management support (see LINT for more options)
-device apm0 at nexus? disable flags 0x20 # Advanced Power
Management
+#device apm0 at nexus? disable flags 0x20 # Advanced
Power Management
# PCCARD (PCMCIA) support
device card
@@ -163,6 +180,11 @@
device sio2 at isa? disable port IO_COM3 irq 5
device sio3 at isa? disable port IO_COM4 irq 9
+options BREAK_TO_DEBUGGER # a BREAK on a comconsole
goes to
+ # DDB, if available.
+options CONSPEED=115200 # speed for serial console
+ # (default 9600)
+
# Parallel port
device ppc0 at isa? irq 7
device ppbus # Parallel port bus (required)
@@ -177,6 +199,10 @@
device em # Intel PRO/1000 adapter Gigabit Ethernet
Card (``Wiseman'')
device txp # 3Com 3cR990 (``Typhoon'')
device vx # 3Com 3c590, 3c595 (``Vortex'')
+
+# ETinc
+device eth0
+#device bw0 at isa ?
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these
NICs!
The problem we are experiencling is that an unusual high number of mbufs are
being used by the system, this problem started about 2 weeks ago with an
April 25th 4.5-STABLE kernel, we have tried, upgrading to 4.6-RC, and
downgrading to 4.4-RELEASE, without any success in solving this problem.
I had posted about this problem before (see "Run away mbufs"), which I
received only one response too. That response had pointed me to a utility
called minfo (by Ian Dowse). The minfo utility had shown quite a few "Free
mbuf not on free list" and "refs missing" in it's output.
We have been using a crontab script to reboot the server when ever the peak
mbufs in use is > 75% (checked at 5 min intervals), currently the server
seems to stay up for only 1Hr 45min, before we need to reboot it.
Mon Jun 17 16:45:00 CDT 2002
0% mbufs in use
1717/1936/240000 mbufs in use (current/peak/max):
1717 mbufs allocated to data
220/278/60000 mbuf clusters in use (current/peak/max)
1040 Kbytes allocated to network (0% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
:
:
Mon Jun 17 18:15:00 CDT 2002
62% mbufs in use
150769/150960/240000 mbufs in use (current/peak/max):
150762 mbufs allocated to data
7 mbufs allocated to packet headers
212/318/60000 mbuf clusters in use (current/peak/max)
38376 Kbytes allocated to network (21% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Mon Jun 17 18:20:00 CDT 2002
66% mbufs in use
160437/160496/240000 mbufs in use (current/peak/max):
160427 mbufs allocated to data
10 mbufs allocated to packet headers
223/318/60000 mbuf clusters in use (current/peak/max)
40760 Kbytes allocated to network (22% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Mon Jun 17 18:25:01 CDT 2002
72% mbufs in use
172748/172864/240000 mbufs in use (current/peak/max):
172739 mbufs allocated to data
9 mbufs allocated to packet headers
216/318/60000 mbuf clusters in use (current/peak/max)
43852 Kbytes allocated to network (1% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Mon Jun 17 18:30:00 CDT 2002
76% mbufs in use
182638/182848/240000 mbufs in use (current/peak/max):
182638 mbufs allocated to data
212/318/60000 mbuf clusters in use (current/peak/max)
46348 Kbytes allocated to network (2% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
We have also sent messages to ET Inc, and they have advised that the problem
is not with their driver, since we are seeing mbufs starvation, and not mbuf
cluster starvation occuring.
We have tried disabling sendmail, amavisd, named, and haven't been able to
figure out what could be causing the mbuf not to be freed properly.
Any ideals as to what else we should do to trouble shoot this problem?
Scot
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?011101c2165e$cbc5f400$11fd2fd8>