From owner-freebsd-hackers@FreeBSD.ORG Wed Jul 23 10:30:12 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 72CD537B401 for ; Wed, 23 Jul 2003 10:30:12 -0700 (PDT) Received: from smartrafficenter.org (pacer.smartrafficenter.org [207.14.56.3]) by mx1.FreeBSD.org (Postfix) with SMTP id BF73943FAF for ; Wed, 23 Jul 2003 10:30:10 -0700 (PDT) (envelope-from kpieckiel@smartrafficenter.org) Received: (qmail 70705 invoked by uid 1500); 23 Jul 2003 17:30:07 -0000 Date: Wed, 23 Jul 2003 13:30:07 -0400 From: "Kevin A. Pieckiel" To: freebsd-hackers@freebsd.org Message-ID: <20030723173007.GD41280@pacer.dmz.smartrafficenter.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="0vzXIDBeUiKkjNJl" Content-Disposition: inline User-Agent: Mutt/1.4i Subject: mbuf cluster shortage caused kernel panic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2003 17:30:12 -0000 --0vzXIDBeUiKkjNJl Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline Content-Transfer-Encoding: quoted-printable #uname -a FreeBSD fileserver1.smartrafficenter.net 4.7-STABLE FreeBSD 4.7-STABLE #0: = Mon Dec 16 19:41:03 EST 2002 toor@fileserver1.smartrafficenter.net:/usr= /obj/usr/src/sys/FILESERVER1 i386 Running 4.7 stable with sources CVSed on 16 Dec 2002. My fileserver has been running since 17 Dec 2002 and suddenly lost its ability to talk on the network today. Went to the console to discover a flood of messages that it was out of mbuf clusters, read tuning(7) for more info. I immediately ran netstat -m and saw indeed there were 6656 mbuf clusters allocated, 6656 used, and 6656 peak. A number of requests for memory were being denied at this point. While browsing the tuning(7) man page for where and how to increase this number, the kernel panicked. The computer is a Dell PowerEdge 1650 with two 1.4 GHz P-III processors and 1 GB RAM. Output of dmesg and gdb on debug kernel and core file is provided below. Kernel config file is attached. What can I do to help solve any problems that might exist in the kernel code, and what suggestions do you have to keep this from happening on my fileserver again? Kernel, debug kernel, CVS date, kernel config, and core file can be made available upon request. Thanks much, Kevin A. Pieckiel ---- #dmesg Copyright (c) 1992-2002 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.7-STABLE #0: Mon Dec 16 19:41:03 EST 2002 toor@fileserver1.smartrafficenter.net:/usr/obj/usr/src/sys/FILESERVER1 Timecounter "i8254" frequency 1193182 Hz CPU: Pentium III/Pentium III Xeon/Celeron (1396.45-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0x6b1 Stepping =3D 1 Features=3D0x383fbff real memory =3D 1073676288 (1048512K bytes) avail memory =3D 1042231296 (1017804K bytes) Changing APIC ID for IO APIC #0 from 0 to 2 on chip Changing APIC ID for IO APIC #1 from 0 to 3 on chip Programming 16 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 Programming 16 pins in IOAPIC #1 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x000f0011, at 0xfec00000 io1 (APIC): apic id: 3, version: 0x000f0011, at 0xfec01000 Preloaded elf kernel "kernel" at 0xc02d1000. Preloaded userconfig_script "/boot/kernel.conf" at 0xc02d109c. Pentium Pro MTRR support enabled Using $PIR table, 9 entries at 0xc00fc220 npx0: on motherboard npx0: INT 16 interface pcib1: on motherboard IOAPIC #1 intpin 1 -> irq 2 IOAPIC #1 intpin 0 -> irq 3 IOAPIC #1 intpin 2 -> irq 5 IOAPIC #1 intpin 3 -> irq 7 pci1: on pcib1 em0: port 0xdce0-0x= dcff mem 0xfeb40000-0xfeb5ffff,0xfeb60000-0xfeb7ffff irq 2 at device 2.0 on= pci1 em0: Speed:1000 Mbps Duplex:Full em1: port 0xdcc0-0x= dcdf mem 0xfeb00000-0xfeb1ffff,0xfeb20000-0xfeb3ffff irq 3 at device 4.0 on= pci1 em1: Speed:N/A Duplex:N/A ahc0: port 0xd800-0xd8ff mem 0xfeb8= 1000-0xfeb81fff irq 5 at device 6.0 on pci1 aic7899: Ultra160 Wide Channel A, SCSI Id=3D7, 32/253 SCBs ahc1: port 0xd400-0xd4ff mem 0xfeb8= 0000-0xfeb80fff irq 7 at device 6.1 on pci1 aic7899: Ultra160 Wide Channel B, SCSI Id=3D7, 32/253 SCBs pcib0: on motherboard pci0: on pcib0 pci0: at 12.0 atapci0: port 0x8b0-0x8bf,0x374-0x377,= 0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 at device 15.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 pci0: at 15.2 irq 11 isab0: at device 15.3 on = pci0 isa0: on isab0 pcib2: on motherboard IOAPIC #1 intpin 4 -> irq 13 IOAPIC #1 intpin 8 -> irq 16 IOAPIC #1 intpin 9 -> irq 17 pci2: on pcib2 ahc2: port 0xcc00-0xccff mem 0xfe903000= -0xfe903fff irq 13 at device 8.0 on pci2 aic7896/97: Ultra2 Wide Channel A, SCSI Id=3D7, 32/253 SCBs ahc3: port 0xc800-0xc8ff mem 0xfe902000= -0xfe902fff irq 13 at device 8.1 on pci2 aic7896/97: Ultra2 Wide Channel B, SCSI Id=3D7, 32/253 SCBs ahc4: port 0xc400-0xc4ff mem 0xfe9010= 00-0xfe901fff irq 16 at device 10.0 on pci2 aic7899: Ultra160 Wide Channel A, SCSI Id=3D7, 32/253 SCBs ahc5: port 0xc000-0xc0ff mem 0xfe9000= 00-0xfe900fff irq 17 at device 10.1 on pci2 aic7899: Ultra160 Wide Channel B, SCSI Id=3D7, 32/253 SCBs pcib3: on motherboard pci3: on pcib3 pcib4: on motherboard pci4: on pcib4 orm0: