From owner-freebsd-current@FreeBSD.ORG Tue Mar 16 06:09:06 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8624C16A4CE for ; Tue, 16 Mar 2004 06:09:06 -0800 (PST) Received: from smtp-gw-cl-d.dmv.com (smtp-gw-cl-d.dmv.com [216.240.97.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id E3FA743D49 for ; Tue, 16 Mar 2004 06:09:03 -0800 (PST) (envelope-from sven@dmv.com) Received: from lanshark.dmv.com (lanshark.dmv.com [216.240.97.46]) i2GE8pRv092949 for ; Tue, 16 Mar 2004 09:08:51 -0500 (EST) (envelope-from sven@dmv.com) From: Sven Willenberger To: freebsd-current@freebsd.org Content-Type: text/plain Message-Id: <1079446098.23554.49.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 Date: Tue, 16 Mar 2004 09:08:18 -0500 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.39 Subject: kmem_map too small, revisited X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2004 14:09:06 -0000 Preamble: I am looking for assistance in how to better troubleshoot this issue of kmem_map. I have googled and read the newsgroups to no avail. The Issue: Ever since having migrated from 4.x to 5.x I have been having an issue (on 2 different hardware setups) with spontaneous reboots or system hangs, usually identified by the kmem_map too small message. The hardware is predominantly Supermicro dual Xeon processor machines (though a single xeon non S/M machine exhibits similar behavior). The machines are used essentially as email filtering machines running sendmail/mimedefang/spamassassin and process upwards of 150K messages per day each. With the exception of 1 single processor machine, all have dual procs with 1GB of ECC DRAM. Where should I begin to better diagnose what exactly is going on here? Latest dmesg: Sorry, need DDB option to print backtrace panic: kmem_malloc(16384): kmem_map too small: 588120064 total allocated cpuid = 2; boot() called on cpu#2 syncing disks, buffers remaining... 6073 6073 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.2.1-RC #0: Mon Feb 2 10:01:14 EST 2004 svenw@cartman.dmv.com:/usr/obj/usr/src/sys/CARTMAN Preloaded elf kernel "/boot/kernel/kernel" at 0xc081d000. Preloaded elf module "/boot/kernel/acpi.ko" at 0xc081d2bc. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.00GHz (1996.60-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 Features=0xbfebfbff Hyperthreading: 2 logical CPUs real memory = 1073217536 (1023 MB) avail memory = 1032687616 (984 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard Pentium Pro MTRR support enabled npx0: [FAST] npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard pcibios: BIOS version 2.10 Using $PIR table, 25 entries at 0xc00fde30 acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 acpi_cpu0: on acpi0 acpi_cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.1 (no driver attached) pcib1: at device 2.0 on pci0 pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.HLB_ - AE_NOT_FOUND pci1: on pcib1 pci1: at device 28.0 (no driver attached) pcib2: at device 29.0 on pci1 pci2: on pcib2 em0: port 0x3000-0x301f mem 0xfc200000-0xfc21ffff,0xfc220000-0xfc23ffff irq 54 at device 3.0 on pci2 em0: Speed:N/A Duplex:N/A pci1: at device 30.0 (no driver attached) pcib3: at device 31.0 on pci1 pci3: on pcib3 ahc0: port 0x4000-0x40ff mem 0xfc300000-0xfc300fff irq 32 at device 2.0 on pci3 aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs ahc1: port 0x4400-0x44ff mem 0xfc301000-0xfc301fff irq 33 at device 2.1 on pci3 aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs uhci0: port 0x2000-0x201f irq 16 at device 29.0 on pci0 usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x2020-0x203f irq 19 at device 29.1 on pci0 usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x2040-0x205f irq 18 at device 29.2 on pci0 usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered pcib4: at device 30.0 on pci0 pci4: on pcib4 pci4: at device 4.0 (no driver attached) fxp0: port 0x5400-0x543f mem 0xfc420000-0xfc43ffff,0xfc401000-0xfc401fff irq 22 at device 5.0 on pci4 fxp0: Ethernet address 00:30:48:24:ca:e1 miibus0: on fxp0 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x2060-0x206f,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 mem 0xfc000000-0xfc0003ff at device 31.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata0: [MPSAFE] ata1: at 0x170 irq 15 on atapci0 ata1: [MPSAFE] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atkbdc0: port 0x64,0x60 irq 1 on acpi0 sio0 port 0x3f8-0x3ff irq 4 on acpi0 sio0: type 16550A sio1 port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A fdc0: port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ppc0 port 0x778-0x77f,0x378-0x37f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: on ppc0 ppi0: on ppbus0 pmtimer0 on isa0 orm0: