Date: Tue, 16 Mar 2004 09:08:18 -0500 From: Sven Willenberger <sven@dmv.com> To: freebsd-current@freebsd.org Subject: kmem_map too small, revisited Message-ID: <1079446098.23554.49.camel@lanshark.dmv.com>
next in thread | raw e-mail | index | archive | help
Preamble: I am looking for assistance in how to better troubleshoot this
issue of kmem_map. I have googled and read the newsgroups to no avail.
The Issue: Ever since having migrated from 4.x to 5.x I have been having
an issue (on 2 different hardware setups) with spontaneous reboots or
system hangs, usually identified by the kmem_map too small message. The
hardware is predominantly Supermicro dual Xeon processor machines
(though a single xeon non S/M machine exhibits similar behavior).
The machines are used essentially as email filtering machines running
sendmail/mimedefang/spamassassin and process upwards of 150K messages
per day each. With the exception of 1 single processor machine, all have
dual procs with 1GB of ECC DRAM.
Where should I begin to better diagnose what exactly is going on here?
Latest dmesg:
Sorry, need DDB option to print backtrace
panic: kmem_malloc(16384): kmem_map too small: 588120064 total allocated
cpuid = 2;
boot() called on cpu#2
syncing disks, buffers remaining... 6073 6073 6070 6070 6070 6070 6070
6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070 6070
6070
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved.
FreeBSD 5.2.1-RC #0: Mon Feb 2 10:01:14 EST 2004
svenw@cartman.dmv.com:/usr/obj/usr/src/sys/CARTMAN
Preloaded elf kernel "/boot/kernel/kernel" at 0xc081d000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc081d2bc.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.00GHz (1996.60-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf27 Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Hyperthreading: 2 logical CPUs
real memory = 1073217536 (1023 MB)
avail memory = 1032687616 (984 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
Pentium Pro MTRR support enabled
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <PTLTD RSDT > on motherboard
pcibios: BIOS version 2.10
Using $PIR table, 25 entries at 0xc00fde30
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.HLB_ -
AE_NOT_FOUND
pci1: <ACPI PCI bus> on pcib1
pci1: <base peripheral, interrupt controller> at device 28.0 (no driver
attached)
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.19> port
0x3000-0x301f mem 0xfc200000-0xfc21ffff,0xfc220000-0xfc23ffff irq 54 at
device 3.0 on pci2
em0: Speed:N/A Duplex:N/A
pci1: <base peripheral, interrupt controller> at device 30.0 (no driver
attached)
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1
pci3: <ACPI PCI bus> on pcib3
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x4000-0x40ff mem
0xfc300000-0xfc300fff irq 32 at device 2.0 on pci3
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x4400-0x44ff mem
0xfc301000-0xfc301fff irq 33 at device 2.1 on pci3
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
uhci0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> port
0x2000-0x201f irq 16 at device 29.0 on pci0
usb0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> port
0x2020-0x203f irq 19 at device 29.1 on pci0
usb1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> port
0x2040-0x205f irq 18 at device 29.2 on pci0
usb2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci4: <display, VGA> at device 4.0 (no driver attached)
fxp0: <Intel 82550 Pro/100 Ethernet> port 0x5400-0x543f mem
0xfc420000-0xfc43ffff,0xfc401000-0xfc401fff irq 22 at device 5.0 on pci4
fxp0: Ethernet address 00:30:48:24:ca:e1
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port
0x2060-0x206f,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 mem
0xfc000000-0xfc0003ff at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata0: [MPSAFE]
ata1: at 0x170 irq 15 on atapci0
ata1: [MPSAFE]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0 port 0x778-0x77f,0x378-0x37f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <Option ROMs> at iomem
0xe0000-0xe3fff,0xdc000-0xdffff,0xc8000-0xc8fff,0xc0000-0xc7fff on isa0
sc0: <System console> on isa0
sc0: VGA <16 virtual consoles, flags=0x200>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
Timecounters tick every 10.000 msec
Waiting 7 seconds for SCSI devices to settle
GEOM: create disk da0 dp=0xc6ba9c50
da0 at ahc1 bus 0 target 3 lun 0
da0: <IBM IC35L018UWD210-0 S5CS> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged
Queueing Enabled
da0: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Mounting root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 1364 files 126
Snippet from the kernel CONF file on what is now a 5.2.1-RC machine:
# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.384.2.2 2003/05/31 15:18:41
scottl Exp $
machine i386
cpu I586_CPU
cpu I686_CPU
ident GENERIC
#To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" #Default places to look for
devices.
#makeoptions DEBUG=-g #Build kernel with gdb(1) debug
symbols
options SCHED_4BSD #4BSD scheduler
options INET #InterNETworking
options INET6 #IPv6 communications protocols
options FFS #Berkeley Fast Filesystem
options SOFTUPDATES #Enable FFS soft updates support
options UFS_ACL #Support for access control
lists
options UFS_DIRHASH #Improve performance on big
directories
options MD_ROOT #MD is a potential root device
options NFSCLIENT #Network Filesystem Client
options NFSSERVER #Network Filesystem Server
options NFS_ROOT #NFS usable as root device,
requires NFSCLIENT
options MSDOSFS #MSDOS Filesystem
options CD9660 #ISO 9660 Filesystem
options PROCFS #Process filesystem (requires
PSEUDOFS)
options PSEUDOFS #Pseudo-filesystem framework
options COMPAT_43 #Compatible with BSD 4.3 [KEEP
THIS!]
options COMPAT_FREEBSD4 #Compatible with FreeBSD4
options SCSI_DELAY=7000 #Delay (in ms) before probing
SCSI
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AHC_REG_PRETTY_PRINT # Print register bitfields in
debug
# output. Adds ~128k to driver.
options AHD_REG_PRETTY_PRINT # Print register bitfields in
debug
# output. Adds ~215k to driver.
# Debugging for use in -current
#options DDB #Enable the kernel debugger
#options INVARIANTS #Enable calls of extra sanity
checking
options INVARIANT_SUPPORT #Extra sanity checks of internal
structures, required by INVARIANTS
#options WITNESS #Enable checks to detect
deadlocks and cycles
#options WITNESS_SKIPSPIN #Don't run witness on spinlocks
for speed
# Deal with kmem issues
options VM_KMEM_SIZE_SCALE="2"
options VM_KMEM_SIZE_MAX="(512*1024*1024)"
#options KVA_PAGES=640
# To make an SMP kernel, the next two are needed
options SMP # Symmetric MultiProcessor
Kernel
device apic # Symmetric (APIC) I/O
Thanks in advance,
Sven
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1079446098.23554.49.camel>