Date: Tue, 20 Apr 2004 13:14:52 +0200 From: Stefan Bethke <stb@lassitu.de> To: freebsd-current@freebsd.org Subject: Tuning KVA_PAGES / kmem? Message-ID: <F2014C7A-92BB-11D8-BB31-000393496BE8@lassitu.de>
next in thread | raw e-mail | index | archive | help
Hi,
I'm struggling a bit with a box I'm setting up right now: 2x Xeon 2.8
GHz with 6 GB RAM. Main disk is a RAID 5 on 3ware controller. I'm
running -current from around Easter.
Occasionally, the machine panics with:
panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated
at line 337 in file /usr/src/sys/vm/vm_kern.c
I had stability problems when I was using PAE, so I disabled that for
the moment; even when running GENERIC, I get this panic.
I dug through the archives, and I found some hints that with >>2 GB
RAM, the auto-tuning might produce undesirable results, so I might need
to tune KVA_PAGES to reserve more address space for the kernel, or
fiddle with VM_KMEM_SIZE_MAX, VM_KMEM_SIZE_SCALE, or some vm sysctls.
Can someone point me in the right direction?
Thanks,
Stefan
=========== sysctl vm =============
vm.vmtotal:
System wide totals computed every five seconds: (values in kilobytes)
===============================================
Processes: (RUNQ: 3 Disk Wait: 1 Page Wait: 0 Sleep: 34)
Virtual Memory: (Total: 2098618K, Active 128332K)
Real Memory: (Total: 2174136K Active 77996K)
Shared Virtual Memory: (Total: 11576K Active: 6316K)
Shared Real Memory: (Total: 10532K Active: 5948K)
Free Memory Pages: 1811516K
vm.loadavg: { 4.05 4.00 4.08 }
vm.v_free_min: 6361
vm.v_free_target: 26823
vm.v_free_reserved: 1379
vm.v_inactive_target: 40234
vm.v_cache_min: 26823
vm.v_cache_max: 53646
vm.v_pageout_free_min: 34
vm.pageout_algorithm: 0
vm.swap_enabled: 1
vm.kmem_size: 209715200
vm.swap_async_max: 4
vm.dmmax: 32
vm.nswapdev: 1
vm.swap_idle_threshold1: 2
vm.swap_idle_threshold2: 10
vm.v_free_severe: 3870
vm.stats.sys.v_swtch: 41683765
vm.stats.sys.v_trap: 98922116
vm.stats.sys.v_syscall: 628250901
vm.stats.sys.v_intr: 22170437
vm.stats.sys.v_soft: 16178982
vm.stats.vm.v_vm_faults: 87981696
vm.stats.vm.v_cow_faults: 1666527
vm.stats.vm.v_cow_optim: 4490
vm.stats.vm.v_zfod: 85148244
vm.stats.vm.v_ozfod: 84790971
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_vnodein: 121341
vm.stats.vm.v_vnodeout: 0
vm.stats.vm.v_vnodepgsin: 269302
vm.stats.vm.v_vnodepgsout: 0
vm.stats.vm.v_intrans: 33
vm.stats.vm.v_reactivated: 115830
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_pdpages: 0
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_pfree: 3307919
vm.stats.vm.v_tfree: 89201119
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_page_count: 996725
vm.stats.vm.v_free_reserved: 1379
vm.stats.vm.v_free_target: 26823
vm.stats.vm.v_free_min: 6361
vm.stats.vm.v_free_count: 452621
vm.stats.vm.v_wire_count: 61828
vm.stats.vm.v_active_count: 18866
vm.stats.vm.v_inactive_target: 40234
vm.stats.vm.v_inactive_count: 463270
vm.stats.vm.v_cache_count: 0
vm.stats.vm.v_cache_min: 26823
vm.stats.vm.v_cache_max: 53646
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_forks: 72171
vm.stats.vm.v_vforks: 6433
vm.stats.vm.v_rforks: 2
vm.stats.vm.v_kthreads: 154
vm.stats.vm.v_forkpages: 5914990
vm.stats.vm.v_vforkpages: 800591
vm.stats.vm.v_rforkpages: 118
vm.stats.vm.v_kthreadpages: 0
vm.stats.misc.zero_page_count: 349073
vm.stats.misc.cnt_prezero: 85182803
vm.max_proc_mmap: 34952
vm.msync_flush_flags: 3
vm.idlezero_enable: 1
vm.idlezero_maxrun: 16
vm.max_launder: 32
vm.pageout_stats_max: 26823
vm.pageout_full_stats_interval: 20
vm.pageout_stats_interval: 5
vm.pageout_stats_free_max: 5
vm.swap_idle_enabled: 0
vm.defer_swapspace_pageouts: 0
vm.disable_swapspace_pageouts: 0
vm.pageout_lock_miss: 0
vm.zone:
ITEM SIZE LIMIT USED FREE REQUESTS
FFS2 dinode: 256, 0, 179487, 13863, 1091123
FFS1 dinode: 128, 0, 0, 0, 0
FFS inode: 140, 0, 179487, 13909, 1091123
SWAPMETA: 276, 121576, 0, 0, 0
unpcb: 140, 25620, 14, 98, 22136
ripcb: 180, 25608, 0, 44, 1
tcpreass: 20, 1737, 0, 386, 23
hostcache: 88, 15390, 3, 177, 12
syncache: 104, 15390, 0, 76, 5
tcptw: 56, 5183, 1, 354, 423
tcpcb: 372, 25600, 11, 59, 547
inpcb: 180, 25608, 12, 120, 547
udpcb: 180, 25608, 2, 108, 1420
socket: 240, 25600, 30, 114, 24106
KNOTE: 64, 0, 0, 310, 1380
PIPE: 376, 0, 3, 97, 40955
DIRHASH: 1024, 0, 1162, 374, 1662
NFSNODE: 452, 0, 0, 0, 0
NFSMOUNT: 424, 0, 0, 0, 0
L VFS Cache: 291, 0, 2223, 2821, 22980
S VFS Cache: 68, 0, 188772, 13303, 1246054
NAMEI: 1024, 0, 3, 45, 8161192
VNODEPOLL: 60, 0, 0, 0, 0
VNODE: 260, 0, 192750, 180, 192750
ata_request: 180, 0, 0, 66, 27
g_bio: 136, 0, 0, 5829, 2658969
VMSPACE: 236, 0, 41, 163, 78606
UPCALL: 44, 0, 0, 0, 0
KSE: 84, 0, 360, 91, 360
KSEGRP: 128, 0, 360, 105, 360
TID: 140, 0, 0, 0, 0
THREAD: 336, 0, 360, 36, 360
PROC: 440, 0, 194, 166, 78759
Files: 68, 0, 111, 420, 1715901
4096: 4096, 0, 235, 160, 78847
2048: 2048, 0, 128, 2888, 5858
1024: 1024, 0, 35, 109, 7646
512: 512, 0, 345, 159, 39721
256: 256, 0, 359, 16621, 345303
128: 128, 0, 2104, 16279, 749702
64: 64, 0, 3388, 8082, 365044
32: 32, 0, 1466, 16861, 665787
16: 16, 0, 1000, 673, 84504
DP fakepg: 72, 0, 0, 0, 0
PV ENTRY: 28, 2248680, 31168, 15452, 119085363
MAP ENTRY: 60, 0, 1312, 1196, 3376100
KMAP ENTRY: 60, 57354, 391, 1787, 281213
MAP: 160, 0, 9, 41, 2
VM OBJECT: 132, 0, 178885, 15485, 1678591
128 Bucket: 524, 0, 1274, 0, 0
64 Bucket: 268, 0, 100, 5, 0
32 Bucket: 140, 0, 108, 4, 0
16 Bucket: 76, 0, 54, 50, 0
UMA Hash: 128, 0, 2, 29, 0
UMA Slabs: 34, 0, 2429, 101, 0
UMA Zones: 432, 0, 55, 8, 0
vm.kvm_size: 1069543424
vm.kvm_free: 398454784
=========== sysctl vm =============
=========== netstat -m ============
322/832/51200 mbufs in use (current/peak/max):
322 mbufs allocated to data
320/608/25600 mbuf clusters in use (current/peak/max)
0/1/6656 sfbufs in use (current/peak/max)
64000 KBytes allocated to network (1.1% in use, 2.2% wired)
0 requests for memory denied
0 requests for memory delayed
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
=========== netstat -m ============
========= kernel config ===========
###
### Kernel configuarion file for majestix.tallence.de
###
machine i386
cpu I686_CPU
ident MAJESTIX
###
### General options: protocols, file systems
###
options SCHED_ULE # ULE scheduler
options INET # InterNETworking
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates
support
options UFS_ACL # Support for access control
lists
options UFS_DIRHASH # Improve performance on big
directories
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires
NFSCLIENT
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options COMPAT_43 # Compatible with BSD 4.3 [KEEP
THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options SCSI_DELAY=15000 # Delay (in ms) before probing
SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AHC_REG_PRETTY_PRINT # Print register bitfields in
debug
# output. Adds ~128k to driver.
options AHD_REG_PRETTY_PRINT # Print register bitfields in
debug
# output. Adds ~215k to driver.
options PFIL_HOOKS # pfil(9) framework
###
### Machine-specific:
###
options SMP # Symmetric MultiProcessor Kernel
device apic # I/O APIC
# PAE is incompatible with modules
#options PAE # Physical Address
Extensions Kernel
device acpi
makeoptions NO_MODULES=yes
###
### Busses and drivers
###
device isa
device eisa
device pci
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI
access)
device ses # SCSI Environmental Services (and
SAF-TE)
device miibus # MII bus support
device fdc
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
device ahc # AHA2940 and onboard AIC7xxx devices
device ahd # AHA39320/29320 and onboard AIC79xx
devices
device twe # 3ware ATA RAID
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device vga # VGA video card driver
device sc
device npx
device sio # 8250, 16[45]50 based serial ports
device em # Intel PRO/1000 adapter Gigabit
Ethernet Card
device fxp # Intel EtherExpress PRO/100B (82557,
82558)
device random # Entropy device
device loop # Network loopback
device ether # Ethernet support
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device bpf # Berkeley packet filter
pseudo-device pf 1
========= kernel config ===========
============= dmesg ===============
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved.
FreeBSD 5.2-CURRENT #1: Thu Apr 15 12:14:11 GMT 2004
root@majestix.tallence.de:/usr/obj/usr/src/sys/MAJESTIX
Preloaded elf kernel "/boot/kernel/kernel" at 0xc07df000.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.21-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf25 Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Hyperthreading: 2 logical CPUs
real memory = 4160684032 (3967 MB)
avail memory = 4078157824 (3889 MB)
ACPI APIC Table: <A M I OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
ioapic4 <Version 2.0> irqs 96-119 on motherboard
Pentium Pro MTRR support enabled
random: <entropy source, Software, Yarrow>
acpi0: <A M I OEMRSDT> on motherboard
acpi0: [GIANT-LOCKED]
pcibios: BIOS version 2.10
Found $PIR table, 13 entries at 0xc00f47e0
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
acpi_cpu2: <CPU> on acpi0
acpi_cpu3: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P5 -
AE_NOT_FOUND
pci5: <ACPI PCI bus> on pcib1
pci5: <base peripheral, interrupt controller> at device 28.0 (no driver
attached)
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci5
pci7: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port
0xd880-0xd8bf mem 0xfe9a0000-0xfe9bffff irq 48 at device 1.0 on pci7
em0: Bus reserved 0x20000 bytes for rid 0x10 type 3 at 0xfe9a0000
em0: Bus reserved 0x40 bytes for rid 0x20 type 4 at 0xd880
em0: [GIANT-LOCKED]
em0: Ethernet address: 00:e0:81:28:97:ea
em0: Speed:N/A Duplex:N/A
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port
0xdc00-0xdc3f mem 0xfe9e0000-0xfe9fffff irq 49 at device 1.1 on pci7
em1: Bus reserved 0x20000 bytes for rid 0x10 type 3 at 0xfe9e0000
em1: Bus reserved 0x40 bytes for rid 0x20 type 4 at 0xdc00
em1: [GIANT-LOCKED]
em1: Ethernet address: 00:e0:81:28:97:eb
em1: Speed:N/A Duplex:N/A
pci5: <base peripheral, interrupt controller> at device 30.0 (no driver
attached)
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci5
pci6: <ACPI PCI bus> on pcib3
twe0: <3ware 7000 series Storage Controller. Driver version
1.50.01.000> port 0xcc00-0xcc0f mem 0xfe000000-0xfe7fffff irq 24 at
device 3.0 on pci6
twe0: Bus reserved 0x10 bytes for rid 0x10 type 4 at 0xcc00
twe0: [GIANT-LOCKED]
twe0: 8 ports, Firmware FE7S 1.05.00.063, BIOS BE7X 1.08.00.048
pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pcib4: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P2 -
AE_NOT_FOUND
pci2: <ACPI PCI bus> on pcib4
pci2: <base peripheral, interrupt controller> at device 28.0 (no driver
attached)
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci2
pci4: <ACPI PCI bus> on pcib5
pci2: <base peripheral, interrupt controller> at device 30.0 (no driver
attached)
pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci2
pci3: <ACPI PCI bus> on pcib6
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib7
fxp0: <Intel 82551 Pro/100 Ethernet> port 0x8080-0x80bf mem
0xfd280000-0xfd29ffff,0xfd2fd000-0xfd2fdfff irq 19 at device 1.0 on
pci1
fxp0: Bus reserved 0x1000 bytes for rid 0x10 type 3 at 0xfd2fd000
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:28:97:d8
fxp0: [GIANT-LOCKED]
pci1: <display, VGA> at device 2.0 (no driver attached)
atapci0: <Promise PDC20318 SATA150 controller> port
0x8400-0x847f,0x8480-0x848f,0x8c00-0x8c3f mem
0xfd2a0000-0xfd2bffff,0xfd2fe000-0xfd2fefff irq 18 at device 3.0 on
pci1
atapci0: rid 0x20 says memory, driver wants 4 failed.
atapci0: Bus reserved 0x1000 bytes for rid 0x1c type 3 at 0xfd2fe000
ata2: at 0xfd2fe000 on atapci0
ata3: at 0xfd2fe000 on atapci0
ata4: at 0xfd2fe000 on atapci0
ata5: at 0xfd2fe000 on atapci0
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH3 UDMA100 controller> port 0xffa0-0xffaf at device
31.1 on pci0
atapci1: Bus reserved 0x10 bytes for rid 0x20 type 4 at 0xffa0
ata0: at 0x1f0 irq 14 on atapci1
ata1: at 0x170 irq 15 on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_button1: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0 on motherboard
cpu1 on motherboard
cpu2 on motherboard
cpu3 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
orm0: <Option ROMs> at iomem 0xcb800-0xcc7ff,0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
Timecounters tick every 10.000 msec
acpi_cpu: throttling enabled, 8 steps (100% to 12.5%), currently 100.0%
acd0: DVDROM <DV-25E> at ata0-master PIO4
ad10: 239372MB <Maxtor 7Y250M0> [486344/16/63] at ata5-master UDMA133
twed0: <Unit 0, RAID5, Normal> on twe0
twed0: 718114MB (1470698112 sectors)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Mounting root from ufs:/dev/twed0s1a
============= dmesg ===============
--
Stefan Bethke <stb@lassitu.de> Fon +49 170 346 0140
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F2014C7A-92BB-11D8-BB31-000393496BE8>