From owner-freebsd-current@FreeBSD.ORG Tue Apr 20 04:14:56 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAF4D16A4CF for ; Tue, 20 Apr 2004 04:14:56 -0700 (PDT) Received: from schlepper.zs64.net (schlepper.zs64.net [212.12.50.230]) by mx1.FreeBSD.org (Postfix) with ESMTP id 382E543D1D for ; Tue, 20 Apr 2004 04:14:55 -0700 (PDT) (envelope-from stb@lassitu.de) Received: from [127.0.0.1] (schlepper [212.12.50.230]) by schlepper.zs64.net (8.12.10/8.11.1) with ESMTP id i3KBEr7D017803 for ; Tue, 20 Apr 2004 13:14:53 +0200 (CEST) (envelope-from stb@lassitu.de) Mime-Version: 1.0 (Apple Message framework v613) Content-Transfer-Encoding: 7bit Message-Id: Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: freebsd-current@freebsd.org From: Stefan Bethke Date: Tue, 20 Apr 2004 13:14:52 +0200 X-Mailer: Apple Mail (2.613) Subject: Tuning KVA_PAGES / kmem? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2004 11:14:56 -0000 Hi, I'm struggling a bit with a box I'm setting up right now: 2x Xeon 2.8 GHz with 6 GB RAM. Main disk is a RAID 5 on 3ware controller. I'm running -current from around Easter. Occasionally, the machine panics with: panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated at line 337 in file /usr/src/sys/vm/vm_kern.c I had stability problems when I was using PAE, so I disabled that for the moment; even when running GENERIC, I get this panic. I dug through the archives, and I found some hints that with >>2 GB RAM, the auto-tuning might produce undesirable results, so I might need to tune KVA_PAGES to reserve more address space for the kernel, or fiddle with VM_KMEM_SIZE_MAX, VM_KMEM_SIZE_SCALE, or some vm sysctls. Can someone point me in the right direction? Thanks, Stefan =========== sysctl vm ============= vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 3 Disk Wait: 1 Page Wait: 0 Sleep: 34) Virtual Memory: (Total: 2098618K, Active 128332K) Real Memory: (Total: 2174136K Active 77996K) Shared Virtual Memory: (Total: 11576K Active: 6316K) Shared Real Memory: (Total: 10532K Active: 5948K) Free Memory Pages: 1811516K vm.loadavg: { 4.05 4.00 4.08 } vm.v_free_min: 6361 vm.v_free_target: 26823 vm.v_free_reserved: 1379 vm.v_inactive_target: 40234 vm.v_cache_min: 26823 vm.v_cache_max: 53646 vm.v_pageout_free_min: 34 vm.pageout_algorithm: 0 vm.swap_enabled: 1 vm.kmem_size: 209715200 vm.swap_async_max: 4 vm.dmmax: 32 vm.nswapdev: 1 vm.swap_idle_threshold1: 2 vm.swap_idle_threshold2: 10 vm.v_free_severe: 3870 vm.stats.sys.v_swtch: 41683765 vm.stats.sys.v_trap: 98922116 vm.stats.sys.v_syscall: 628250901 vm.stats.sys.v_intr: 22170437 vm.stats.sys.v_soft: 16178982 vm.stats.vm.v_vm_faults: 87981696 vm.stats.vm.v_cow_faults: 1666527 vm.stats.vm.v_cow_optim: 4490 vm.stats.vm.v_zfod: 85148244 vm.stats.vm.v_ozfod: 84790971 vm.stats.vm.v_swapin: 0 vm.stats.vm.v_swapout: 0 vm.stats.vm.v_swappgsin: 0 vm.stats.vm.v_swappgsout: 0 vm.stats.vm.v_vnodein: 121341 vm.stats.vm.v_vnodeout: 0 vm.stats.vm.v_vnodepgsin: 269302 vm.stats.vm.v_vnodepgsout: 0 vm.stats.vm.v_intrans: 33 vm.stats.vm.v_reactivated: 115830 vm.stats.vm.v_pdwakeups: 0 vm.stats.vm.v_pdpages: 0 vm.stats.vm.v_dfree: 0 vm.stats.vm.v_pfree: 3307919 vm.stats.vm.v_tfree: 89201119 vm.stats.vm.v_page_size: 4096 vm.stats.vm.v_page_count: 996725 vm.stats.vm.v_free_reserved: 1379 vm.stats.vm.v_free_target: 26823 vm.stats.vm.v_free_min: 6361 vm.stats.vm.v_free_count: 452621 vm.stats.vm.v_wire_count: 61828 vm.stats.vm.v_active_count: 18866 vm.stats.vm.v_inactive_target: 40234 vm.stats.vm.v_inactive_count: 463270 vm.stats.vm.v_cache_count: 0 vm.stats.vm.v_cache_min: 26823 vm.stats.vm.v_cache_max: 53646 vm.stats.vm.v_pageout_free_min: 34 vm.stats.vm.v_interrupt_free_min: 2 vm.stats.vm.v_forks: 72171 vm.stats.vm.v_vforks: 6433 vm.stats.vm.v_rforks: 2 vm.stats.vm.v_kthreads: 154 vm.stats.vm.v_forkpages: 5914990 vm.stats.vm.v_vforkpages: 800591 vm.stats.vm.v_rforkpages: 118 vm.stats.vm.v_kthreadpages: 0 vm.stats.misc.zero_page_count: 349073 vm.stats.misc.cnt_prezero: 85182803 vm.max_proc_mmap: 34952 vm.msync_flush_flags: 3 vm.idlezero_enable: 1 vm.idlezero_maxrun: 16 vm.max_launder: 32 vm.pageout_stats_max: 26823 vm.pageout_full_stats_interval: 20 vm.pageout_stats_interval: 5 vm.pageout_stats_free_max: 5 vm.swap_idle_enabled: 0 vm.defer_swapspace_pageouts: 0 vm.disable_swapspace_pageouts: 0 vm.pageout_lock_miss: 0 vm.zone: ITEM SIZE LIMIT USED FREE REQUESTS FFS2 dinode: 256, 0, 179487, 13863, 1091123 FFS1 dinode: 128, 0, 0, 0, 0 FFS inode: 140, 0, 179487, 13909, 1091123 SWAPMETA: 276, 121576, 0, 0, 0 unpcb: 140, 25620, 14, 98, 22136 ripcb: 180, 25608, 0, 44, 1 tcpreass: 20, 1737, 0, 386, 23 hostcache: 88, 15390, 3, 177, 12 syncache: 104, 15390, 0, 76, 5 tcptw: 56, 5183, 1, 354, 423 tcpcb: 372, 25600, 11, 59, 547 inpcb: 180, 25608, 12, 120, 547 udpcb: 180, 25608, 2, 108, 1420 socket: 240, 25600, 30, 114, 24106 KNOTE: 64, 0, 0, 310, 1380 PIPE: 376, 0, 3, 97, 40955 DIRHASH: 1024, 0, 1162, 374, 1662 NFSNODE: 452, 0, 0, 0, 0 NFSMOUNT: 424, 0, 0, 0, 0 L VFS Cache: 291, 0, 2223, 2821, 22980 S VFS Cache: 68, 0, 188772, 13303, 1246054 NAMEI: 1024, 0, 3, 45, 8161192 VNODEPOLL: 60, 0, 0, 0, 0 VNODE: 260, 0, 192750, 180, 192750 ata_request: 180, 0, 0, 66, 27 g_bio: 136, 0, 0, 5829, 2658969 VMSPACE: 236, 0, 41, 163, 78606 UPCALL: 44, 0, 0, 0, 0 KSE: 84, 0, 360, 91, 360 KSEGRP: 128, 0, 360, 105, 360 TID: 140, 0, 0, 0, 0 THREAD: 336, 0, 360, 36, 360 PROC: 440, 0, 194, 166, 78759 Files: 68, 0, 111, 420, 1715901 4096: 4096, 0, 235, 160, 78847 2048: 2048, 0, 128, 2888, 5858 1024: 1024, 0, 35, 109, 7646 512: 512, 0, 345, 159, 39721 256: 256, 0, 359, 16621, 345303 128: 128, 0, 2104, 16279, 749702 64: 64, 0, 3388, 8082, 365044 32: 32, 0, 1466, 16861, 665787 16: 16, 0, 1000, 673, 84504 DP fakepg: 72, 0, 0, 0, 0 PV ENTRY: 28, 2248680, 31168, 15452, 119085363 MAP ENTRY: 60, 0, 1312, 1196, 3376100 KMAP ENTRY: 60, 57354, 391, 1787, 281213 MAP: 160, 0, 9, 41, 2 VM OBJECT: 132, 0, 178885, 15485, 1678591 128 Bucket: 524, 0, 1274, 0, 0 64 Bucket: 268, 0, 100, 5, 0 32 Bucket: 140, 0, 108, 4, 0 16 Bucket: 76, 0, 54, 50, 0 UMA Hash: 128, 0, 2, 29, 0 UMA Slabs: 34, 0, 2429, 101, 0 UMA Zones: 432, 0, 55, 8, 0 vm.kvm_size: 1069543424 vm.kvm_free: 398454784 =========== sysctl vm ============= =========== netstat -m ============ 322/832/51200 mbufs in use (current/peak/max): 322 mbufs allocated to data 320/608/25600 mbuf clusters in use (current/peak/max) 0/1/6656 sfbufs in use (current/peak/max) 64000 KBytes allocated to network (1.1% in use, 2.2% wired) 0 requests for memory denied 0 requests for memory delayed 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines =========== netstat -m ============ ========= kernel config =========== ### ### Kernel configuarion file for majestix.tallence.de ### machine i386 cpu I686_CPU ident MAJESTIX ### ### General options: protocols, file systems ### options SCHED_ULE # ULE scheduler options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. options PFIL_HOOKS # pfil(9) framework ### ### Machine-specific: ### options SMP # Symmetric MultiProcessor Kernel device apic # I/O APIC # PAE is incompatible with modules #options PAE # Physical Address Extensions Kernel device acpi makeoptions NO_MODULES=yes ### ### Busses and drivers ### device isa device eisa device pci # SCSI peripherals device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) device miibus # MII bus support device fdc device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering device ahc # AHA2940 and onboard AIC7xxx devices device ahd # AHA39320/29320 and onboard AIC79xx devices device twe # 3ware ATA RAID device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device sc device npx device sio # 8250, 16[45]50 based serial ports device em # Intel PRO/1000 adapter Gigabit Ethernet Card device fxp # Intel EtherExpress PRO/100B (82557, 82558) device random # Entropy device device loop # Network loopback device ether # Ethernet support device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device bpf # Berkeley packet filter pseudo-device pf 1 ========= kernel config =========== ============= dmesg =============== Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.2-CURRENT #1: Thu Apr 15 12:14:11 GMT 2004 root@majestix.tallence.de:/usr/obj/usr/src/sys/MAJESTIX Preloaded elf kernel "/boot/kernel/kernel" at 0xc07df000. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.21-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff Hyperthreading: 2 logical CPUs real memory = 4160684032 (3967 MB) avail memory = 4078157824 (3889 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard ioapic3 irqs 72-95 on motherboard ioapic4 irqs 96-119 on motherboard Pentium Pro MTRR support enabled random: acpi0: on motherboard acpi0: [GIANT-LOCKED] pcibios: BIOS version 2.10 Found $PIR table, 13 entries at 0xc00f47e0 acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_cpu0: on acpi0 acpi_cpu1: on acpi0 acpi_cpu2: on acpi0 acpi_cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P5 - AE_NOT_FOUND pci5: on pcib1 pci5: at device 28.0 (no driver attached) pcib2: at device 29.0 on pci5 pci7: on pcib2 em0: port 0xd880-0xd8bf mem 0xfe9a0000-0xfe9bffff irq 48 at device 1.0 on pci7 em0: Bus reserved 0x20000 bytes for rid 0x10 type 3 at 0xfe9a0000 em0: Bus reserved 0x40 bytes for rid 0x20 type 4 at 0xd880 em0: [GIANT-LOCKED] em0: Ethernet address: 00:e0:81:28:97:ea em0: Speed:N/A Duplex:N/A em1: port 0xdc00-0xdc3f mem 0xfe9e0000-0xfe9fffff irq 49 at device 1.1 on pci7 em1: Bus reserved 0x20000 bytes for rid 0x10 type 3 at 0xfe9e0000 em1: Bus reserved 0x40 bytes for rid 0x20 type 4 at 0xdc00 em1: [GIANT-LOCKED] em1: Ethernet address: 00:e0:81:28:97:eb em1: Speed:N/A Duplex:N/A pci5: at device 30.0 (no driver attached) pcib3: at device 31.0 on pci5 pci6: on pcib3 twe0: <3ware 7000 series Storage Controller. Driver version 1.50.01.000> port 0xcc00-0xcc0f mem 0xfe000000-0xfe7fffff irq 24 at device 3.0 on pci6 twe0: Bus reserved 0x10 bytes for rid 0x10 type 4 at 0xcc00 twe0: [GIANT-LOCKED] twe0: 8 ports, Firmware FE7S 1.05.00.063, BIOS BE7X 1.08.00.048 pcib4: at device 3.0 on pci0 pcib4: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P2 - AE_NOT_FOUND pci2: on pcib4 pci2: at device 28.0 (no driver attached) pcib5: at device 29.0 on pci2 pci4: on pcib5 pci2: at device 30.0 (no driver attached) pcib6: at device 31.0 on pci2 pci3: on pcib6 pci0: at device 29.0 (no driver attached) pci0: at device 29.1 (no driver attached) pci0: at device 29.2 (no driver attached) pcib7: at device 30.0 on pci0 pci1: on pcib7 fxp0: port 0x8080-0x80bf mem 0xfd280000-0xfd29ffff,0xfd2fd000-0xfd2fdfff irq 19 at device 1.0 on pci1 fxp0: Bus reserved 0x1000 bytes for rid 0x10 type 3 at 0xfd2fd000 miibus0: on fxp0 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:e0:81:28:97:d8 fxp0: [GIANT-LOCKED] pci1: at device 2.0 (no driver attached) atapci0: port 0x8400-0x847f,0x8480-0x848f,0x8c00-0x8c3f mem 0xfd2a0000-0xfd2bffff,0xfd2fe000-0xfd2fefff irq 18 at device 3.0 on pci1 atapci0: rid 0x20 says memory, driver wants 4 failed. atapci0: Bus reserved 0x1000 bytes for rid 0x1c type 3 at 0xfd2fe000 ata2: at 0xfd2fe000 on atapci0 ata3: at 0xfd2fe000 on atapci0 ata4: at 0xfd2fe000 on atapci0 ata5: at 0xfd2fe000 on atapci0 isab0: at device 31.0 on pci0 isa0: on isab0 atapci1: port 0xffa0-0xffaf at device 31.1 on pci0 atapci1: Bus reserved 0x10 bytes for rid 0x20 type 4 at 0xffa0 ata0: at 0x1f0 irq 14 on atapci1 ata1: at 0x170 irq 15 on atapci1 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 acpi_button1: on acpi0 atkbdc0: port 0x64,0x60 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0 port 0x3f8-0x3ff irq 4 on acpi0 sio0: type 16550A, console sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1 port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A fdc0: port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0 on motherboard cpu1 on motherboard cpu2 on motherboard cpu3 on motherboard npx0: [FAST] npx0: on motherboard npx0: INT 16 interface orm0: