Date: Sat, 4 Nov 2006 00:36:38 +0100 From: Stefan Bethke <stb@lassitu.de> To: FreeBSD Stable <freebsd-stable@freebsd.org> Cc: Andreas Sons <sons@zs64.net>, Robert Watson <rwatson@freebsd.org> Subject: Re: panic: kmem_map too small Message-ID: <7E5496DA-06AC-4347-A807-DFB43D7E3571@lassitu.de> In-Reply-To: <02A6C9B5-1227-4D0A-AEF2-73B6187E1A08@lassitu.de> References: <CD619308-6B7B-4B68-B011-CE979D67E0C1@lassitu.de> <20061025183308.L33725@fledge.watson.org> <838FCA83-20F8-4A09-A025-E69956032F86@lassitu.de> <45400286.9020402@samsco.org> <20061026091253.J69980@fledge.watson.org> <20061026111723.K33725@fledge.watson.org> <20152564-EC53-4210-9590-23817A6A3E42@lassitu.de> <E4AD8D20-E934-4E3E-9485-FE4985CDF57A@lassitu.de> <20061026143337.L29443@fledge.watson.org> <02A6C9B5-1227-4D0A-AEF2-73B6187E1A08@lassitu.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Am 26.10.2006 um 15:47 schrieb Stefan Bethke: > Am 26.10.2006 um 15:37 schrieb Robert Watson: > >> On Thu, 26 Oct 2006, Stefan Bethke wrote: >> >>> acpica 3024 159K 20026966 >> ... >>> db> show uma >>> Zone Allocs Frees Used Cache >>> 64 9990754 9986054 4700 9980755 >> >> Looks like acpica has gone crazy performing allocation/freeing at >> a very high rate, and that for some reason, UMA is failing to >> properly reuse/release memory. So there are two bugs/problems >> here: whatever is causing ACPI to behave this way, and then the >> fact that UMA is failing to deal properly with its misbehavior. > > We had the machines running with ACPI disabled for a week or so, > and we were still getting these panics, but I'll disable it again > in the BIOS to make sure. > >> Alternatively, that we have a bug in the way statistics are >> handled. If you can generate a coredump, it would be quite useful >> to be able to run umstat (src/tools/tools/umastat in HEAD) on it. >> The tool probably needs a bit of tweaking to run on the core dump >> -- in particular, the first and second arguments of kvm_open() >> need to be the name of the kernel and dumpfile, rather than NULL. >> This would help confirm what actual state UMA is in. > > So far, the machines always just hang instead of dumping core; I'll > see if I can get them to write a dump. On two of the three boxes, the panics have not happened for a week. On the one I disabled ACPI on, no panics were seen for a week as well, so I enabled it again. Within an hour, the panic occurred. call doadump just hangs, so I won't be able to run umastat. I'll try to kick the box via the IPMI card and wait for the next panic. Fri Nov 3 23:22:58 CET 2006 FreeBSD/i386 (discovery.tallence.de) (ttyd0) login: -0255: *** Error: UtCallocate: Could not allocate size 30 -0255: *** Error: UtCallocate: Could not allocate size 30 -0255: *** Error: UtCallocate: Could not allocate size 30 ACPI-1304: *** Error: Method execution failed [\SWFS] (Node 0xc63ee220), AE_NO_MEMORY ACPI-1304: *** Error: Method execution failed [\RBYT] (Node 0xc63ee1a0), AE_NO_MEMORY ACPI-1304: *** Error: Method execution failed [\RTMP] (Node 0xc63ee380), AE_NO_MEMORY ACPI-1304: *** Error: Method execution failed [\_TZ_.THRM._TMP] (Node 0xc63ede00), AE_NO_MEMORY panic: kmem_malloc(4096): kmem_map too small: 699756544 total allocated KDB: enter: panic [thread pid 1119 tid 100074 ] Stopped at kdb_enter+0x30: leave db> show uma Zone Allocs Frees Used Cache pfosfp 188 0 188 193 pfospfen 345 0 345 51 pfiaddrpl 0 0 0 0 pfstatescrub 0 0 0 0 pffrcent 0 0 0 0 pffrcache 0 0 0 0 pffrag 0 0 0 0 pffrent 0 0 0 0 pfrkentry2 0 0 0 0 pfrkentry 0 0 0 0 pfrktable 0 0 0 0 pfpooladdrpl 0 0 0 0 pfaltqpl 0 0 0 0 pfstatepl 873 844 29 76 pfrulepl 62 0 62 10 pfsrctrpl 1 0 1 77 FFS2 dinode 2117 193 1924 11 FFS1 dinode 0 0 0 0 FFS inode 2117 193 1924 48 Mountpoints 19 0 19 5 SWAPMETA 0 0 0 0 rtentry 10 0 10 48 unpcb 664 613 51 33 ripcb 0 0 0 0 sackhole 0 0 0 0 tcpreass 9 9 0 169 hostcache 39 0 39 61 syncache 150 150 0 78 tcptw 55 55 0 156 tcpcb 229 198 31 25 inpcb 229 198 31 35 udpcb 979 959 20 24 ipq 0 0 0 0 socket 1873 1771 102 8 KNOTE 1242 1242 0 112 PIPE 1033 1022 11 16 NFSNODE 0 0 0 0 NFSMOUNT 0 0 0 0 DIRHASH 91 0 91 5 NAMEI 42625 42625 0 20 L VFS Cache 0 0 0 0 S VFS Cache 6029 4101 1928 32 VNODEPOLL 0 0 0 0 VNODE 2211 199 2012 4 ata_composite 0 0 0 0 ata_request 12690 12690 0 380 g_bio 47901 47901 0 1247 ACL UMA zone 0 0 0 0 mbuf_jumbo_16k 0 0 0 0 mbuf_jumbo_9k 0 0 0 0 mbuf_jumbo_pagesize 0 0 0 0 mbuf_cluster 128 0 128 6 mbuf 8800 8772 28 114 mbuf_packet 47880 47812 68 60 VMSPACE 2384 2317 67 37 UPCALL 6 0 6 150 KSEGRP 129 0 129 71 THREAD 216 81 135 15 PROC 2424 2317 107 19 Files 17415 17156 259 59 4096 5808 5666 142 25 2048 554 185 369 5 1024 2266 2218 48 144 512 3396 3339 57 15 256 6678 6240 438 42 128 24937 22580 2357 73 64 9966167 9961335 4832 9947229 32 5515 2447 3068 96 16 10000705 9997868 2837 208 mt_zone 163 0 163 73 DP fakepg 0 0 0 0 PV ENTRY 1242630 1213645 28985 1320 MAP ENTRY 88676 86884 1792 0 KMAP ENTRY 171204 171178 26 142 MAP 7 0 7 33 VM OBJECT 45712 43104 2608 321 128 Bucket 58 2 56 0 64 Bucket 49 1 48 8 32 Bucket 31 0 31 25 16 Bucket 30 0 30 20 UMA Hash 6 1 5 25 UMA RCntSlabs 67 0 67 7 UMA Slabs 669 193 476 55 UMA Zones 83 0 83 7 UMA Kegs 83 0 83 13 db> show malloc Type InUse MemUse Requests NULLFS mount 8 1K 8 NULLFS hash 1 1K 1 NULLFS node 8 1K 8 pflog 1 1K 1 pf_if 26 6K 26 MADT Table 0 0K 0 acpipwr 0 0K 0 acpi_perf 0 0K 0 acpidev 93 3K 93 acpisem 17 2K 17 acpicmbat 0 0K 0 PCI Link 64 6K 64 acpitask 0 0K 2 acpica 3024 159K 19943808 KTRACE 100 13K 100 prison 4 8K 4 $PIR 0 0K 0 DEVFS3 562 71K 563 nexusdev 3 1K 3 MP Table 0 0K 0 memdesc 1 4K 1 legacydrv 0 0K 0 ithread 66 6K 66 I/O APIC 1 1K 1 zombie 0 0K 2317 proc-args 52 4K 5377 kqueue 0 0K 1234 kenv 113 8K 114 atkbddev 2 1K 2 sigio 1 1K 1 file desc to leader 0 0K 0 VM pgdata 2 65K 2 file desc 108 27K 2425 DEVFS2 100 4K 400 USBHC 0 0K 0 cdev 20 3K 20 USBdev 3 1K 9 UMAHash 1 1K 2 UFS mount 18 82K 18 UFS quota 0 0K 0 UFS dirhash 87 16K 87 savedino 0 0K 0 newdirblk 0 0K 0 dirrem 0 0K 0 mkdir 0 0K 0 diradd 0 0K 0 freefile 0 0K 0 freeblks 0 0K 0 freefrag 0 0K 0 allocindir 0 0K 0 indirdep 0 0K 0 allocdirect 0 0K 0 bmsafemap 0 0K 0 newblk 1 1K 1 inodedep 1 256K 1 pagedep 1 64K 1 rpcclnt 0 0K 0 p1003.1b 1 1K 1 agp 0 0K 0 NFS daemon 5 10K 5 NFSV3 srvdesc 0 0K 0 NFS srvsock 1 1K 1 nlminfo 0 0K 0 NFS lock 0 0K 0 NFS DirectIO 0 0K 0 NFS hash 0 0K 0 NFSV3 diroff 0 0K 0 NFSV3 bigfh 0 0K 0 NFS req 0 0K 0 NFS srvsock 0 0K 0 idmap 0 0K 0 NFS4 dev 0 0K 0 syncache 1 8K 1 USB 31 3K 31 hostcache 1 24K 1 ip_moptions 0 0K 0 Export Host 0 0K 0 in_multi 3 1K 3 igmp 0 0K 0 routetbl 20 2K 93 entropy 1024 64K 1024 ad_driver 3 1K 3 ata_generic 4 4K 4 vlan 2 1K 2 tun 0 0K 0 lo 1 1K 1 arpcom 3 1K 3 clone 3 12K 3 ether_multi 12 1K 14 ifaddr 30 7K 30 ifnet 6 6K 6 BPF 9 65K 10 acd_driver 1 2K 1 Export Host 0 0K 0 vnodemarker 0 0K 2420 mount 223 6K 314 vnodes 1 1K 1 VFS hash 1 256K 1 Export Host 0 0K 0 cluster_save buffer 0 0K 6 vfscache 1 512K 1 BIO buffer 224 448K 235 isadev 19 2K 19 pcb 21 5K 703 soname 13 1K 3568 mbuf_tag 0 0K 2 mbextcnt 0 0K 0 accf 0 0K 0 ptys 0 0K 0 ttys 1177 165K 2648 shm 1 12K 1 sem 4 7K 4 msg 4 25K 4 iov 0 0K 828 select 0 0K 0 ioctlops 0 0K 2054 Unitno 6 1K 178 turnstiles 151 10K 151 taskqueue 9 1K 9 stack 0 0K 0 MD sectors 0 0K 0 MD disk 0 0K 0 sleep queues 151 5K 151 sbuf 0 0K 906 rman 176 11K 544 LED 0 0K 0 kbdmux 6 9K 6 ata_pci 0 0K 0 kobj 115 230K 134 eventhandler 55 4K 55 devstat 14 29K 14 mirror_data 3 1K 625 bus-sc 82 32K 1841 bus 799 38K 4368 SWAP 2 549K 2 umtx 150 10K 150 sysctltmp 0 0K 333 sysctloid 3214 97K 3214 sysctl 0 0K 634 uidinfo 6 2K 141 plimit 25 7K 1334 ata_dma 6 1K 6 cred 50 7K 12035 subproc 235 464K 2552 proc 2 8K 2 session 46 6K 223 pgrp 46 3K 223 GEOM 156 17K 832 mtx_pool 1 8K 1 module 183 12K 183 pfs_vncache 0 0K 0 free 0 0K 0 pfs_nodes 0 0K 0 ip6ndp 0 0K 0 ip6opt 0 0K 0 temp 13 211K 6822 devbuf 1562 3592K 1564 cache 0 0K 0 lockf 21 2K 6581 DEVFS 62 2K 63 linker 50 243K 105 DEVFS_RULE 0 0K 0 DEVFS1 100 25K 102 db> call doadump -- Stefan Bethke <stb@lassitu.de> Fon +49 170 346 0140
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7E5496DA-06AC-4347-A807-DFB43D7E3571>