From owner-freebsd-bugs@FreeBSD.ORG Sun Dec 26 11:20:18 2004 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 17E7C16A4CE for ; Sun, 26 Dec 2004 11:20:18 +0000 (GMT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B32243D3F for ; Sun, 26 Dec 2004 11:20:17 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.1/8.13.1) with ESMTP id iBQBKHFZ032873 for ; Sun, 26 Dec 2004 11:20:17 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.1/8.13.1/Submit) id iBQBKHeX032872; Sun, 26 Dec 2004 11:20:17 GMT (envelope-from gnats) Resent-Date: Sun, 26 Dec 2004 11:20:17 GMT Resent-Message-Id: <200412261120.iBQBKHeX032872@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Rob Swindell Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 93C7216A4CE for ; Sun, 26 Dec 2004 11:11:04 +0000 (GMT) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3872643D1D for ; Sun, 26 Dec 2004 11:11:04 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id iBQBB3Nj003166 for ; Sun, 26 Dec 2004 11:11:03 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id iBQBB30H003165; Sun, 26 Dec 2004 11:11:03 GMT (envelope-from nobody) Message-Id: <200412261111.iBQBB30H003165@www.freebsd.org> Date: Sun, 26 Dec 2004 11:11:03 GMT From: Rob Swindell To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Subject: misc/75510: panic: kmem_malloc(4096): kmem_map too small X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Dec 2004 11:20:18 -0000 >Number: 75510 >Category: misc >Synopsis: panic: kmem_malloc(4096): kmem_map too small >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Dec 26 11:20:17 GMT 2004 >Closed-Date: >Last-Modified: >Originator: Rob Swindell >Release: 5.3-RELEASE >Organization: Synchronet >Environment: FreeBSD devil.synchro.net 5.3-RELEASE FreeBSD 5.3-RELEASE #0: Sat Dec 25 02:42:54 PST 2004 root@devil.synchro.net:/usr/src/sys/i386/compile/MYKERNEL i386 >Description: When running a multi-threaded server application (http://www.synchro.net), accessing configuration and data files on an smbfs-mounted volume, after 4-8 hours, I get the following panic (routinely): panic: kmem_malloc(4096): kmem_map too small: 40898560 total allocated I've seen another PR related to a panic like this one, but it was in relation to a system with 4GB+ of RAM. This particular system only has 128MB of RAM. Here is a current "top" output at the time of one of the panics: last pid: 6429; load averages: 0.09, 0.07, 0.21 up 0+07:27:31 05:52:23 84 processes: 2 running, 57 sleeping, 25 waiting CPU states: 0.4% user, 0.0% nice, 3.5% system, 2.3% interrupt, 93.8% idle Mem: 31M Active, 2652K Inact, 47M Wired, 1556K Cache, 22M Buf, 35M Free Swap: 231M Total, 88K Used, 231M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 3158 root 20 0 19928K 17596K kserel 22:14 0.00% 0.00% sbbs 583 root 96 0 6092K 2172K select 0:01 0.00% 0.00% sshd 1298 root 96 0 6092K 2168K select 0:01 0.00% 0.00% sshd 529 root 96 0 6092K 2152K select 0:01 0.00% 0.00% sshd 476 root 96 0 6092K 1856K select 0:14 0.00% 0.00% sshd 379 root 96 0 3360K 1560K select 0:00 0.00% 0.00% sshd 528 root 96 0 2296K 1308K RUN 1:23 0.00% 0.00% top 1314 root 8 0 2208K 1616K wait 0:00 0.00% 0.00% bash 586 root 5 0 2204K 1620K ttyin 0:00 0.00% 0.00% bash 532 root 8 0 2204K 1580K wait 0:00 0.00% 0.00% bash 479 root 8 0 2204K 1524K wait 0:00 0.00% 0.00% bash 218 root 96 0 1784K 1048K select 0:01 0.00% 0.00% dhclient 392 root 8 0 1356K 860K nanslp 0:00 0.00% 0.00% cron 268 root 96 0 1312K 760K select 0:08 0.00% 0.00% syslogd 465 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 471 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 470 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 468 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 467 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 469 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 466 root 5 0 1280K 772K ttyin 0:00 0.00% 0.00% getty 1611 root 5 0 1280K 764K ttyin 0:00 0.00% 0.00% getty 343 root 96 0 1236K 656K select 0:00 0.00% 0.00% usbd 428 root 96 0 1232K 656K select 0:00 0.00% 0.00% moused 3500 root 4 0 1216K 580K kqread 0:00 0.00% 0.00% tail 167 root 20 0 1184K 572K pause 0:00 0.00% 0.00% adjkerntz 1 root 8 0 740K 220K wait 0:00 0.00% 0.00% init 248 root 111 0 516K 224K select 0:00 0.00% 0.00% devd 11 root 112 0 0K 12K RUN 340:17 89.36% 89.36% idle 407 root 8 0 0K 12K 90idle 4:01 0.93% 0.93% smbiod0 27 root -28 -147 0K 12K WAIT 14:22 0.15% 0.15% swi5: clock s 29 root -44 -163 0K 12K WAIT 2:41 0.00% 0.00% swi1: net 21 root -68 -187 0K 12K WAIT 2:40 0.00% 0.00% irq10: de0 uh 30 root 76 0 0K 12K - 0:38 0.00% 0.00% yarrow 36 root -24 -143 0K 12K WAIT 0:14 0.00% 0.00% swi6:+ 40 root 171 52 0K 12K pgzero 0:09 0.00% 0.00% pagezero 42 root 20 0 0K 12K syncer 0:07 0.00% 0.00% syncer 4 root -8 0 0K 12K - 0:07 0.00% 0.00% g_down Here's the top output at the time of another such panic: last pid: 2904; load averages: 0.13, 0.11, 0.08 up 0+01:35:27 14:39:41 75 processes: 2 running, 48 sleeping, 25 waiting CPU states: 0.4% user, 0.0% nice, 5.4% system, 2.3% interrupt, 91.8% idle Mem: 22M Active, 7184K Inact, 47M Wired, 1188K Cache, 22M Buf, 40M Free Swap: 231M Total, 231M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 474 root 20 0 11908K 9956K kserel 4:10 0.00% 0.00% sbbs 482 root 96 0 6092K 2240K select 0:03 0.00% 0.00% sshd 470 root 96 0 6092K 2236K select 0:00 0.00% 0.00% sshd 380 root 96 0 3360K 1936K select 0:00 0.00% 0.00% sshd 487 root 8 0 2208K 1576K wait 0:00 0.00% 0.00% bash 473 root 8 0 2208K 1568K wait 0:00 0.00% 0.00% bash 488 root 96 0 2280K 1316K RUN 0:17 0.00% 0.00% top 216 root 96 0 1784K 1064K select 0:00 0.00% 0.00% dhclient 393 root 8 0 1356K 916K nanslp 0:00 0.00% 0.00% cron 462 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 463 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 469 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 466 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 465 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 468 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 467 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 464 root 5 0 1280K 808K ttyin 0:00 0.00% 0.00% getty 266 root 96 0 1312K 800K select 0:02 0.00% 0.00% syslogd 342 root 96 0 1236K 688K select 0:00 0.00% 0.00% usbd 429 root 118 0 1232K 664K select 0:00 0.00% 0.00% moused 165 root 20 0 1184K 580K pause 0:00 0.00% 0.00% adjkerntz 1 root 8 0 740K 244K wait 0:00 0.00% 0.00% init 246 root 111 0 516K 224K select 0:00 0.00% 0.00% devd 11 root 104 0 0K 12K RUN 59:07 90.23% 90.23% idle 408 root 8 0 0K 12K 90idle 1:11 0.29% 0.29% smbiod0 27 root -44 -163 0K 12K WAIT 0:42 0.10% 0.10% swi1: net 28 root -28 -147 0K 12K WAIT 1:45 0.00% 0.00% swi5: clock sio 21 root -68 -187 0K 12K WAIT 1:19 0.00% 0.00% irq10: de0 uhci0 30 root 76 0 0K 12K - 0:09 0.00% 0.00% yarrow 40 root 171 52 0K 12K pgzero 0:04 0.00% 0.00% pagezero 32 root -24 -143 0K 12K WAIT 0:04 0.00% 0.00% swi6:+ 43 root 20 0 0K 12K syncer 0:01 0.00% 0.00% syncer 3 root -8 0 0K 12K - 0:01 0.00% 0.00% g_up 4 root -8 0 0K 12K - 0:01 0.00% 0.00% g_down 45 root -8 0 0K 12K - 0:01 0.00% 0.00% schedcpu 2 root -8 0 0K 12K - 0:01 0.00% 0.00% g_event 25 root -64 -183 0K 12K WAIT 0:01 0.00% 0.00% irq14: ata0 410 root 8 0 0K 12K 90idle 0:00 0.00% 0.00% smbiod1 41 root -16 0 0K 12K psleep 0:00 0.00% 0.00% bufdaemon 42 root -4 0 0K 12K vlruwt 0:00 0.00% 0.00% vnlru 7 root -8 0 0K 12K - 0:00 0.00% 0.00% fdc0 Here is the mbuf allocation status not too long before a panic: [devil:~]# netstat -m 108 mbufs in use 34/4800 mbuf clusters in use (current/max) 0/3/1456 sfbufs in use (current/peak/max) 95 KBytes allocated to network 0 requests for sfbufs denied 0 requests for sfbufs delayed 2 requests for I/O initiated by sendfile 57 calls to protocol drain routines Here's a /var/crash/info file for a recent panic: Architecture: i386 Architecture version: 1 Dump length: 134205440B (127 MB) Blocksize: 512 Dumptime: Sat Dec 25 14:39:42 2004 Hostname: devil.synchro.net Versionstring: FreeBSD 5.3-RELEASE #0: Sat Dec 25 02:42:54 PST 2004 root@devil.synchro.net:/usr/src/sys/i386/compile/MYKERNEL Panicstring: kmem_malloc(4096): kmem_map too small: 40898560 total allocated Bounds: 0 Here's a kgdb backtrace from that same panic: (kgdb) bt #0 doadump () at pcpu.h:159 #1 0xc05fdc49 in boot (howto=260) at ../../../kern/kern_shutdown.c:397 #2 0xc05fdf05 in panic ( fmt=0xc07c38d3 "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at ../../../kern/kern_shutdown.c:553 #3 0xc06fc8d9 in kmem_malloc (map=0xc103a0c0, size=4096, flags=258) at ../../../vm/vm_kern.c:300 #4 0xc070b7c2 in page_alloc (zone=0xc1044d80, bytes=4096, pflag=0x0, wait=258) at ../../../vm/uma_core.c:935 #5 0xc070b301 in slab_zalloc (zone=0xc1044d80, wait=258) at ../../../vm/uma_core.c:805 #6 0xc070ca54 in uma_zone_slab (zone=0xc1044d80, flags=2) at ../../../vm/uma_core.c:1962 #7 0xc070cc74 in uma_zalloc_bucket (zone=0xc1044d80, flags=2) at ../../../vm/uma_core.c:2071 #8 0xc070c900 in uma_zalloc_arg (zone=0xc1044d80, udata=0x0, flags=2) at ../../../vm/uma_core.c:1889 #9 0xc06455ba in cache_enter (dvp=0xc17afb58, vp=0xc17dc318, cnp=0xcc83fa48) at uma.h:274 #10 0xc1575310 in ?? () #11 0xc17afb58 in ?? () #12 0xc17dc318 in ?? () #13 0xcc83fa48 in ?? () I don't know for sure that this is an smbfs-related problem, but I have my suspicions. I don't know if it's any clue, but for every file close(), I get the following error: "smb_maperror: Unmapped error 1:158". Merry Christmas! >How-To-Repeat: Run Synchronet (http://www.synchro.net) (or probably any other multi-threaded server) with the configuration and data files stored in an smbfs-mounted directory. >Fix: >Release-Note: >Audit-Trail: >Unformatted: