From owner-freebsd-current@FreeBSD.ORG Fri Jun 11 15:44:20 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0771916A4CE for ; Fri, 11 Jun 2004 15:44:20 +0000 (GMT) Received: from main.gmane.org (main.gmane.org [80.91.224.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2B1D043D45 for ; Fri, 11 Jun 2004 15:44:19 +0000 (GMT) (envelope-from freebsd-current@m.gmane.org) Received: from list by main.gmane.org with local (Exim 3.35 #1 (Debian)) id 1BYoCT-0001ZI-00 for ; Fri, 11 Jun 2004 17:44:14 +0200 Received: from mulder.f5.com ([205.229.151.150]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 Jun 2004 17:44:13 +0200 Received: from atkin901 by mulder.f5.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 Jun 2004 17:44:13 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-current@freebsd.org From: othermark Date: Fri, 11 Jun 2004 08:44:10 -0700 Lines: 120 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7Bit X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: mulder.f5.com User-Agent: KNode/0.7.6 Sender: news Subject: Re: Today's -current panics X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jun 2004 15:44:20 -0000 Robert Watson wrote: > On Fri, 11 Jun 2004, othermark wrote: >> I get a very similar stack track traversing through sossend(), under >> heavy NFS load on a 1GB machine. Note the panic message here, and the >> peculiarity that previous incarnations of -current did not panic under >> similar load. It is highly reproduceable via a 'make installworld' via >> NFS with /usr/src and /usr/obj mounted. The NFS serving machine will >> always panic using vanilla GENERIC: > > Hrmm. It's certainly similar, but it's not clear to me that it's > necessarily related (although I wouldn't preclude that). Could you tell > me what date you cvsup'd this tree? Jun 10 07:35 PST > Also, since you have a dump (yay!), could you try running vmstat -m, > vmstat -z, and netstat -mb against the dump? It could be that a bug was > introduced in nfsd that leaks mbufs or the like, or it could be a nit in > mbuma. [root@pippin GENERIC]$ vmstat -z -M /var/crash/vmcore.3 vmstat: not implemented DOH! (same goes for -m) [root@pippin GENERIC]$ netstat -mb -M /var/crash/vmcore.3 netstat: calloc: cannot allocate memory for mbtypes seen flag: Cannot allocate memory DOH! I'm very willing to help further so I can continue to do nfs installs of -current for my test harness. The NFS bits panic looks like this (same panic message): (kgdb) symbol-file kernel.debug Reading symbols from kernel.debug...done. (kgdb) exec-file kernel (kgdb) core-file /var/crash/vmcore.4 panic: kmem_malloc(4096): kmem_map too small: 40894464 total allocated panic messages: --- panic: kmem_malloc(4096): kmem_map too small: 40894464 total allocated cpuid = 0; Debugger("panic") panic: from debugger cpuid = 0; Uptime: 7m9s Dumping 127 MB 16 32 48 64 80 96 112 --- Reading symbols from /boot/kernel/acpi.ko...done. Loaded symbols for /boot/kernel/acpi.ko #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:236 236 dumping++; (kgdb) bt #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:236 #1 0xc064ffd9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:370 #2 0xc06503dd in panic () at /usr/src/sys/kern/kern_shutdown.c:548 #3 0xc0469902 in db_panic () at /usr/src/sys/ddb/db_command.c:453 #4 0xc0469862 in db_command (last_cmdp=0xc09250a0, cmd_table=0x0, aux_cmd_tablep=0xc08a6f00, aux_cmd_tablep_end=0xc08a6f18) at /usr/src/sys/ddb/db_command.c:348 #5 0xc04699a5 in db_command_loop () at /usr/src/sys/ddb/db_command.c:475 #6 0xc046cb05 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:73 #7 0xc080132c in kdb_trap (type=3, code=0, regs=0xc80f28cc) at /usr/src/sys/i386/i386/db_interface.c:159 #8 0xc0817948 in trap (frame= {tf_fs = -1066991592, tf_es = -1064763376, tf_ds = -1064828912, tf_edi = - 1064721812, tf_esi = 1, tf_ebp = -938530536, tf_isp = -938530568, tf_ebx = 0, tf _edx = 0, tf_ecx = -1056882688, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1065347531, tf_cs = 8, tf_eflags = 642, tf_esp = -1064702569, tf_ss = -1064817 463}) at /usr/src/sys/i386/i386/trap.c:579 #9 0xc0801635 in Debugger (msg=0x0) at machine/cpufunc.h:56 #10 0xc0650376 in panic ( fmt=0xc089a26c "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at /usr/src/sys/kern/kern_shutdown.c:532 #11 0xc07c43cb in kmem_malloc (map=0xc103a0a0, size=4096, flags=258) at /usr/src/sys/vm/vm_kern.c:324 #12 0xc07d59a7 in page_alloc (zone=0xc1021b00, bytes=0, pflag=0x0, wait=0) at /usr/src/sys/vm/uma_core.c:892 #13 0xc07d562d in slab_zalloc (zone=0xc1021b00, wait=258) at /usr/src/sys/vm/uma_core.c:789 #14 0xc07d6d4a in uma_zone_slab (zone=0xc1021b00, flags=2) at /usr/src/sys/vm/uma_core.c:1772 #15 0xc07d70a1 in uma_zalloc_internal (zone=0xc1021b00, udata=0x0, flags=2) at /usr/src/sys/vm/uma_core.c:1936 #16 0xc07d6c52 in uma_zalloc_arg (zone=0xc1021b00, udata=0xc80f2b14, flags=2) at /usr/src/sys/vm/uma_core.c:1711 #17 0xc0741728 in nfsrv_read (nfsd=0xc1704000, slp=0x0, td=0xc137d2c0, mrq=0xc80f2c90) at /usr/src/sys/sys/mbuf.h:334 #18 0xc075203a in nfssvc_nfsd (td=0x0) at /usr/src/sys/nfsserver/nfs_syscalls.c:465 #19 0xc07518bd in nfssvc (td=0xc137d2c0, uap=0xc80f2d14) at /usr/src/sys/nfsserver/nfs_syscalls.c:181 ---Type to continue, or q to quit--- #20 0xc08182f0 in syscall (frame= {tf_fs = -1078001617, tf_es = 47, tf_ds = -1078001617, tf_edi = -107794054 0, tf_esi = 4, tf_ebp = -1077941432, tf_isp = -938529420, tf_ebx = 0, tf_edx = 6 72430424, tf_ecx = 25, tf_eax = 155, tf_trapno = 12, tf_err = 2, tf_eip = 671893 695, tf_cs = 31, tf_eflags = 646, tf_esp = -1077941460, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1004 #21 0x280c48bf in ?? () ---Can't read userspace from dump, or kernel process--- -- othermark atkin901 at nospam dot yahoo dot com (!wired)?(coffee++):(wired);