From owner-freebsd-stable@FreeBSD.ORG Sun Nov 22 07:31:35 2009 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89889106566B for ; Sun, 22 Nov 2009 07:31:35 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 399548FC0A for ; Sun, 22 Nov 2009 07:31:34 +0000 (UTC) Received: (qmail 21531 invoked by uid 0); 22 Nov 2009 07:04:54 -0000 Received: from unknown (HELO ?10.3.2.41?) (spork@96.57.144.66) by smtp.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 22 Nov 2009 07:04:54 -0000 Date: Sun, 22 Nov 2009 02:04:53 -0500 (EST) From: Charles Sprickman X-X-Sender: spork@hotlap.local To: stable@freebsd.org Message-ID: User-Agent: Alpine 2.00 (OSX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: Subject: panic in 7.2 (ffs_alloc.c?) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2009 07:31:35 -0000 Howdy, I'm not expert at getting info out of a dump, but I'll do my best to provide some information. This is a Dell PE2970 w/PERC6/i RAID running FreeBSD 7.2/amd64. Brand new box, has been doing very light work for about two weeks. Last night I started a very long mstone run on a jailed mail server and found that quite a way into this burn-in, the box paniced. I was going to put it in service Monday (after punishing it all weekend). Looking for some input on what the root cause is and whether going to a -stable snapshot might be worthwhile. I can tell you there was a good deal of disk activity at the time in the jail - mstone was simulating 100 POP and SMTP clients hitting the machine at once. This is qmail+courier. So messages are coming in, hitting the queue, hitting a user's maildir, getting read and deleted via the POP "client" over and over again. I do see lots of "ffs_*" stuff in the backtrace, which is a little scary. Here's my stab at a kgdb session (also @ pastie for easier reading: http://pastie.org/709671): [root@bigmail /usr/obj/usr/src/sys/BWAY7-64]# kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x12d4b9f5c fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff8050382e stack pointer = 0x10:0xffffffff281a75b0 frame pointer = 0x10:0xffffff000455f800 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 6324 (vdelivermail) trap number = 12 panic: page fault cpuid = 0 Uptime: 12d0h32m3s Physical memory: 6130 MB Dumping 725 MB: 710 694 678 662 646 630 614 598 582 566 550 534 518 502 486 470 454 438 422 406 390 374 358 342 326 310 294 278 262 246 230 214 198 182 166 150 134 118 102 86 70 54 38 22 6 Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/nullfs.ko Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/fdescfs.ko #0 doadump () at pcpu.h:195 195 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); #3 0xffffffff8034cba2 in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:574 #4 0xffffffff80574823 in trap_fatal (frame=0xffffff00046c8000, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:757 #5 0xffffffff80574bf5 in trap_pfault (frame=0xffffffff281a7500, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:673 #6 0xffffffff80575534 in trap (frame=0xffffffff281a7500) at /usr/src/sys/amd64/amd64/trap.c:444 #7 0xffffffff8055969e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:209 #8 0xffffffff8050382e in ffs_realloccg (ip=0xffffff00267f75c0, lbprev=0, bprev=6288224785898156086, bpref=593305256, osize=0, nsize=2048, flags=33619968, cred=0xffffff00927fe800, bpp=0xffffffff281a7800) at /usr/src/sys/ufs/ffs/ffs_alloc.c:1349 #9 0xffffffff80506e8e in ffs_balloc_ufs2 (vp=0xffffff0027a64dc8, startoffset=Variable "startoffset" is not available. ) at /usr/src/sys/ufs/ffs/ffs_balloc.c:692 #10 0xffffffff805223e5 in ffs_write (ap=0xffffffff281a7a10) at /usr/src/sys/ufs/ffs/ffs_vnops.c:724 #11 0xffffffff805a0645 in VOP_WRITE_APV (vop=0xffffffff80793d20, a=0xffffffff281a7a10) at vnode_if.c:691 #12 0xffffffff803dd731 in vn_write (fp=0xffffff001027cd00, uio=0xffffffff281a7b00, active_cred=Variable "active_cred" is not available. ) at vnode_if.h:373 #13 0xffffffff80388768 in dofilewrite (td=0xffffff00046c8000, fd=5, fp=0xffffff001027cd00, auio=dwarf2_read_address: Corrupted DWARF expression. ) at file.h:257 #14 0xffffffff80388a6e in kern_writev (td=0xffffff00046c8000, fd=5, auio=0xffffffff281a7b00) at /usr/src/sys/kern/sys_generic.c:402 #15 0xffffffff80388aec in write (td=0x800, uap=0x12d4b9f50) at /usr/src/sys/kern/sys_generic.c:318 #16 0xffffffff80596a66 in ia32_syscall (frame=0xffffffff281a7c80) at /usr/src/sys/amd64/ia32/ia32_syscall.c:182 #17 0xffffffff80559ad0 in Xint0x80_syscall () at ia32_exception.S:65 #18 0x0000000028167928 in ?? () Previous frame inner to this frame (corrupt stack?) Full dmesg, verbose boot and kernel config at pastie as well. Actually no verbose boot... I rebooted the box after setting verbose boot with "nextboot" and it didn't come back. Hrmph. No remote console, so I don't know what's up, perhaps waiting on some manual fsck action. Thanks, Charles