From owner-freebsd-bugs@FreeBSD.ORG Tue Mar 4 09:50:03 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C26C1065677 for ; Tue, 4 Mar 2008 09:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 755D78FC38 for ; Tue, 4 Mar 2008 09:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m249o3kM038988 for ; Tue, 4 Mar 2008 09:50:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m249o3jh038987; Tue, 4 Mar 2008 09:50:03 GMT (envelope-from gnats) Resent-Date: Tue, 4 Mar 2008 09:50:03 GMT Resent-Message-Id: <200803040950.m249o3jh038987@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, tadokoro Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 051851065673 for ; Tue, 4 Mar 2008 09:44:26 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 00FBA8FC1F for ; Tue, 4 Mar 2008 09:44:26 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m249fQ9g073146 for ; Tue, 4 Mar 2008 09:41:26 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.2/8.14.1/Submit) id m249fQVJ073145; Tue, 4 Mar 2008 09:41:26 GMT (envelope-from nobody) Message-Id: <200803040941.m249fQVJ073145@www.freebsd.org> Date: Tue, 4 Mar 2008 09:41:26 GMT From: tadokoro To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/121350: panic: initiate_write_inodeblock_ufs2: already started X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Mar 2008 09:50:03 -0000 >Number: 121350 >Category: kern >Synopsis: panic: initiate_write_inodeblock_ufs2: already started >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Mar 04 09:50:03 UTC 2008 >Closed-Date: >Last-Modified: >Originator: tadokoro >Release: 6.2-RELEASE-p11 >Organization: tokyo institute of technology >Environment: FreeBSD yeti.local 6.2-RELEASE-p11 FreeBSD 6.2-RELEASE-p11 #17: Fri Feb 15 21:29:36 JST 2008 root@yeti.local:/usr/obj/usr/src/sys/YETI amd64 I'm running software RAID-1 with atacontrol. % grep ata /var/run/dmesg.boot atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe800-0xe80f at device 6.0 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xd400-0xd40f mem 0xfebfc000-0xfebfcfff irq 23 at device 7.0 on pci0 ata2: on atapci1 ata3: on atapci1 atapci2: port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xc000-0xc00f mem 0xfebfb000-0xfebfbfff irq 21 at device 8.0 on pci0 ata4: on atapci2 ata5: on atapci2 acd0: DVDR at ata0-master UDMA66 ad4: 238475MB at ata2-master SATA150 ad6: 238475MB at ata3-master SATA150 ar0: disk0 READY (master) using ad4 at ata2-master ar0: disk1 READY (mirror) using ad6 at ata3-master >Description: The machine got panic during "portupgrade -rf gnutls". The machine was maybe under heavy I/O because of portupgrade. But I did nothing other. /var/log/messages before and after the panic is below. Mar 4 10:50:39 yeti nmbd[11859]: ***** Mar 4 15:06:08 yeti kernel: DOH! ata_alloc_request failed! Mar 4 15:06:08 yeti kernel: FAILURE - out of memory in ata_raid_init_request Mar 4 15:06:08 yeti last message repeated 8 times Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632301056, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632317440, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632333824, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632350208, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632366592, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632382976, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632415744, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632432128, length=16384)]error = 5 Mar 4 15:06:08 yeti kernel: g_vfs_done():ar0s1f[WRITE(offset=9632448512, length=16384)]error = 5 Mar 4 15:09:02 yeti syslogd: kernel boot file is /boot/kernel/kernel Mar 4 15:09:02 yeti kernel: Copyright (c) 1992-2007 The FreeBSD Project. Mar 4 15:09:02 yeti kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Mar 4 15:09:02 yeti kernel: The Regents of the University of California. All rights reserved. Mar 4 15:09:02 yeti kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Mar 4 15:09:02 yeti kernel: FreeBSD 6.2-RELEASE-p11 #17: Fri Feb 15 21:29:36 JST 2008 Mar 4 15:09:02 yeti kernel: root@yeti.local:/usr/obj/usr/src/sys/YETI A kernel crash dump with kgdb is below. yeti# cd /usr/obj/usr/src/sys/YETI yeti# kgdb kernel.debug /var/crash/vmcore.1 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: panic: initiate_write_inodeblock_ufs2: already started cpuid = 0 Uptime: 16d18h58m18s Dumping 2046 MB (2 chunks) chunk 0: 1MB (148 pages) ... ok chunk 1: 2046MB (523744 pages) 2030 2014 1998 1982 1966 1950 1934 1918 1902 1886 1870 1854 1838 1822 1806 1790 1774 1758 1742 1726 1710 1694 1678 1662 1646 1630 1614 1598 1582 1566 1550 1534 1518 1502 1486 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:172 172 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) bt full #0 doadump () at pcpu.h:172 No locals. #1 0x0000000000000004 in ?? () No symbol table info available. #2 0xffffffff803f90b7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 first_buf_printf = 1 #3 0xffffffff803f9751 in panic (fmt=0xffffff007b838980 "°\026Üy") at /usr/src/sys/kern/kern_shutdown.c:565 bootopt = 260 newpanic = 0 ap = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 0xffffffffb1d4ea20, reg_save_area = 0xffffffffb1d4e950}} buf = "initiate_write_inodeblock_ufs2: already started", '\0' #4 0xffffffff80577bf9 in softdep_disk_io_initiation (bp=0xffffffff9fd77fe0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3763 wk = (struct worklist *) 0xffffff000b5f5400 marker = {wk_mp = 0xffffff007fec2238, wk_list = {le_next = 0xffffff000f7b9600, le_prev = 0xffffff000b5f5408}, wk_type = 14, wk_state = 0} #5 0xffffffff80584256 in ffs_geom_strategy (bo=0xffffff0061c4bb00, bp=0xffffffff9fd77fe0) at buf.h:433 vp = (struct vnode *) 0xffffff0061c4b9b0 error = 0 tbp = (struct buf *) 0xa0020024 #6 0xffffffff804546b0 in bufwrite (bp=0xffffffff9fd77fe0) at buf.h:426 No locals. #7 0xffffffff8044f6cd in vfs_bio_awrite (bp=0xffffffff9fd77fe0) at buf.h:410 i = -1 j = -2138334496 lblkno = -1097871344896 vp = (struct vnode *) 0x4000 ncl = 1640282544 size = 0 #8 0xffffffff80458d58 in vop_stdfsync (ap=0xffffffffb1d4ebc0) at /usr/src/sys/kern/vfs_default.c:428 vp = (struct vnode *) 0xffffff0061c4b9b0 bp = (struct buf *) 0xffffffff9fd77fe0 bo = (struct bufobj *) 0xffffffff9fd77fe0 nbp = (struct buf *) 0xffffffffa0133688 error = 0 maxretry = 1000 #9 0xffffffff806702cd in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020 rc = 0 #10 0xffffffff80463651 in sched_sync () at vnode_if.h:537 next = (struct synclist *) 0xffffff00796bf5a0 slp = (struct synclist *) 0xffffff00796bf598 bo = (struct bufobj *) 0xffffff0061c4bb00 starttime = 1204610797 td = (struct thread *) 0xffffff007b838980 dummychan = 0 last_work_seen = 20 net_worklist_len = 1640282880 syncer_final_iter = 0 first_printf = 1 #11 0xffffffff803de087 in fork_exit (callout=0xffffffff80463250 , arg=0x0, frame=0xffffffffb1d4ec50) at /usr/src/sys/kern/kern_fork.c:821 p = (struct proc *) 0xffffff0079dc16b0 #12 0xffffffff8060530e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:394 No locals. >How-To-Repeat: Sorry, I don't know. >Fix: >Release-Note: >Audit-Trail: >Unformatted: