From owner-freebsd-bugs@FreeBSD.ORG Sat Apr 3 07:47:08 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 956991065670 for ; Sat, 3 Apr 2010 07:47:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 594048FC15 for ; Sat, 3 Apr 2010 07:47:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o337l8Aa097876 for ; Sat, 3 Apr 2010 07:47:08 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o337l8ak097875; Sat, 3 Apr 2010 07:47:08 GMT (envelope-from gnats) Resent-Date: Sat, 3 Apr 2010 07:47:08 GMT Resent-Message-Id: <201004030747.o337l8ak097875@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Alex Bakhtin Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEE0C106564A for ; Sat, 3 Apr 2010 05:08:41 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 9D29E8FC14 for ; Sat, 3 Apr 2010 05:08:41 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o3358fF0097038 for ; Sat, 3 Apr 2010 05:08:41 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o3358fhm097037; Sat, 3 Apr 2010 05:08:41 GMT (envelope-from nobody) Message-Id: <201004030508.o3358fhm097037@www.freebsd.org> Date: Sat, 3 Apr 2010 05:08:41 GMT From: Alex Bakhtin To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/145339: [zfs] deadlock after detaching block device from raidz pool X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Apr 2010 07:47:08 -0000 >Number: 145339 >Category: kern >Synopsis: [zfs] deadlock after detaching block device from raidz pool >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Apr 03 07:47:07 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Alex Bakhtin >Release: 8.0-STABLE >Organization: >Environment: FreeBSD tarzan-new.private.flydrag.ru 8.0-STABLE FreeBSD 8.0-STABLE #1: Sat Apr 3 04:54:06 UTC 2010 bakhtin@tarzan-new.private.flydrag.ru:/mnt/obj/usr/src.old/sys/DEBUG amd64 >Description: Detaching (physically) block device when there is intensive writing to the pool causes deadlock. Tested on 8.0-STABLE/amd64 csuped at 02 Apr 2010. gmirror on the same system handles device detach properly. Detaching device when zfs is idle doesn't cause any problem. Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x48 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff805815f9 stack pointer = 0x28:0xffffff8000065b80 frame pointer = 0x28:0xffffff8000065bb0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3 (g_up) exclusive spin mutex uart_hwmtx (uart_hwmtx) r = 0 (0xffffff0002a62838) locked @ /usr/src.old/sys/dev/uart/uart_cpu.h:92 exclusive lockmgr zfs (zfs) r = 0 (0xffffff0123079098) locked @ /usr/src.old/sys/kern/vfs_vnops.c:607 exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff000c3728f0) locked @ /usr/src.old/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff000c35a8f0) locked @ /usr/src.old/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0121a31648) locked @ /usr/src.old/sys/kern/uipc_sockbuf.c:148 0xffffff0123079000: tag zfs, type VREG usecount 1, writecount 1, refcount 1 mountedhere 0 flags () v_object 0xffffff0126114e58 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff000c2d7740 (pid 2134) #0 0xffffffff80579d27 at __lockmgr_args+0x777 #1 0xffffffff80613339 at vop_stdlock+0x39 #2 0xffffffff808d020b at VOP_LOCK1_APV+0x9b #3 0xffffffff806300d7 at _vn_lock+0x57 #4 0xffffffff806316d8 at vn_write+0x218 #5 0xffffffff805d71e5 at dofilewrite+0x85 #6 0xffffffff805d89e0 at kern_writev+0x60 #7 0xffffffff805d8ae5 at write+0x55 #8 0xffffffff8087b488 at syscall+0x118 #9 0xffffffff80861611 at Xfast_syscall+0xe1 db:0:kdb.enter.default> bt Tracing pid 3 tid 100010 td 0xffffff0002899740 _mtx_lock_flags() at _mtx_lock_flags+0x39 vdev_geom_io_intr() at vdev_geom_io_intr+0x62 g_io_schedule_up() at g_io_schedule_up+0xed g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000065d30, rbp = 0 --- tarzan-new# zdb -vvv storage version=14 name='storage' state=0 txg=578 pool_guid=3309800284037274155 hostid=4266611921 hostname='tarzan-new.private.flydrag.ru' vdev_tree type='root' id=0 guid=3309800284037274155 children[0] type='raidz' id=0 guid=11076638880661644944 nparity=1 metaslab_array=23 metaslab_shift=36 ashift=9 asize=10001970626560 is_log=0 children[0] type='disk' id=0 guid=134064330288565023 path='/dev/ad10' whole_disk=0 DTL=33 children[1] type='disk' id=1 guid=6567589632071309972 path='/dev/ad12' whole_disk=0 DTL=32 children[2] type='disk' id=2 guid=6024702546194706986 path='/dev/ad14' whole_disk=0 DTL=27 children[3] type='disk' id=3 guid=10837092740689261565 path='/dev/ad16' whole_disk=0 DTL=31 children[4] type='disk' id=4 guid=4165337351109841378 path='/dev/ad18' whole_disk=0 DTL=30 tarzan-new# Core: http://flydrag.dyndns.org:9090/freebsd/zfs-deadlock/core.txt.9 >How-To-Repeat: Install 8.0-STABLE, create raidz pool, run command dd if=/dev/zero of=/zfs/test bs=10m detach (physically) one hard disk when it is writing. >Fix: >Release-Note: >Audit-Trail: >Unformatted: