From owner-freebsd-bugs@FreeBSD.ORG Wed Sep 8 16:10:03 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 291AB10656EF for ; Wed, 8 Sep 2010 16:10:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E0A7B8FC12 for ; Wed, 8 Sep 2010 16:10:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o88GA2UQ070901 for ; Wed, 8 Sep 2010 16:10:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o88GA2NO070900; Wed, 8 Sep 2010 16:10:02 GMT (envelope-from gnats) Resent-Date: Wed, 8 Sep 2010 16:10:02 GMT Resent-Message-Id: <201009081610.o88GA2NO070900@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Rich Ercolani Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22D2910656BE for ; Wed, 8 Sep 2010 16:07:46 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 125448FC08 for ; Wed, 8 Sep 2010 16:07:46 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o88G7iW4062318 for ; Wed, 8 Sep 2010 16:07:44 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o88G7i7U062316; Wed, 8 Sep 2010 16:07:44 GMT (envelope-from nobody) Message-Id: <201009081607.o88G7i7U062316@www.freebsd.org> Date: Wed, 8 Sep 2010 16:07:44 GMT From: Rich Ercolani To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: misc/150390: zfs deadlock when arcmsr reports drive faulted X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 16:10:03 -0000 >Number: 150390 >Category: misc >Synopsis: zfs deadlock when arcmsr reports drive faulted >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Sep 08 16:10:02 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Rich Ercolani >Release: 8.1 >Organization: JHU ACM >Environment: FreeBSD manticore.acm.jhu.edu 8.1-STABLE FreeBSD 8.1-STABLE #4 r211397M: Mon Aug 16 18:47:31 EDT 2010 root@manticore.acm.jhu.edu:/usr/obj/usr/local/ncvs/src/sys/DTRACE amd64 >Description: System deadlocks 100% reliably when a disk is reported FAULTED in the arcmsr card. dmesg looks like: arcmsr0:block 'read/write' command with gone raid volume Cmd= 8, TargetId=1, Lun=5 arcmsr0:block 'read/write' command with gone raid volume Cmd= 8, TargetId=1, Lun=5 arcmsr0:block 'read/write' command with gone raid volume Cmd= 8, TargetId=1, Lun=5 zpool and zfs-related commands, and all IO to the affected pool, hang forever in state D. procstat reports: [root@manticore ~]# ps aux | grep zpool stump 3287 0.0 0.0 15700 1540 0 D+ 12:03PM 0:00.00 zpool status root 3286 0.0 0.0 15700 1528 1 T+ 12:03PM 0:00.00 zpool status root 3316 0.0 0.0 9120 1164 3 S+ 12:07PM 0:00.00 grep zpool [root@manticore ~]# procstat -k 3286 PID TID COMM TDNAME KSTACK 3286 100484 zpool - mi_switch sleepq_wait _cv_wait spa_config_enter spa_config_generate spa_open_common spa_get_stats zfs_ioc_pool_stats zfsdev_ioctl devfs_ioctl_f kern_ioctl ioctl syscall Xfast_syscall [root@manticore ~]# procstat -k 3287 PID TID COMM TDNAME KSTACK 3287 100532 zpool - mi_switch sleepq_wait _cv_wait spa_config_enter spa_config_generate spa_open_common spa_get_stats zfs_ioc_pool_stats zfsdev_ioctl devfs_ioctl_f kern_ioctl ioctl syscall Xfast_syscall >How-To-Repeat: 1) Have a disk fault on an arcmsr card. 2) Hang! >Fix: >Release-Note: >Audit-Trail: >Unformatted: