From owner-freebsd-fs@FreeBSD.ORG  Fri Oct 10 01:35:18 2014
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id B3D6263A;
 Fri, 10 Oct 2014 01:35:18 +0000 (UTC)
Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35])
 by mx1.freebsd.org (Postfix) with ESMTP id 77A04B2B;
 Fri, 10 Oct 2014 01:35:18 +0000 (UTC)
Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534)
 id 389C420E709A4; Fri, 10 Oct 2014 01:35:16 +0000 (UTC)
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
 smtp1.multiplay.co.uk
X-Spam-Level: **
X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX,
 FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1
Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170])
 by smtp1.multiplay.co.uk (Postfix) with ESMTPS id B797620E709A2;
 Fri, 10 Oct 2014 01:35:14 +0000 (UTC)
Message-ID: <F93FC06BE5854556BF1F4318690C728C@multiplay.co.uk>
From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Steve Wills" <swills@freebsd.org>,
	"Andriy Gapon" <avg@freebsd.org>
References: <20141008004045.GA24762__48659.9047123038$1412728878$gmane$org@mouf.net>
 <5434D1CE.8010801@FreeBSD.org> <20141010012724.GD79158@mouf.net>
Subject: Re: zfs hang
Date: Fri, 10 Oct 2014 02:35:14 +0100
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1";
 reply-type=original
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5931
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
Cc: current@freebsd.org, fs@freebsd.org
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 10 Oct 2014 01:35:18 -0000


----- Original Message ----- 
From: "Steve Wills" <swills@freebsd.org>
To: "Andriy Gapon" <avg@freebsd.org>
Cc: <current@freebsd.org>; <fs@freebsd.org>
Sent: Friday, October 10, 2014 2:27 AM
Subject: Re: zfs hang


> On Wed, Oct 08, 2014 at 08:55:26AM +0300, Andriy Gapon wrote:
>> On 08/10/2014 03:40, Steve Wills wrote:
>> > Hi,
>> > 
>> > Not sure which thread this belongs to, but I have a zfs hang on one of my boxes
>> > running r272152. Running procstat -kka looks like:
>> > 
>> > http://pastebin.com/szZZP8Tf
>> > 
>> > My zpool commands seem to be hung in spa_errlog_lock while others are hung in
>> > zfs_lookup. Suggestions?
>> 
>> There are several threads in zio_wait.  If this is their permanent state then
>> there is some problem with I/O somewhere below ZFS.
> 
> Thanks for the feedback. It seems one of my disks is dying, I rebooted and it
> came up OK, but today I got:
> 
>  panic: I/O to pool 'rpool' appears to be hung on vdev guid ..... at '/dev/ada0p3'
> 
> I have screenshots and backtrace if anyone is interested. Dying drives
> shouldn't cause panic, right?

Its the deadman timer kicking in so yes, thats expected.

The following sysctls control this behaviour if you want to try and recover:
vfs.zfs.deadman_synctime_ms: 1000000
vfs.zfs.deadman_checktime_ms: 5000
vfs.zfs.deadman_enabled: 1

    Regards
    Steve