From owner-freebsd-hackers@freebsd.org  Wed Dec 28 18:12:09 2016
Return-Path: <owner-freebsd-hackers@freebsd.org>
Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E974C95D7B
 for <freebsd-hackers@mailman.ysv.freebsd.org>;
 Wed, 28 Dec 2016 18:12:09 +0000 (UTC)
 (envelope-from wayne@manor.msen.com)
Received: from manor.msen.com (manor.msen.com [148.59.4.66])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 097F71E7E
 for <freebsd-hackers@freebsd.org>; Wed, 28 Dec 2016 18:12:08 +0000 (UTC)
 (envelope-from wayne@manor.msen.com)
Received: from manor.msen.com (localhost [127.0.0.1])
 by manor.msen.com (8.12.11/8.12.11) with ESMTP id uBSHj9Ml048262
 for <freebsd-hackers@freebsd.org>; Wed, 28 Dec 2016 12:45:10 -0500 (EST)
 (envelope-from wayne@manor.msen.com)
Received: (from wayne@localhost)
 by manor.msen.com (8.12.11/8.12.11/Submit) id uBSHj9X4048261
 for freebsd-hackers@freebsd.org; Wed, 28 Dec 2016 12:45:09 -0500 (EST)
 (envelope-from wayne)
Date: Wed, 28 Dec 2016 12:45:09 -0500
From: Michael Wayne <freebsd07@wayne47.com>
To: freebsd-hackers@freebsd.org
Subject: ZFS failure leads to panic/boot loop
Message-ID: <20161228174509.GR32352@manor.msen.com>
Mail-Followup-To: freebsd-hackers@freebsd.org
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4.2.1i
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 28 Dec 2016 18:12:09 -0000

11.0 server installed from initial setup with ZFS mirrored drives. 

Ran fine for a few weeks sitting in the rack, then became unresponsive.

Plugging in a console showed it was in a panic / boot loop due to ZFS.

Would like suggestions on what is wrong that caused this problem
as well as how to proceed to recover from it.

Typing this from a phone photo of the console screen so showing
only what I believe to be relevant; I can supply actual addresses
if required.  Boot gets to:

[...]
uhub2: 4 ports...
panic: Solaris(panic) blkptr at 0xfffffe0003244b80 has invalid CHECKSUM 0
cpuid - 0
KDB: stack backtrace
[...]
   zfs_panic_recover
   zfs_blkptr_verify
   zio_read
   spa_load_verify