From owner-freebsd-fs@freebsd.org Sat Aug 29 06:28:00 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C98989C50A2 for ; Sat, 29 Aug 2015 06:28:00 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (unknown [IPv6:2001:388:f000::349d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps.rulingia.com", Issuer "CAcert Class 3 Root" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 58D531FAB for ; Sat, 29 Aug 2015 06:27:59 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (c220-239-242-83.belrs5.nsw.optusnet.com.au [220.239.242.83]) by vps.rulingia.com (8.15.2/8.15.2) with ESMTPS id t7T6Rop3030472 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Sat, 29 Aug 2015 16:27:55 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.9/8.14.9) with ESMTP id t7T6Rhs4003014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 29 Aug 2015 16:27:43 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.9/8.14.9/Submit) id t7T6RhVa003013 for freebsd-fs@freebsd.org; Sat, 29 Aug 2015 16:27:43 +1000 (AEST) (envelope-from peter) Date: Sat, 29 Aug 2015 16:27:43 +1000 From: Peter Jeremy To: freebsd-fs@freebsd.org Subject: Panic in zfs_blkptr_verify() Message-ID: <20150829062743.GA2996@server.rulingia.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ZGiS0Q5IWpPtfppv" Content-Disposition: inline X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender succeeded STARTTLS authentication, not delayed by milter-greylist-4.4.3 (vps.rulingia.com [103.243.244.15]); Sat, 29 Aug 2015 16:27:55 +1000 (AEST) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Aug 2015 06:28:01 -0000 --ZGiS0Q5IWpPtfppv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm trying to upgrade my main (amd64) server from 10-stable r276177 to r287251 but the new kernel consistently panics: panic: Solaris(panic): blkptr at 0xfffff80015961848 DVA 0 has invalid OFFSE= T 15724224479232 cpuid =3D 2 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe086027a= ff0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe086027b0a0 vpanic() at vpanic+0x126/frame 0xfffffe086027b0e0 panic() at panic+0x43/frame 0xfffffe086027b140 vcmn_err() at vcmn_err+0xcf/frame 0xfffffe086027b270 zfs_panic_recover() at zfs_panic_recover+0x60/frame 0xfffffe086027b2d0 zfs_blkptr_verify() at zfs_blkptr_verify+0x297/frame 0xfffffe086027b310 zio_read() at zio_read+0x2f/frame 0xfffffe086027b3a0 arc_read() at arc_read+0xb1e/frame 0xfffffe086027b450 dmu_objset_open_impl() at dmu_objset_open_impl+0x196/frame 0xfffffe086027b4= e0 dsl_pool_init() at dsl_pool_init+0x2a/frame 0xfffffe086027b510 spa_load() at spa_load+0xa20/frame 0xfffffe086027b650 spa_load_best() at spa_load_best+0x6f/frame 0xfffffe086027b6c0 spa_open_common() at spa_open_common+0x102/frame 0xfffffe086027b730 pool_status_check() at pool_status_check+0x4e/frame 0xfffffe086027b760 zfsdev_ioctl() at zfsdev_ioctl+0x52e/frame 0xfffffe086027b800 devfs_ioctl_f() at devfs_ioctl_f+0x121/frame 0xfffffe086027b860 kern_ioctl() at kern_ioctl+0x160/frame 0xfffffe086027b8c0 sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe086027b9a0 amd64_syscall() at amd64_syscall+0x22e/frame 0xfffffe086027bab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe086027bab0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x801a1dc5a, rsp =3D 0x= 7fffffffd028, rbp =3D 0x7fffffffd0a0 --- Unfortunately, I'm not sure how to resolve this. zfs_blkptr_verify() was MFH in r277582 so the checks don't exist in my old kernel. This means it could be a problem with one of my pools, rather than a software bug. But there's no information in the panic that would let me identify where the dodgy offset was found other than it's DVA 0 of some undefined blkptr_t. I've tried setting dumpon prior to this point (and I can see that dumpdev is set) but I don't get a crashdump. Since this is my primary server, I'd prefer not to have it down for an extended period whilst I rummaged around with ddb. Any suggestions as to how to proceed? --=20 Peter Jeremy --ZGiS0Q5IWpPtfppv Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJV4VDfXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs03iQP/3vVZyoSKQd/GMKaNMZJBP8P 9h0JAtr8h5byoQiOH4yTitsLnF378HaK67Ay/Uf0uGbITFo28qe5fK1DU4YCBCXA lUIXouCUCK3i20e7NFdA1hb9xxd7zsbJUOvURY8bKD9wRVFKK54hgNcAZQ8c3qmw 9gk5EcJ9I1seAF782iIwgrN/9TuLC06w6q2s/v6Mk0afSyiS5TAN93sLuXbJE1os qXXkIqSBczLKu3HAHNenS0XlfL0RsSLgDvpU94YR68oLoxKmb9Atap7/zzv/QDWZ TNRjgXWBuMBlvcIkNUUhcuXeVdcNEe0q/wrH70PmTTmYNsHRrURfEZFgdxS/T6Ze H5MdQSQDugTsxNws6fasCbsChCIsNpU9CWmRPXpBK2To+V1D5JOLyX9YpaeIpU0v FEj99zB4DSOLL7+vPDxCs5meaiiCjmi3O9ys3GVLB9lHI2wmLMzPwcDXkWhTYvAc R+I7KJXuChnCIzGZ+J3dSPVHX26jNwkhOzaDX2EKq7r+ABKmHCoSpieMKI5ky008 by59Z2o5+27TMGOk3yf+sKOIbLW7xlkenhxDn4lY9BHUjMxid/6x/MIRg+huInKj K9EY4SiVbKgNNxrGAj9hoR4kGrN5mYs3Q3Q4h7WDBgWYgxyU6ybF2F/zewEad9zG SZwGMZJeZ5S+3tScBEqy =N9jw -----END PGP SIGNATURE----- --ZGiS0Q5IWpPtfppv--