From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 14:07:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 99174E57 for ; Wed, 28 Nov 2012 14:07:45 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 332C78FC14 for ; Wed, 28 Nov 2012 14:07:44 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so6785749vba.13 for ; Wed, 28 Nov 2012 06:07:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=8ohSdWJuOlW9Sh/TM6q41SpFguJmKAoO7COLeb2hfm4=; b=J1GTWxBxhhqPRo59SD3wMEJXdtPomtTy5M0axbGGww8RIyDQGv1oJs2kGGtoEVay8Y jLS8f4KuE0vzfe4zoWkTkIkAUukXHl+MaFdZ9V7+ll8lhzd7iCEZU8B/6vULY5c4f/0j gkdOi+pLeYM2jX4d+YQY4Xqo19D/8AdA0IZlw9L4VGGBS2tvjXxedwzxz8I0an9Zihza 7Usd1Eh1Djpk9NtFv3KTl65/1P22PqOqHFfxXBW8Kbjqz5kiFIc+B9wzqJlFT5wuGglQ ElnRfP9eQLUg9seUyHH3DDmjQxVm7KHcaPGq68YK3qopDvT8ho4OIp7CVxNTBXhbngRX /kYw== Received: by 10.220.221.5 with SMTP id ia5mr28223054vcb.65.1354111664221; Wed, 28 Nov 2012 06:07:44 -0800 (PST) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx.google.com with ESMTPS id q7sm11800287vdw.22.2012.11.28.06.07.42 (version=SSLv3 cipher=OTHER); Wed, 28 Nov 2012 06:07:43 -0800 (PST) Received: by mail-vc0-f182.google.com with SMTP id fo13so18450181vcb.13 for ; Wed, 28 Nov 2012 06:07:42 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.91.73 with SMTP id cc9mr16204302vdb.48.1354111662158; Wed, 28 Nov 2012 06:07:42 -0800 (PST) Received: by 10.220.137.8 with HTTP; Wed, 28 Nov 2012 06:07:41 -0800 (PST) In-Reply-To: <50B61672.2070406@FreeBSD.org> References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> <50B5CFAF.8070306@FreeBSD.org> <50B61672.2070406@FreeBSD.org> Date: Wed, 28 Nov 2012 07:07:41 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Andriy Gapon X-Gm-Message-State: ALoCoQks9ltvXZUEiWfaHsQcb69L2LZGyFcvRZCK45bUJ/VEQriztCO3D8wJFPAtgM8uWQ7nUjCw Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 14:07:45 -0000 On Wed, Nov 28, 2012 at 6:49 AM, Andriy Gapon wrote: > on 28/11/2012 15:41 Josh Beard said the following: > > > > > > On Wed, Nov 28, 2012 at 1:47 AM, Andriy Gapon > > wrote: > > > > on 27/11/2012 23:47 Josh Beard said the following: > > > Thanks! Here we go: > > > > > > (kgdb) frame 7 > > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is > not available. > > > ) at > > > > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > (kgdb) list > > > 1850 &xattr_obj, sizeof (xattr_obj)); > > > 1851 if (error == 0 && xattr_obj) { > > > 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); > > > 1853 ASSERT3U(error, ==, 0); > > > 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); > > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > 1856 } > > > 1857 > > > 1858 mutex_enter(&zp->z_lock); > > > 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && > may_delete_now) > > > > That's what I suspected. > > > > > (kgdb) frame 7 > > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is > not available. > > > ) at > > > > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > (kgdb) info local > > > No locals. > > > (kgdb) > > > > A little bit unfortunate. > > > > > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) > > > 249868 > > > > > > ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > > > > > # zdb -ddddd store/tdxs1 249868 > > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 > objects, rootbp > > > DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU > objset] fletcher4 > > > lzjb LE contiguous unique double size=800L/200P > birth=1167710L/1167710P > > > fill=1106389 > cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d > > > > > > Object lvl iblk dblk dsize lsize %full type > > > 249868 1 16K 512 0 512 0.00 ZFS plain file > > > 201 bonus System > attributes > > > dnode flags: USERUSED_ACCOUNTED > > > dnode maxblkid: 0 > > > path > > > > > > /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > > uid 300002 > > > gid 80 > > > atime Tue Nov 27 14:43:00 2012 > > > mtime Thu Feb 23 08:59:21 2012 > > > ctime Fri Sep 14 01:12:37 2012 > > > crtime Fri Sep 14 01:11:50 2012 > > > gen 81430 > > > mode 120755 > > > size 17 > > > parent 249866 > > > links 1 > > > pflags 40800000104 > > > xattr 230 > > > > > > # zdb -ddddd store/tdxs1 230 > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, > rootbp > > DVA[0]=<0:8000284800:400> DVA[1]=<0:308006b3800:400> [L0 DMU objset] > fletcher4 > > lzjb LE contiguous unique double size=800L/200P birth=1167748L/1167748P > > fill=1106389 cksum=16e1e08cb8:70a50f1ec5a:13419c71d1cda:260fbed28af8f5 > > > > Object lvl iblk dblk dsize lsize %full type > > zdb: dmu_bonus_hold(230) failed, errno 2 > > > > errno 2 is ENOENT, so all pieces match. > The file appears to have extended attributes, but they do not actually > exist. > If your ZFS module was built with debug, then ASSERT3U(error, ==, 0) would > be > triggered. > > This is something that I said earlier in the private conversation that I > mentioned: > > Andriy Gapon said: > > The relevant commits are r240632 and r240345. I can't recall when I > MFC-ed them > > to stable/9. Most likely they are not in 9.1 releng branch. > [snip] > > I also don't have a good advice on how to fix the existing corruption. > > I'd probably go with using something like tar/cpio/pax to move data to > fresh > > filesystem. Please be sure to use latest stable/9 or 8 to not run into > the > > issue again. > > > > zfs send/recv won't help, it would mindlessly replicate the corrupted > attributes. > > To this I might add that the bugs were not FreeBSD-specifc and they may be > still > present in ZFS upstream and other ZFS ports. > > Thank you for the debugging information. > -- > Andriy Gapon > Andriy, Does this mean re-creating the dataset (or the pool?) would resolve this under a recent stable/9 build would resolve this? I have no problem doing so - as the data is redundant. Thanks for your help on this. Josh