From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 21:47:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0B5578 for ; Tue, 27 Nov 2012 21:47:31 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8B9E48FC15 for ; Tue, 27 Nov 2012 21:47:30 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so14724903obc.13 for ; Tue, 27 Nov 2012 13:47:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=1NCWJfB7phAIetHJ7OfEkIy1yh8xr8FGAP58XSsTkfk=; b=dbCgqCz3KZV/XkwppAKsn2v+OuKpJwFU5X4H+HZkNpIgVlbliT8VtIlkLM/JOAugvA fbwmuBl2boFXAHJgqbZaqT8TQpvI/NrZrCCMurIfnjOd1X2dyHNqE3500nEzK7cc83Lo GVZQv5K8q/bfKjLkhVnrsc+JKtgreKPTzUdWpODbe2M737VXvrIMk6HCyAK89C4q6Y0f FzXdL45hxC9zZqOPEUDRuA/hMuJbbbmIsHt3+qnWjE+8FNYPb3tdXpl+gOw3tC9znei5 6XyqxHHFmg6q85QKfHZJLOPQvLAnz63O+dyZGjKEM6TOSh9oH4xWAHRWSPozTbiwTN20 XZZg== Received: by 10.182.131.100 with SMTP id ol4mr1068990obb.38.1354052850435; Tue, 27 Nov 2012 13:47:30 -0800 (PST) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx.google.com with ESMTPS id m8sm17726171oeb.3.2012.11.27.13.47.28 (version=SSLv3 cipher=OTHER); Tue, 27 Nov 2012 13:47:29 -0800 (PST) Received: by mail-oa0-f54.google.com with SMTP id n9so16609447oag.13 for ; Tue, 27 Nov 2012 13:47:28 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.4.168 with SMTP id l8mr7315744oel.48.1354052848009; Tue, 27 Nov 2012 13:47:28 -0800 (PST) Received: by 10.60.14.194 with HTTP; Tue, 27 Nov 2012 13:47:27 -0800 (PST) In-Reply-To: <50B52CEC.9080208@FreeBSD.org> References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> Date: Tue, 27 Nov 2012 14:47:27 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Andriy Gapon X-Gm-Message-State: ALoCoQnTNv0KWkqOWw0Uf98pAOIUDCDEe0e5d8T1v3eG2dN4daHUu2nvwF467NpD1CyREE0BJqCa Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 21:47:31 -0000 On Tue, Nov 27, 2012 at 2:13 PM, Andriy Gapon wrote: > on 27/11/2012 22:32 Josh Beard said the following: > > > > > > On Tue, Nov 27, 2012 at 11:48 AM, Andriy Gapon > > wrote: > > > > on 27/11/2012 20:25 Josh Beard said the following: > > > Hello, > > > > > > I have a system that I can consistently reproduce a panic on when > trying to > > > delete certain data. The data is data that was rsynced from > another system > > > - nothing terribly unique. This has been ongoing from several > months, > > > starting with 9.0-RELEASE and now running 9.1-RC3. > > > > > > I can't find anything in common with the files that I can trigger > the > > > panics with. One is a simple gzipped archive where some are plain > text. > > > Strangely, I can only reproduce it with data that was rsynced > from that > > > particular system (which is a Mac). > > > > Josh, > > > > I am collecting these cases, thank you for another one. > > I had an interesting investigation of > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173747 > > Unfortunately, for some reason the whole conversation stayed private. > > I see that also opened a PR earlier: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/170238 > > > > Could you please provide the following info? > > >From kgdb: > > - list in frame 7 (zfs_freebsd_remove), so that I can see the code > line > > - local variables from frame 7 (info local) > > > > > > > > Andriy, > > > > Thanks for your quick response. I've never used kgdb, so forgive my > ignorance > > here. Is this what you're looking for? If not, if you could you > elaborate on > > those? > > > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not > available. > > ) at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > > > > (kgdb) list zfs_freebsd_remove > > 5796 struct vop_remove_args /* { > > 5797 struct vnode *a_dvp; > > 5798 struct vnode *a_vp; > > 5799 struct componentname *a_cnp; > > 5800 } */ *ap; > > 5801 { > > 5802 > > 5803 ASSERT(ap->a_cnp->cn_flags & SAVENAME); > > 5804 > > 5805 return (zfs_remove(ap->a_dvp, ap->a_cnp->cn_nameptr, > > Not quite :-) > frame 7 > list > > > (kgdb) info frame 7 > > Stack frame at 0xffffff8466a6a920: > > rip = 0xffffffff80ebd45a in zfs_freebsd_remove > > > (/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855); > > saved rip 0xffffffff8081cf13 > > called by frame at 0xffffff8466a6a940, caller of frame at > 0xffffff8466a6a7a0 > > source language c. > > Arglist at 0xffffff8466a6a910, args: ap=Variable "ap" is not available. > > frame 7 > info local > Thanks! Here we go: (kgdb) frame 7 #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); (kgdb) list 1850 &xattr_obj, sizeof (xattr_obj)); 1851 if (error == 0 && xattr_obj) { 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); 1853 ASSERT3U(error, ==, 0); 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); 1856 } 1857 1858 mutex_enter(&zp->z_lock); 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && may_delete_now) (kgdb) frame 7 #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); (kgdb) info local No locals. (kgdb) > > Also, for one of the files that trigger the problem: > > - ls -i to obtain its inode number > > - zdb -ddddd > > > > > > # ls -i kyofilter\ v2.2.pax.gz (this is a symlink. the file that it's > linked > > to does *not* panic the system if I try to delete it). > > Hmm, then could you please rather do 'ls -Pi kyofilter\ v2.2.pax.gz' and > then > use that value for the zdb command? > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) 249868 ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz # zdb -ddddd store/tdxs1 249868 Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1167710L/1167710P fill=1106389 cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d Object lvl iblk dblk dsize lsize %full type 249868 1 16K 512 0 512 0.00 ZFS plain file 201 bonus System attributes dnode flags: USERUSED_ACCOUNTED dnode maxblkid: 0 path /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz uid 300002 gid 80 atime Tue Nov 27 14:43:00 2012 mtime Thu Feb 23 08:59:21 2012 ctime Fri Sep 14 01:12:37 2012 crtime Fri Sep 14 01:11:50 2012 gen 81430 mode 120755 size 17 parent 249866 links 1 pflags 40800000104 xattr 230 Indirect blocks: > > > 247126 kyofilter v2.2.pax.gz > > > > # zdb -ddddd store/tdxs1 247126 > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, > rootbp > > DVA[0]=<0:80001a2400:400> DVA[1]=<0:30800610000:400> [L0 DMU objset] > fletcher4 > > lzjb LE contiguous unique double size=800L/200P birth=1166838L/1166838P > > fill=1106389 cksum=19391f0f67:78eb24a9cca:1439005549d01:275015332d1bdf > > > > Object lvl iblk dblk dsize lsize %full type > > 247126 1 16K 512 0 512 0.00 ZFS plain file > > 201 bonus System attributes > > dnode flags: USERUSED_ACCOUNTED > > dnode maxblkid: 0 > > path > /tech/2012-09-14-01-00/Drivers/Kyocera/.old/C2126.old/Kyocera OS > > X 10.5+ Web build 2011.01.27.mpkg/Contents/Packages/Kyocera OS X > > subinstaller.mpkg/Contents/Packages/kyofilter > > v2.2.pkg/Contents/Resources/kyofilter v2.2.pax.gz > > uid 1001 > > gid 80 > > atime Tue Nov 27 13:27:57 2012 > > mtime Tue Jul 12 14:17:16 2011 > > ctime Fri Sep 14 01:05:23 2012 > > crtime Fri Sep 14 01:04:11 2012 > > gen 81338 > > mode 120755 > > size 17 > > parent 247122 > > links 1 > > pflags 40800000104 > > xattr 155 > > -- > Andriy Gapon >