From owner-freebsd-current@FreeBSD.ORG Sat Sep 24 04:34:25 2005 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B828416A41F; Sat, 24 Sep 2005 04:34:25 +0000 (GMT) (envelope-from Tor.Egge@cvsup.no.freebsd.org) Received: from pil.idi.ntnu.no (pil.idi.ntnu.no [129.241.107.93]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3BB6943D48; Sat, 24 Sep 2005 04:34:22 +0000 (GMT) (envelope-from Tor.Egge@cvsup.no.freebsd.org) Received: from cvsup.no.freebsd.org (c2h5oh.idi.ntnu.no [129.241.103.69]) by pil.idi.ntnu.no (8.13.1/8.13.1) with ESMTP id j8O4YK1x009689 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 24 Sep 2005 06:34:20 +0200 (MEST) Received: from localhost (localhost [127.0.0.1]) by cvsup.no.freebsd.org (8.13.1/8.13.1) with ESMTP id j8O4YKH0077270; Sat, 24 Sep 2005 04:34:20 GMT (envelope-from Tor.Egge@cvsup.no.freebsd.org) Date: Sat, 24 Sep 2005 04:34:19 +0000 (UTC) Message-Id: <20050924.043419.74681996.Tor.Egge@cvsup.no.freebsd.org> To: truckman@freebsd.org From: Tor Egge In-Reply-To: <200509232147.j8NLl5tw087085@gw.catspoiler.org> References: <200509232147.j8NLl5tw087085@gw.catspoiler.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned-By: mimedefang.idi.ntnu.no, using CLAMD X-SMTP-From: Sender=, Relay/Client=c2h5oh.idi.ntnu.no [129.241.103.69], EHLO=cvsup.no.freebsd.org X-Scanned-By: MIMEDefang 2.48 on 129.241.107.38 X-Scanned-By: mimedefang.idi.ntnu.no, using MIMEDefang 2.48 with local filter 16.42-idi X-Filter-Time: 1 seconds Cc: scottl@freebsd.org, current@freebsd.org, mckusick@freebsd.org Subject: Re: soft updates / background fsck directory link count bug X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Sep 2005 04:34:25 -0000 > It appears that there is some sort of interaction between soft updates > and background fsck that results in the link count of the parent of one > of these directories being double decremented, resulting in the file > system being put into an invalid state. If the snapshot for background fsck is taken on a file system which has pending softupdate dependencies then this can happen. For this particular case, the system had a pending dirrem dependency. > The following transcript demonstrates what happens if a background fsck > is run after the leaf directory is removed. What is interesting is that > after the directory the leaf directory has been removed, the effective > link count of the parent directory (displayed by ls) has been > decremented from 3 to 2, whereas the on-disk link count shown by fsdb is > still 3. The background fsck appears to detect the link count as 3, and > executes the sysctl call to decrement the link count, causing both the > effective and actual link counts to be decremented to 1. > My suspicion is that the physical update of the parent directory's link > count after the rmdir of the leaf directory has been deferred until the > leaf directory's inode is zeroed, which turns out to be an indefinite > wait because the inode doesn't get zeroed until fsck is run. ufs_rmdir() calls ufs_dirremove() after having lowered i_effnlink in memory for both leaf and parent directory. ufs_dirremmove() calls softdep_setup_remove() which sets up the softupdates dependencies for reducing di_nlink on disk for leaf and parent directory when it's safe to do so (i.e. after the directory entry referencing the leaf directory has been cleared on disk). See code in reassignbuf() for various delays before the syncer process pushes the dirty buffers to disk. The background fsck found the the di_nlink value being 3 on the parent directory and issued an FFS_ADJ_REFCNT sysctl to reduce it by one, having no knowledge about the pending dirrem dependency. See sysctl_ffs_fsck() for the handling of that sysctl. After background fsck has run and the dirrem dependency has been processed, the link counts for the parent directory are both 1. The latest panic shown on , "panic: handle_written_inodeblock: live inodedep" was probably caused by this issue. If the snapshot was taken while a directory or file was being removed then it might contain an unreferenced inode with a nonzero link count. The background fsck would reduce the link count for the inode, triggering freeing of the inode (c.f. ufs_inactive(), UFS_VFREE(), ffs_vfree() and softdep_freefile()). After writing the zeroed inode to disk the system would panic due to the still pending dirrem dependency. - Tor Egge