From owner-freebsd-current Fri Feb 21 1:58:29 2003 Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DEBC537B401 for ; Fri, 21 Feb 2003 01:58:27 -0800 (PST) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2351D43FA3 for ; Fri, 21 Feb 2003 01:58:27 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0025.cvx22-bradley.dialup.earthlink.net ([209.179.198.25] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 18m9wj-0006yw-00; Fri, 21 Feb 2003 01:58:22 -0800 Message-ID: <3E55F7E8.BD5BF990@mindspring.com> Date: Fri, 21 Feb 2003 01:56:56 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: kalts@estpak.ee Cc: Darryl Okahata , current@FreeBSD.ORG Subject: Re: Vinum R5 [was: Re: background fsck deadlocks with ufs2 and big disk] References: <20030220200317.GA5136@kevad.internal> <200302202228.OAA03775@mina.soco.agilent.com> <20030221080046.GA1103@kevad.internal> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a48b8a9d524fb2957cd685c6de5b7d9845a8438e0f32a48e08350badd9bab72f9c350badd9bab72f9c Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Vallo Kallaste wrote: > The crashes and anomalies with filesystem residing on R5 volume were > related to vinum(R5)/softupdates combo. The vinum R5 and system as > a whole were stable without softupdates. Only one problem remained > after disabling softupdates, while being online and user I/O going > on, rebuilding of failed disk corrupt the R5 volume completely. > Don't know is it fixed or not as I don't have necessary hardware at > the moment. The only way around was to quiesce the volume before > rebuilding, umount it, and wait until rebuild finished. I'll suggest > extensive testing cycle for everyone who's going to work with > vinum R5. Concat, striping and mirroring has been a breeze but not > so with R5. I think this is an expected problem with a lot of concatenation, whether through Vinum, GEOM, RAIDFrame, or whatever. This comes about for the same reason that you can't "mount -u" to turn Soft Updates from "off" to "on": Soft Updates does not tolerate dirty buffers for which a dependency does not exist, and will crap out when a pending dirty buffer causes a write. This could be fixed in the "mount -u" case for Soft Updates, and it can also be fixed for Vinum (et. al.). The key is the difference between a "mount -u" vs. a "umount ; mount", which comes down to flushing and invalidating all buffers on the underlying device, e.g.: vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY, p); vinvalbuf(devvp, V_SAVE, NOCRED, p, 0, 0); error = VOP_CLOSE(devvp, ronly ? FREAD : FREAD|FWRITE, FSCRED, p); error = VOP_OPEN(devvp, ronly ? FREAD : FREAD|FWRITE, FSCRED, p); VOP_UNLOCK(devvp, 0, p); ... Basically, after rebuilding, before allowing the mount to proceed, the Vinum (and GEOM and RAIDFRame, etc.) code needs to cause all the pending dirty buffers to be written. This will guarantee that there are no outstanding dirty buffers at mount time, which in turn guarantees that there will be no dirty buffers that the dependency tracking in Soft Updates does not know about. FWIW: I've maintained for over 6 years now that the mount update code should be modified to do this automatically (and provided patches; see early 1997 mailing list archives), essentially turning a "mount -u" into a "umount ; mount", without invalidating outstanding vnodes and in-core inodes or their references (so that open files do not break... they just get all their buffers taken away from them). Of course, the only open files that matter for device layering are the device exporting the layered block store, and the underlying component block stores that make it up (i.e. no open files there). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message