From owner-freebsd-geom@FreeBSD.ORG Sun Feb 6 04:24:00 2005 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0650216A4D0 for ; Sun, 6 Feb 2005 04:23:58 +0000 (GMT) Received: from FS.denninger.net (wsip-68-15-213-52.at.at.cox.net [68.15.213.52]) by mx1.FreeBSD.org (Postfix) with ESMTP id 448E743D2F for ; Sun, 6 Feb 2005 04:23:57 +0000 (GMT) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net (localhost [127.0.0.1]) by FS.denninger.net (8.13.1/8.13.1) with SMTP id j164NpV5015372 for ; Sat, 5 Feb 2005 22:23:51 -0600 (CST) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net [127.0.0.1] by Spamblock-sys; Sat Feb 5 22:23:51 2005 Received: (from karl@localhost) by FS.denninger.net (8.13.1/8.13.1/Submit) id j164NprT015370; Sat, 5 Feb 2005 22:23:51 -0600 (CST) (envelope-from karl) Message-ID: <20050205222351.A15328@denninger.net> Date: Sat, 5 Feb 2005 22:23:51 -0600 From: Karl Denninger To: Paul Mather References: <20050205135705.A10437@denninger.net> <20050205201237.GB1666@darkness.comp.waw.pl> <20050205170415.A12620@denninger.net> <20050205230842.GD1666@darkness.comp.waw.pl> <20050205173326.B12620@denninger.net> <1107655941.4894.29.camel@zappa.Chelsea-Ct.Org> <20050205205815.B14091@denninger.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.93.2i In-Reply-To: <20050205205815.B14091@denninger.net>; from Karl Denninger on Sat, Feb 05, 2005 at 08:58:15PM -0600 Organization: Karl's Sushi and Packet Smashers X-Die-Spammers: Spammers cheerfully broiled for supper and served with ketchup! cc: Pawel Jakub Dawidek cc: freebsd-geom@freebsd.org Subject: Re: Gmirror - how to do? X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2005 04:24:00 -0000 On Sat, Feb 05, 2005 at 08:58:15PM -0600, Karl Denninger wrote: > I think you're missing what I'm trying to accomplish, and where (I think) > it won't work. > > Given the following configuration: > > 1. Two internal SATA drives, each of size "X". > 2. An external SATA drive bay, with two disk carriers each of size "X" (or > larger) > 3. An initial RAID-1 boot volume that is comprised of the two internal disks. > 4. FreeBSD 5-Stable > > The goal is that: > > 1. I can and it will NOT be > used, by default, by the mirror, even if the system should reboot for > some odd reason. This is important, because if I screw myself in some > fashion on the running machine, having the backup disk scribbled on > at the same time defeats the purpose of a backup. > > 2. I can cause the carrier disk to be recognized, and if it is, the system > will bring it current. A physical command to initiate this is > acceptable (indeed, required, since I do not want that process to start > unexpectedly.) > > 3. Once the disk in the carrier IS current, I can detach it from the > running configuration and return to State #1 above. I now have a > "near-line" backup that can be mounted (SEPARATELY!) at any time to > recover data if for some reason there is a problem (e.g. I 'fat-finger' > an "rm -rf" from a directory that I really wanted) I can also rotate > these disks in the carriers so that I have an offsite copy, plus a > near-line copy. > > 4. If the original machine takes a dump entirely (e.g. the controller > or OS goes insane and scribbles on both disks, there is a physical > catastrophe with the system, etc) I can take the disk in either the > carrier or the safe deposit box, plug it into an arbitrary machine > and boot it without drama. I could also (if I had TWO disks) boot the > most current one and then bring the other into the mirror, effectively > recovering both the system and the redundancy all at once. An FSCK in > that instance is acceptable since I don't want to quiesce the system > (which I understand is necessary to unmount/etc) Critical applications > can be stopped in (3) before the disk in the carrier is detached (e.g. > dbms, etc) so I know THAT data is current. > > The problem(s) I see with the possible ways to do this are: > > 1. A "gmirror remove" removes the metadata on the disk and it will no > longer boot, since the root filesystem isn't on a mirror anymore (the > metadata is gone). > > 2. A "fake out" with a gmirror label leaves me vulnerable to an unexpected > resync without being prompted if the system reboots for any reason, and > the possibility of the system activating the root volume. The > latter could be catastrophically bad, especially if gmirror then rebuilt > the STALE disk back onto the other two, immediately destroying the > current data set! It leaves me with the possibility that the > mirror created on that disk (if its one of the "larger" ones) is too big, > and if that one is booted in recovery mode that the other backup disk > could not be made part of the mirror due to it being physically smaller > in block count on the same slice. In other words, whether the system is > truly recoverable transparently then depends on which disk is the one > you boot singly. That's not good. > > 3. A "gmirror deactivate" followed by a "gmirror forget" is likely (not > sure yet, will test) to leave me with an unbootable disk as in (1), > because while the metadata is still there it is marked "do not use"; > thus, the boot will fail (I think) > > 4. A "atacontrol detach" leaves me with the situation where a reboot > finds the disk and can leave me in exactly the same situation as (2). > > Something I have not tried is to: > > 1. Mark the mirror as "no automatic rebuilds." > 2. Bring the third disk current. > 3. Stop the critical processes, and perform a couple of sync's. > 4. Forcibly detach it using "atacontrol detach 2". This appears to > also spin the disk down. > 5. Clean up the idea that there is another disk out there with > "gmirror forget". I should now have a "complete" mirror set with > two components, and a detached disk. Since the mirror is set "no > automatic rebuilds", if the system reboots it should NOT reattach the > backup volume. > 6. The backup volume SHOULD be bootable on its own, without drama, as the > mirror, while it will be seen as degraded, should still be operational > off that volume. > > The only problem is that there is no way to insure any buffers for that > disk have been flushed, so its very similar to a "plug pull" for an > unprotected machine. > > I think I'll try that one next, although I don't care for the semantics. > Worst thing I can get is a panic for the unexpected detach, I think :-> > > I can always use the disk as if it was a great big tape drive, and just > dump each filesystem to it, which avoids all of this tomfoolery (maybe > with a minimal system on it so I can restore the dumps onto a new set of > mirrors) but while that will work, it kinda evades the purpose of what I'm > trying to do - come up with a way to "hot mirror" the machine in such a > fashion that recovery is painless and possible even for someone who is > completely untrained, being told only to stick the backup disk into a > carrier on a new CPU and turn on the power. > > If this sort of thing is a model that the gmirror authors didn't think of, > its something they might want to in the future..... As expected, a "gmirror deactivate" leaves you with an unbootable GEOM-Mirror disk, as when you attempt to boot from that disk the system complains that the volume was marked inactive, and it is then skipped. You can mount the 'raw' disk root partition (e.g. /dev/ad4s1a), and I suspect, but do not know, if doing so and then attempting to activate the mirror would succeed or cause really bad things to happen (since you already have the slice open with the root mount.) A forcible "detach" on the SATA bus works (you can boot the resulting volume), but is extremely messy. It also leaves you with a disk that when reinserted requires a manual "insert" command. However, it has one truly ugly potential in that if the disk is in the carrier when the machine is booted, it would appear possible that any of the providers - including the "stale" one you detached - could get picked up as the 'master'. That could have truly ugly consequences. If this is all I've got available I guess I have to live with it, but forcibly ripping a bus out from under a disk seems pretty nasty to me. I suspect that for safety reasons "deactivate" will have to be the way I go, and live with the fact that a manual boot in the event of a recovery will be necessary, along with lots of care. I will run a few more tests once I have a clean third disk again with the 'deactivate' option - I suspect that I may be able to clean it up from the "fixit disk" with some care manually and then boot from it. Not ideal, but probably workable. -- -- Karl Denninger (karl@denninger.net) Internet Consultant & Kids Rights Activist http://www.denninger.net My home on the net - links to everything I do! http://scubaforum.org Your UNCENSORED place to talk about DIVING! http://www.spamcuda.net SPAM FREE mailboxes - FREE FOR A LIMITED TIME! http://genesis3.blogspot.com Musings Of A Sentient Mind