From owner-freebsd-stable@FreeBSD.ORG Tue Nov 30 10:34:33 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CB71116A4CE for ; Tue, 30 Nov 2004 10:34:33 +0000 (GMT) Received: from lakermmtao02.cox.net (lakermmtao02.cox.net [68.230.240.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3C30443D45 for ; Tue, 30 Nov 2004 10:34:33 +0000 (GMT) (envelope-from ski@indymedia.org) Received: from [10.0.0.42] (really [68.106.120.181]) by lakermmtao02.cox.net (InterMail vM.6.01.04.00 201-2131-117-20041022) with ESMTP id <20041130103432.BFEA329.lakermmtao02.cox.net@[10.0.0.42]> for ; Tue, 30 Nov 2004 05:34:32 -0500 Received: from 10.0.0.26 (SquirrelMail authenticated user ski); by wuhjuhbuh.afraid.org with HTTP; Tue, 30 Nov 2004 05:35:15 -0500 (EST) Message-ID: <2958.10.0.0.26.1101810915.squirrel@10.0.0.26> Date: Tue, 30 Nov 2004 05:35:15 -0500 (EST) From: "Brian Szymanski" To: freebsd-stable@freebsd.org User-Agent: SquirrelMail/1.4.3a X-Mailer: SquirrelMail/1.4.3a MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: vinum raid 5 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Nov 2004 10:34:33 -0000 >>>>>> "Brian" == Brian Szymanski writes: > > Brian> Actually I experienced a number of bugs with vinum in 5.3 that > Brian> proved fatal to a root vinum install (in fact, everything on > Brian> the second ATA channel was marked down after every reboot if I > Brian> recall correctly). > > I had some problems with a vinum machine (still back at 5.1) > recently. I removed a pair of drives and the remaining setup wouldn't > come back up. Turns out that multiple incantations of 'vinum > setdaemon 0' and 'vinum saveconfig' finally fixed things. > > vinum is fragile. Hmm, the vinum setdaemon 0 ; vinum saveconfig ; trick didn't work for me. If anyone else finds themself in a similar situation and feels like living dangerously, I've implemented a solution with the below shell script, to be called from /etc/rc.local. Use at your own risk. First more on the symptom: on reboot, vinum looses all config information. You can load the vinum.cfg again, but this puts every subdisk in the Raid5 in the empty state (when in fact they should usually be in the up state). You can manually set these subdisks to be up, and find all your information as it was, a nice "feature". The solution: write a crappy little shell script to remember vinum's Raid5 config information, and set its state accordingly on boot before mounting/fscking, etc. If anyone uses this, I'd be interested to hear their results. It works for me with drive power pulls. Right now it depends on state being recorded at boot time, so if vinum thinks a subdisk is down (and thus out of date), then you reboot and the drive comes back to life for a moment, you're in deep doo-doo. That is to say: flaky drives may introduce errors onto the R5, but a drive that fails once and then stays down forever will be fine with this script (this is the usual case from what I've seen, but I haven't seen everything)... Also I'm not sure why bgfsck doesn't catch the filesystem automatically, so I used atd to schedule a job to run at +7 minutes with priority 10 to do the background fsck. The idea here being that if bgfsck does for some reason catch your Raid5, this fsck should lag behind and not cause any race condition problems. A future improvement would be to call the state saving portion of the script in rc.shutdown and periodically from cron. There are many others, but it "works for me" (tm). Of course the right solution is to fix {g}vinum so this hideousness isn't necessary... Cheers, Brian Szymanski ski@indymedia.org step 1: in /etc/rc.local, add: sh /etc/rc.vinum.raid5 step 2: create /etc/rc.vinum.raid5: #!/bin/sh ### configuration # ie, mount /dev/vinum/$VINUM_NAME $VINUM_MOUNTPT VINUM_MOUNTPT="/home" VINUM_NAME="big" # location of the configuration file with just the Raid5 info VINUM_CFG="/etc/vinum.big.cfg" # email address to send notifications to ADMIN_EMAIL="ski@wjb" # place to store state DEGRADE_FILE="/vinum.degraded" # if n=# subdisks, SUBDISKS=0..n-1, e.g. the below works for a 4 drive raid5 SUBDISKS="0 1 2 3" ### no config below this point #if already mounted, abort now mount | grep -q "^/dev/vinum/$VINUM_NAME on $VINUM_MOUNTPT" && exit 0 # mount R5 device # (why on earth doesn't vinum info persist across reboot?!?) /sbin/vinum create -f "$VINUM_CFG" if [ -e "$DEGRADE_FILE" ] ; then # start subdisks based on content of DEGRADE_FILE for i in $SUBDISKS ; do #start all non-degraded subdisks grep -q $VINUM_NAME.p0.s${i} "$DEGRADE_FILE" && \ echo $VINUM_NAME.p0.s$i down || \ vinum start $VINUM_NAME.p0.s${i} done #this maintains degraded state even if new drive is inserted #this is important or else we will lose major information else /sbin/vinum start $VINUM_NAME.p0 fi #let vinum catch up... sleep 1 #check the state, mail $ADMIN_EMAIL if things are horked... state=`/sbin/vinum l | grep "^P.$VINUM_NAME\.p0" | awk '{ print $5 }'` echo state: $state if [ "z$state" = zup ] ; then # mount /dev/vinum/$VINUM_NAME "$VINUM_MOUNTPT" #fsck and mount fsck -F -p "/dev/vinum/$VINUM_NAME" mount "/dev/vinum/$VINUM_NAME" "$VINUM_MOUNTPT" #bgfsck should do this automagically, but it doesn't? at "+7 minutes" <>"$DEGRADE_FILE" #fsck and mount fsck -F -p "/dev/vinum/$VINUM_NAME" mount "/dev/vinum/$VINUM_NAME" "$VINUM_MOUNTPT" #bgfsck should do this automagically, but it doesn't? at "+7 minutes" <