From owner-freebsd-stable Sun Jan 27 15:45:31 2002 Delivered-To: freebsd-stable@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id DC14E37B41B for ; Sun, 27 Jan 2002 15:42:49 -0800 (PST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 6831778312; Sun, 27 Jan 2002 14:26:43 +1030 (CST) Date: Sun, 27 Jan 2002 14:26:43 +1030 From: Greg Lehey To: Martin Blapp Cc: freebsd-stable@freebsd.org Subject: Re: double fault with vinum and 4.5 RC3 Message-ID: <20020127142643.A19909@wantadilla.lemis.com> References: <20020124184810.M47234-100000@levais.imp.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020124184810.M47234-100000@levais.imp.ch> User-Agent: Mutt/1.3.23i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thursday, 24 January 2002 at 19:14:10 +0100, Martin Blapp wrote: > > ... unfortunatly I'm not able to recover from a simulated > disk-crash. Here is what I made: > > Two disks: da0, da0, both 35GB > > 1) Installed FreeBSD on da0s1a > > 2) mount so single user mode > dd bs=4096k if=/dev/da0 of=/dev/da1 > > (to have the root on disk 2 too, I did this cold mirror) > > I plan to do a cold mirror procedure each evening, since > the root will be mostly static anyway. > > mount /dev/da1a /mnt > cd /mnt && dump -rf 0 /dev/da0a -f - | restore -xf - > rm /mnt/restoresytable > umount /mnt > > 3) I edited the disklabels and set da0e, da0f, da0g > to type vinum > > # vinum mirror -n var /dev/da0e /dev/da1e > # vinum mirror -n docsis /dev/da0f /dev/da1f > # vinum mirror -n docsisvar /dev/da0g /dev/da1g As I've mentioned elsewhere, this is seriously suboptimal. The "mirror" command is a toy for people getting used to Vinum. You want a proper config file. Then create one drive per spindle, and choose your subdisk sizes to match what you want. Specifically, your config file should look like this: drive 0 device /dev/da0e drive 1 device /dev/da1e volume var setupstate plex org concat sd drive 0 len 2096000s plex org concat sd drive 1 len 2096000s volume docsis setupstate plex org concat sd drive 0 len 1257000s plex org concat sd drive 1 len 1257000s volume docsisvar setupstate plex org concat sd drive 0 len 4651000s plex org concat sd drive 1 len 4651000s This will require repartitioning the disks, of course, and in the process you'll regain a little additional space. Theoretically you could repartition and use the existing data, but that is rather risky. > But now I tried what happens if the disk 0 fails, I mean da0. I > took da0 out and replaced it with a new disk. I switched jumpers, so > da0 got da1, and da1 got da0 and rebooted. I still don't know why you're switching jumpers. That's not needed. But it doesn't change anything. > I could boot fine, vinum failed to start and I was in single user > mode. The I saw: > > # vinum start > Warning: defective objects > > D vinumdrive0 State: referenced Device Avail: 0/0 MB > D vinumdrive2 State: referenced Device Avail: 0/0 MB > D vinumdrive4 State: referenced Device Avail: 0/0 MB > P var.p0 C State: faulty Subdisks: 1 Size: 1023 > P docsis.p0 C State: faulty Subdisks: 1 Size: 6143 > P docsisvar.p0 C State: faulty Subdisks: 1 Size: 22 > S var.p0.s0 State: stale PO: 0 B Size: 1023 > S docsis.p0.s0 State: stale PO: 0 B Size: 6143 > S docsisvar.p0.s0 State: stale PO: 0 B Size: 22 > > where is p1 ??? Not defective. I've repeated this configuration almost exactly (I was using much smaller disks, so the size is different). What I got was: vinum -> start Warning: defective objects D vinumdrive0 State: referenced A: 0/0 MB D vinumdrive2 State: referenced A: 0/0 MB D vinumdrive4 State: referenced A: 0/0 MB P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB This is simply a list of the defective objects, as the warning message states. To see them all, use the list (or l) command: vinum -> l 3 drives: D vinumdrive1 State: up /dev/da0e A: 0/1023 MB (0%) D vinumdrive3 State: up /dev/da0f A: 0/614 MB (0%) D vinumdrive5 State: up /dev/da0g A: 0/2271 MB (0%) D vinumdrive0 State: referenced A: 0/0 MB D vinumdrive2 State: referenced A: 0/0 MB D vinumdrive4 State: referenced A: 0/0 MB 3 volumes: V var State: up Plexes: 2 Size: 1023 MB V docsis State: up Plexes: 2 Size: 613 MB V docsisvar State: up Plexes: 2 Size: 2270 MB 6 plexes: P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB P var.p1 C State: up Subdisks: 1 Size: 1023 MB P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB P docsis.p1 C State: up Subdisks: 1 Size: 613 MB P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB P docsisvar.p1 C State: up Subdisks: 1 Size: 2270 MB 6 subdisks: S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB S docsis.p1.s0 State: up D: vinumdrive3 Size: 613 MB S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB S docsisvar.p1.s0 State: up D: vinumdrive5 Size: 2270 MB > Then I tried to do: > > # echo "drive evar device /dev/da1e" >> configfile > # echo "drive edocsis device /dev/da1f" >> configfile > # echo "drive edocsisvar device /dev/da1g" >> configfile > # vinum create configfile > > And boom I got a double fault. Yes, we've found the cause of that. What I get now is: vinum -> create /wantadilla/var/tmp/vinumconfig2-bp 6 drives: D vinumdrive1 State: up /dev/da0e A: 0/1023 MB (0%) D vinumdrive3 State: up /dev/da0f A: 0/614 MB (0%) D vinumdrive5 State: up /dev/da0g A: 0/2271 MB (0%) D vinumdrive0 State: referenced A: 0/0 MB D vinumdrive2 State: referenced A: 0/0 MB D vinumdrive4 State: referenced A: 0/0 MB D evar State: up /dev/da1e A: 1023/1023 MB (100%) D edocsis State: up /dev/da1f A: 614/614 MB (100%) D edocsisvar State: up /dev/da1g A: 2271/2271 MB (100%) 3 volumes: V var State: up Plexes: 2 Size: 1023 MB V docsis State: up Plexes: 2 Size: 613 MB V docsisvar State: up Plexes: 2 Size: 2270 MB 6 plexes: P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB P var.p1 C State: up Subdisks: 1 Size: 1023 MB P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB P docsis.p1 C State: up Subdisks: 1 Size: 613 MB P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB P docsisvar.p1 C State: up Subdisks: 1 Size: 2270 MB 6 subdisks: S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB S docsis.p1.s0 State: up D: vinumdrive3 Size: 613 MB S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB S docsisvar.p1.s0 State: up D: vinumdrive5 Size: 2270 MB See those referenced drives? It's going to be almost impossible to get rid of them without removing the subdisks and plexes. Then you're going to have to create new ones. All that is unnecessary, though. The whole point of "referenced" drives is so that you can add them later. Use this config file: drive vinumdrive0 device /dev/da1e drive vinumdrive2 device /dev/da1f drive vinumdrive4 device /dev/da1g With this, I get: vinum -> create /wantadilla/var/tmp/vinumconfig3-bp 6 drives: D vinumdrive1 State: up /dev/da0e A: 0/1023 MB (0%) D vinumdrive3 State: up /dev/da0f A: 0/614 MB (0%) D vinumdrive5 State: up /dev/da0g A: 0/2271 MB (0%) D vinumdrive0 State: up /dev/da1e A: 0/1023 MB (0%) D vinumdrive2 State: up /dev/da1f A: 0/614 MB (0%) D vinumdrive4 State: up /dev/da1g A: 0/2271 MB (0%) 3 volumes: V var State: up Plexes: 2 Size: 1023 MB V docsis State: up Plexes: 2 Size: 613 MB V docsisvar State: up Plexes: 2 Size: 2270 MB 6 plexes: P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB P var.p1 C State: up Subdisks: 1 Size: 1023 MB P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB P docsis.p1 C State: up Subdisks: 1 Size: 613 MB P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB P docsisvar.p1 C State: up Subdisks: 1 Size: 2270 MB 6 subdisks: S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB S docsis.p1.s0 State: up D: vinumdrive3 Size: 613 MB S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB S docsisvar.p1.s0 State: up D: vinumdrive5 Size: 2270 MB The subdisks are still down, of course. You need to start them: vinum -> start var.p0 Reviving var.p0.s0 in the background vinum[327]: reviving var.p0.s0 vinum -> l -r var V var State: up Plexes: 2 Size: 1023 MB P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB P var.p1 C State: up Subdisks: 1 Size: 1023 MB S var.p0.s0 State: R 14% D: vinumdrive0 Size: 1023 MB S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB That 14% by var.p0.s0 shows how far the revive has progressed. You'll have to wait, of course, and you shouldn't try to do the revives in parallel on a single spindle. That will greatly slow things down due to unnecessary seeks. With only one subdisk reviving at a time, iostat shows (on these old disks): tty ad0 da0 da1 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 230 4.00 2 0.01 64.00 43 2.68 64.00 43 2.68 0 0 1 2 98 0 76 0.00 0 0.00 64.00 44 2.72 64.00 45 2.78 0 0 2 1 98 If you try to do two at the same time, the aggregate data rate drops: tty ad0 da0 da1 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 229 0.00 0 0.00 64.00 35 2.17 64.00 35 2.17 0 0 0 2 98 0 76 0.00 0 0.00 64.00 35 2.17 64.00 35 2.17 0 0 0 2 98 When the revive completes, Vinum sets the states of the defective objects to "up". Greg -- See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message