Date: Sun, 27 Jan 2002 14:26:43 +1030 From: Greg Lehey <grog@FreeBSD.org> To: Martin Blapp <mb@imp.ch> Cc: freebsd-stable@freebsd.org Subject: Re: double fault with vinum and 4.5 RC3 Message-ID: <20020127142643.A19909@wantadilla.lemis.com> In-Reply-To: <20020124184810.M47234-100000@levais.imp.ch> References: <20020124184810.M47234-100000@levais.imp.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday, 24 January 2002 at 19:14:10 +0100, Martin Blapp wrote:
>
> ... unfortunatly I'm not able to recover from a simulated
> disk-crash. Here is what I made:
>
> Two disks: da0, da0, both 35GB
>
> 1) Installed FreeBSD on da0s1a
>
> 2) mount so single user mode
> dd bs=4096k if=/dev/da0 of=/dev/da1
>
> (to have the root on disk 2 too, I did this cold mirror)
>
> I plan to do a cold mirror procedure each evening, since
> the root will be mostly static anyway.
>
> mount /dev/da1a /mnt
> cd /mnt && dump -rf 0 /dev/da0a -f - | restore -xf -
> rm /mnt/restoresytable
> umount /mnt
>
> 3) I edited the disklabels and set da0e, da0f, da0g
> to type vinum
>
> # vinum mirror -n var /dev/da0e /dev/da1e
> # vinum mirror -n docsis /dev/da0f /dev/da1f
> # vinum mirror -n docsisvar /dev/da0g /dev/da1g
As I've mentioned elsewhere, this is seriously suboptimal. The
"mirror" command is a toy for people getting used to Vinum. You want
a proper config file. Then create one drive per spindle, and choose
your subdisk sizes to match what you want. Specifically, your config
file should look like this:
drive 0 device /dev/da0e
drive 1 device /dev/da1e
volume var setupstate
plex org concat
sd drive 0 len 2096000s
plex org concat
sd drive 1 len 2096000s
volume docsis setupstate
plex org concat
sd drive 0 len 1257000s
plex org concat
sd drive 1 len 1257000s
volume docsisvar setupstate
plex org concat
sd drive 0 len 4651000s
plex org concat
sd drive 1 len 4651000s
This will require repartitioning the disks, of course, and in the
process you'll regain a little additional space. Theoretically you
could repartition and use the existing data, but that is rather risky.
> But now I tried what happens if the disk 0 fails, I mean da0. I
> took da0 out and replaced it with a new disk. I switched jumpers, so
> da0 got da1, and da1 got da0 and rebooted.
I still don't know why you're switching jumpers. That's not needed.
But it doesn't change anything.
> I could boot fine, vinum failed to start and I was in single user
> mode. The I saw:
>
> # vinum start
> Warning: defective objects
>
> D vinumdrive0 State: referenced Device Avail: 0/0 MB
> D vinumdrive2 State: referenced Device Avail: 0/0 MB
> D vinumdrive4 State: referenced Device Avail: 0/0 MB
> P var.p0 C State: faulty Subdisks: 1 Size: 1023
> P docsis.p0 C State: faulty Subdisks: 1 Size: 6143
> P docsisvar.p0 C State: faulty Subdisks: 1 Size: 22
> S var.p0.s0 State: stale PO: 0 B Size: 1023
> S docsis.p0.s0 State: stale PO: 0 B Size: 6143
> S docsisvar.p0.s0 State: stale PO: 0 B Size: 22
>
> where is p1 ???
Not defective. I've repeated this configuration almost exactly (I
was using much smaller disks, so the size is different). What I got
was:
vinum -> start
Warning: defective objects
D vinumdrive0 State: referenced A: 0/0 MB
D vinumdrive2 State: referenced A: 0/0 MB
D vinumdrive4 State: referenced A: 0/0 MB
P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB
P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB
P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB
S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB
S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB
S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB
This is simply a list of the defective objects, as the warning message
states. To see them all, use the list (or l) command:
vinum -> l
3 drives:
D vinumdrive1 State: up /dev/da0e A: 0/1023 MB (0%)
D vinumdrive3 State: up /dev/da0f A: 0/614 MB (0%)
D vinumdrive5 State: up /dev/da0g A: 0/2271 MB (0%)
D vinumdrive0 State: referenced A: 0/0 MB
D vinumdrive2 State: referenced A: 0/0 MB
D vinumdrive4 State: referenced A: 0/0 MB
3 volumes:
V var State: up Plexes: 2 Size: 1023 MB
V docsis State: up Plexes: 2 Size: 613 MB
V docsisvar State: up Plexes: 2 Size: 2270 MB
6 plexes:
P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB
P var.p1 C State: up Subdisks: 1 Size: 1023 MB
P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB
P docsis.p1 C State: up Subdisks: 1 Size: 613 MB
P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB
P docsisvar.p1 C State: up Subdisks: 1 Size: 2270 MB
6 subdisks:
S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB
S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB
S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB
S docsis.p1.s0 State: up D: vinumdrive3 Size: 613 MB
S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB
S docsisvar.p1.s0 State: up D: vinumdrive5 Size: 2270 MB
> Then I tried to do:
>
> # echo "drive evar device /dev/da1e" >> configfile
> # echo "drive edocsis device /dev/da1f" >> configfile
> # echo "drive edocsisvar device /dev/da1g" >> configfile
> # vinum create configfile
>
> And boom I got a double fault.
Yes, we've found the cause of that. What I get now is:
vinum -> create /wantadilla/var/tmp/vinumconfig2-bp
6 drives:
D vinumdrive1 State: up /dev/da0e A: 0/1023 MB (0%)
D vinumdrive3 State: up /dev/da0f A: 0/614 MB (0%)
D vinumdrive5 State: up /dev/da0g A: 0/2271 MB (0%)
D vinumdrive0 State: referenced A: 0/0 MB
D vinumdrive2 State: referenced A: 0/0 MB
D vinumdrive4 State: referenced A: 0/0 MB
D evar State: up /dev/da1e A: 1023/1023 MB (100%)
D edocsis State: up /dev/da1f A: 614/614 MB (100%)
D edocsisvar State: up /dev/da1g A: 2271/2271 MB (100%)
3 volumes:
V var State: up Plexes: 2 Size: 1023 MB
V docsis State: up Plexes: 2 Size: 613 MB
V docsisvar State: up Plexes: 2 Size: 2270 MB
6 plexes:
P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB
P var.p1 C State: up Subdisks: 1 Size: 1023 MB
P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB
P docsis.p1 C State: up Subdisks: 1 Size: 613 MB
P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB
P docsisvar.p1 C State: up Subdisks: 1 Size: 2270 MB
6 subdisks:
S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB
S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB
S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB
S docsis.p1.s0 State: up D: vinumdrive3 Size: 613 MB
S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB
S docsisvar.p1.s0 State: up D: vinumdrive5 Size: 2270 MB
See those referenced drives? It's going to be almost impossible to
get rid of them without removing the subdisks and plexes. Then you're
going to have to create new ones. All that is unnecessary, though.
The whole point of "referenced" drives is so that you can add them
later. Use this config file:
drive vinumdrive0 device /dev/da1e
drive vinumdrive2 device /dev/da1f
drive vinumdrive4 device /dev/da1g
With this, I get:
vinum -> create /wantadilla/var/tmp/vinumconfig3-bp
6 drives:
D vinumdrive1 State: up /dev/da0e A: 0/1023 MB (0%)
D vinumdrive3 State: up /dev/da0f A: 0/614 MB (0%)
D vinumdrive5 State: up /dev/da0g A: 0/2271 MB (0%)
D vinumdrive0 State: up /dev/da1e A: 0/1023 MB (0%)
D vinumdrive2 State: up /dev/da1f A: 0/614 MB (0%)
D vinumdrive4 State: up /dev/da1g A: 0/2271 MB (0%)
3 volumes:
V var State: up Plexes: 2 Size: 1023 MB
V docsis State: up Plexes: 2 Size: 613 MB
V docsisvar State: up Plexes: 2 Size: 2270 MB
6 plexes:
P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB
P var.p1 C State: up Subdisks: 1 Size: 1023 MB
P docsis.p0 C State: faulty Subdisks: 1 Size: 613 MB
P docsis.p1 C State: up Subdisks: 1 Size: 613 MB
P docsisvar.p0 C State: faulty Subdisks: 1 Size: 2270 MB
P docsisvar.p1 C State: up Subdisks: 1 Size: 2270 MB
6 subdisks:
S var.p0.s0 State: stale D: vinumdrive0 Size: 1023 MB
S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB
S docsis.p0.s0 State: stale D: vinumdrive2 Size: 613 MB
S docsis.p1.s0 State: up D: vinumdrive3 Size: 613 MB
S docsisvar.p0.s0 State: stale D: vinumdrive4 Size: 2270 MB
S docsisvar.p1.s0 State: up D: vinumdrive5 Size: 2270 MB
The subdisks are still down, of course. You need to start them:
vinum -> start var.p0
Reviving var.p0.s0 in the background
vinum[327]: reviving var.p0.s0
vinum -> l -r var
V var State: up Plexes: 2 Size: 1023 MB
P var.p0 C State: faulty Subdisks: 1 Size: 1023 MB
P var.p1 C State: up Subdisks: 1 Size: 1023 MB
S var.p0.s0 State: R 14% D: vinumdrive0 Size: 1023 MB
S var.p1.s0 State: up D: vinumdrive1 Size: 1023 MB
That 14% by var.p0.s0 shows how far the revive has progressed. You'll
have to wait, of course, and you shouldn't try to do the revives in
parallel on a single spindle. That will greatly slow things down due
to unnecessary seeks. With only one subdisk reviving at a time,
iostat shows (on these old disks):
tty ad0 da0 da1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 230 4.00 2 0.01 64.00 43 2.68 64.00 43 2.68 0 0 1 2 98
0 76 0.00 0 0.00 64.00 44 2.72 64.00 45 2.78 0 0 2 1 98
If you try to do two at the same time, the aggregate data rate drops:
tty ad0 da0 da1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 229 0.00 0 0.00 64.00 35 2.17 64.00 35 2.17 0 0 0 2 98
0 76 0.00 0 0.00 64.00 35 2.17 64.00 35 2.17 0 0 0 2 98
When the revive completes, Vinum sets the states of the defective
objects to "up".
Greg
--
See complete headers for address and phone numbers
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020127142643.A19909>