Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 19 Jan 2005 22:57:21 -0800
From:      "Ted Mittelstaedt" <tedm@toybox.placo.com>
To:        "Drew Tomlinson" <drew@mykitchentable.net>, "FreeBSD Questions" <freebsd-questions@freebsd.org>
Subject:   RE: One Last Plea For Vinum Assistance
Message-ID:  <LOBBIFDAGNMAMLGJJCKNAEBEFAAA.tedm@toybox.placo.com>
In-Reply-To: <41ED86F4.1080301@mykitchentable.net>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi Drew,

  Please read the following:

http://www.vinumvm.org/vinum/how-to-debug.html

  And follow the instructions exactly.  And I mean exactly.
Also keep the following in mind, Greg will try to help but
note carefully the sentence on this webpage:

"Since I wrote it, FreeBSD has changed its I/O structure, breaking many
things in Vinum. At the time of writing, a new version, provisionally
called gvinum, is being written"

  I myself have had one serious crash on a vinum RAID volume as
a result of a SCSI cable problem that blew away the volume.  (2
drives were corrupted, instead of just one, making it impossible
for the volume manager to repair by itself)  I sent all the info
to Greg but ultimately he wasn't able to offer any suggestions
on recovering the array so I just wiped it and started over.

Note that Greg DID NOT recommend wiping the array.  In fact he
didn't recommend anything.  The lack of any recommendation
appears to be his way of telling you "your volume is screwed,
wipe it and start over"  Like most UNIX commands, if Greg
has nothing to offer, he says nothing at all, he won't tell you he
has nothing to offer.  So, the lack of a response to your
original post you can probably take as an answer, to be honest.

  This did teach me a lesson that I kind of knew already but
didn't think too much about.  That is, a software array is no substitute
for a hardware array.  In other words, vinum is a great thing
if what your wanting to do is use a bunch of cheap disks and
cheap controller cards to either get a giant partition, or to
stripe them together and get faster access.  But it's not so
good if the intent is to get some crash recovery.

  I don't use and have never used vinum for /etc, /, /usr, /var
or any other system partitions.  I only use it for partitions
that I want to mount AFTER the system is booted.  If I were in
your shoes I'd nuke your system and start all over again and
rethink how I had it laid out.  I would use a single disk for
the system then take the rest of the disks and put them together
under vinum.  Then I'd mount that on /ftp and I'd softlink
whatever thing is gopping up space under /usr, for example
/usr/local/www, to a directory under /ftp

Vinum isn't going to give you any crash recovery
for /usr so there is really no point in making /usr a vinum
volume.  Beyond that I really don't understand why you
are putting /usr as a vinum volume, espically as you yourself
said "Fortunately this volume is up and running or I would
really be in a mess"  I mean, your basically saying your
hitting yourself in the face and you feel fortunate you
haven't broken your nose yet.

Anyway, one other thing I will bring up:  How exactly did
you update your system?  Did you nuke and repave it?  Or did
you follow the instructions here EXACTLY:

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html

If you didn't do one or the other of these things then nobody is going
to help you.

Ted

> -----Original Message-----
> From: owner-freebsd-questions@freebsd.org
> [mailto:owner-freebsd-questions@freebsd.org]On Behalf Of Drew Tomlinson
> Sent: Tuesday, January 18, 2005 2:00 PM
> To: FreeBSD Questions
> Subject: One Last Plea For Vinum Assistance
>
>
> I sent the message below a couple of times but did not receive
> any response.
> I assume that it's because either I have a really difficult
> problem or am
> asking something really stupid.  :)  But anyway, I want to install
> additional
> memory in this machine and am sure I will run across the same problems
> after
> shutting down.  So if anyone has any suggestions on how I
> might solve this
> issue, I'd really appreciate the input.
>
> Thanks,
>
> Drew
>
> --- Original Message ---
> Since an upgrade from 4.9 to 4.10, I've had problems with
> vinum.  The basic
> problem is that upon reboot, two of my vinum drives show up as
> "referenced" and
> thus create the associated chaos.  I've tried many things and fiddled
> around
> quite a bit so I can't say exactly what I've done.  I can
> include all of
> the
> entries in the history file since Oct. 31 if that's a help but
> it would
> be a
> long list.
>
> So prior to digging that deep, I will describe where I stand
> currently and
> where I want to finish.  Currently, I have one vinum volume
> that I use for
> /usr. Fortunately this volume is up and running or I would
> really be in a
> mess. Here's the 'vinum list' output in this state:
>
> blacklamb# vinum
> vinum -> list
> 2 drives:
> D disk1                 State: up       Device /dev/da0s1h      Avail:
> 0/8383 MB (0%)
> D disk2                 State: up       Device /dev/da1s1h      Avail:
> 0/8383 MB (0%)
>
> 1 volumes:
> V usr                   State: up       Plexes:       1 Size:
>        16 GB
>
> 1 plexes:
> P usr.p0              S State: up       Subdisks:     2 Size:
>        16 GB
>
> 2 subdisks:
> S usr.p0.s0             State: up       PO:        0  B Size:
>      8383 MB
> S usr.p0.s1             State: up       PO:      256 kB Size:
>      8383 MB
>
> I want to add another volume and mount it on /ftp.  After creating the
> volume,
> vinum sees it and it appears OK as indicated in this output:
>
> vinum -> list
> 5 drives:
> D disk1                 State: up       Device /dev/da0s1h      Avail:
> 0/8383 MB (0%)
> D disk2                 State: up       Device /dev/da1s1h      Avail:
> 0/8383 MB (0%)
> D ftp1                  State: up       Device /dev/ad0s1h      Avail:
> 0/76319 MB (0%)
> D ftp2                  State: up       Device /dev/ad1s1h      Avail:
> 0/76319 MB (0%)
> D ftp3                  State: up       Device /dev/da3s1h      Avail:
> 0/114473 MB (0%)
>
> 2 volumes:
> V usr                   State: up       Plexes:       1 Size:
>        16 GB
> V ftp                   State: up       Plexes:       1 Size:
>       260 GB
>
> 2 plexes:
> P usr.p0              S State: up       Subdisks:     2 Size:
>        16 GB
> P ftp.p0              C State: up       Subdisks:     3 Size:
>       260 GB
>
> 5 subdisks:
> S usr.p0.s0             State: up       PO:        0  B Size:
>      8383 MB
> S usr.p0.s1             State: up       PO:      256 kB Size:
>      8383 MB
> S ftp.p0.s0             State: up       PO:        0  B Size:
>        74 GB
> S ftp.p0.s1             State: up       PO:       74 GB Size:
>        74 GB
> S ftp.p0.s2             State: up       PO:      149 GB Size:
>       111 GB
>
> Next I 'quit' the vinum command line and fsck my newly created
> volume.  It
> finishes successfully and I mount it:
>
> blacklamb# df
> Filesystem     1K-blocks      Used     Avail Capacity  Mounted on
> /dev/da0s1a       302350    102088    176074    37%    /
> /dev/vinum/usr  16639674   5433762   9874739    35%    /usr
> procfs                 4         4         0   100%    /proc
> /dev/vinum/ftp 265119539 119729707 132133856    48%    /ftp
>
> I recheck with the 'vinum list' command and the output is the same as
> above.
> Now I cross my fingers and reboot to see if the volume comes
> up properly on
> startup.  However ftp3 (/dev/da3s1h) comes up 'referenced' and causes
> problems.
>
> Mounting root from ufs:/dev/da0s1a
> dumpon: crash dumps to /dev/da1s1b (13, 131081)
> vinum: loaded
> vinum: reading configuration from /dev/ad1s1h
> vinum: ftp.p0.s2 is crashed
> vinum: ftp.p0 is corrupt
> vinum: updating configuration from /dev/ad0s1h
> vinum: updating configuration from /dev/da1s1h
> vinum: updating configuration from /dev/da0s1h
> vinum: /dev is mounted read-only, not rebuilding /dev/vinum
> Warning: defective objects
>
> D ftp3                  State: referenced       Device  Avail: 0/0 MB
> P ftp.p0              C State: corrupt  Subdisks:     3 Size:
>       260 GB
> S ftp.p0.s2             State: crashed  PO:      149 GB Size:
>       111 GB
> swapon: adding /dev/da1s1b as swap device
> Automatic boot in progress...
> /dev/da0s1a: FILESYSTEM CLEAN; SKIPPING CHECKS
> /dev/da0s1a: clean, 100130 free (938 frags, 12399 blocks, 0.6%
> fragmentation)
> /dev/vinum/ftp: CANNOT READ: BLK 546979872
> /dev/vinum/ftp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
> /dev/vinum/ftp: CANNOT WRITE: BLK 2800
> /dev/vinum/ftp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
> /dev/vinum/usr: FILESYSTEM CLEAN; SKIPPING CHECKS
> /dev/vinum/usr: clean, 11205903 free (152463 frags, 1381680
> blocks, 0.9%
> fragmentation)
> THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
>      /dev/vinum/ftp (/ftp)
> File system preen failed, trying fsck -y . . .
> ** /dev/da0s1a
> ** Last Mounted on /
> ** Root file system
> ** Phase 1 - Check Blocks and Sizes
> ** Phase 2 - Check Pathnames
> ** Phase 3 - Check Connectivity
> ** Phase 4 - Check Reference Counts
> ** Phase 5 - Check Cyl groups
> 2053 files, 51045 used, 100130 free (938 frags, 12399 blocks, 0.6%
> fragmentation)
> ** /dev/vinum/usr
> ** Last Mounted on /usr
> ** Phase 1 - Check Blocks and Sizes
> ** Phase 2 - Check Pathnames
> ** Phase 3 - Check Connectivity
> ** Phase 4 - Check Reference Counts
> ** Phase 5 - Check Cyl groups
> 401072 files, 5433771 used, 11205903 free (152463 frags,
> 1381680 blocks,
> 0.9% fragmentation)
> ** /dev/vinum/ftp
>
> CANNOT READ: BLK 546979872
> UNEXPECTED SOFT UPDATE INCONSISTENCY
>
> CONTINUE? yes
>
> THE FOLLOWING DISK SECTORS COULD NOT BE READ: 546979872, 546979873,
> 546979874, 546979875,
> /dev/vinum/ftp: CANNOT FIGURE OUT FILE SYSTEM PARTITION
>
> UNEXPECTED SOFT UPDATE INCONSISTENCY
> WARNING: R/W mount of /ftp denied.  Filesystem is not clean - run fsck
> mount: /dev/vinum/ftp: Operation not permitted
> Mounting /etc/fstab filesystems failed, startup aborted
> Enter full pathname of shell or RETURN for /bin/sh:
>
> I enter the shell and run 'vinum list':
>
> vinum -> list
> 4 drives:
> D ftp1                  State: up       Device /dev/ad0s1h      Avail:
> 0/76319 MB (0%)
> D ftp2                  State: up       Device /dev/ad1s1h      Avail:
> 0/76319 MB (0%)
> D disk1                 State: up       Device /dev/da0s1h      Avail:
> 0/8383 MB (0%)
> D disk2                 State: up       Device /dev/da1s1h      Avail:
> 0/8383 MB (0%)
> D ftp3                  State: referenced       Device  Avail: 0/0 MB
>
> 2 volumes:
> V usr                   State: up       Plexes:       1 Size:
>        16 GB
> V ftp                   State: up       Plexes:       1 Size:
>       260 GB
>
> 2 plexes:
> P usr.p0              S State: up       Subdisks:     2 Size:
>        16 GB
> P ftp.p0              C State: corrupt  Subdisks:     3 Size:
>       260 GB
>
> 5 subdisks:
> S usr.p0.s0             State: up       PO:        0  B Size:
>      8383 MB
> S usr.p0.s1             State: up       PO:      256 kB Size:
>      8383 MB
> S ftp.p0.s0             State: up       PO:        0  B Size:
>        74 GB
> S ftp.p0.s1             State: up       PO:       74 GB Size:
>        74 GB
> S ftp.p0.s2             State: crashed  PO:      149 GB Size:
>       111 GB
>
> Now I delete all references to the ftp volume by running these
> commands:
>
> vinum -> rm -f ftp.p0.s2
> vinum: removing ftp.p0.s2
> Correcting length of ftp.p0: was 547043829, is 312602446
> vinum: ftp.p0 is up
> vinum -> rm -f ftp.p0.s1
> vinum: removing ftp.p0.s1
> Correcting length of ftp.p0: was 312602446, is 156301223
> vinum -> rm -f ftp.p0.s0
> vinum: removing ftp.p0.s0
> vinum -> rm -f ftp.p0  vinum: removing ftp.p0
> vinum: ftp is down
> vinum -> rm -f ftp  vinum: removing ftp
> vinum -> rm -f ftp3
> vinum -> rm -f ftp2
> vinum -> rm -f ftp1
>
> Yet the 'reference to ftp3 remains.
>
> vinum -> list
> 1 drives:
> D disk1                 State: up       Device /dev/da0s1h      Avail:
> 0/8383 MB (0%)
> D disk2                 State: up       Device /dev/da1s1h      Avail:
> 0/8383 MB (0%)
> D ftp3                  State: referenced       Device  Avail: 0/0 MB
>
> 1 volumes:
> V usr                   State: up       Plexes:       1 Size:
>        16 GB
>
> 1 plexes:
> P usr.p0              S State: up       Subdisks:     2 Size:
>        16 GB
>
> 2 subdisks:
> S usr.p0.s0             State: up       PO:        0  B Size:
>      8383 MB
> S usr.p0.s1             State: up       PO:      256 kB Size:
>      8383 MB
>
> Why does the list output show "1 drives:" when there are three listed
> and two
> working?  Anyway, I've been here before and I quit vinum.
> Then I issue
> these
> commands:
>
> # df
> Filesystem     512-blocks     Used    Avail Capacity  Mounted on
> /dev/da0s1a        604700   204180   352144    37%    /
> /dev/vinum/usr   33279348 10867544 19749458    35%    /usr
> procfs                  8        8        0   100%    /proc
> # umount /usr
> # vinum stop
> vinum: unloaded
> vinum unloaded
> # vinum start
> vinum: loaded
> vinum: reading configuration from /dev/da1s1h
> vinum: updating configuration from /dev/da0s1h
>
> Then I issue a 'vinum list' and all seems well.
>
> # vinum
> vinum -> list
> 2 drives:
> D disk1                 State: up       Device /dev/da0s1h      Avail:
> 0/8383 MB (0%)
> D disk2                 State: up       Device /dev/da1s1h      Avail:
> 0/8383 MB (0%)
>
> 1 volumes:
> V usr                   State: up       Plexes:       1 Size:
>        16 GB
>
> 1 plexes:
> P usr.p0              S State: up       Subdisks:     2 Size:
>        16 GB
>
> 2 subdisks:
> S usr.p0.s0             State: up       PO:        0  B Size:
>      8383 MB
> S usr.p0.s1             State: up       PO:      256 kB Size:
>      8383 MB
> vinum ->
>
> Since everything appears correct, this seems like a good place
> to do 'vinum
> saveconfig'.  Next I edit /etc/fstab so only /ftp is not mounted and
> reboot.
> The system reboots fine and the 'vinum list' is as it is
> immediately above.
>
>
> Running in full production mode, I create the ftp volume again:
>
> vinum -> create -f vinum_ftp.conf
> Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp1 is up
> Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp2 is up
> Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp3 is up
> 5 drives:
> D disk1                 State: up       Device /dev/da0s1h      Avail:
> 0/8383 MB (0%)
> D disk2                 State: up       Device /dev/da1s1h      Avail:
> 0/8383 MB (0%)
> D ftp1                  State: up       Device /dev/ad0s1h      Avail:
> 0/76319 MB (0%)
> D ftp2                  State: up       Device /dev/ad1s1h      Avail:
> 0/76319 MB (0%)
> D ftp3                  State: up       Device /dev/da3s1h      Avail:
> 0/114473 MB (0%)
>
> 2 volumes:
> V usr                   State: up       Plexes:       1 Size:
>        16 GB
> V ftp                   State: up       Plexes:       1 Size:
>       260 GB
>
> 2 plexes:
> P usr.p0              S State: up       Subdisks:     2 Size:
>        16 GB
> P ftp.p0              C State: up       Subdisks:     3 Size:
>       260 GB
>
> 5 subdisks:
> S usr.p0.s0             State: up       PO:        0  B Size:
>      8383 MB
> S usr.p0.s1             State: up       PO:      256 kB Size:
>      8383 MB
> S ftp.p0.s0             State: up       PO:        0  B Size:
>        74 GB
> S ftp.p0.s1             State: up       PO:       74 GB Size:
>        74 GB
> S ftp.p0.s2             State: up       PO:      149 GB Size:
>       111 GB
> Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s0 is up
> Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s1 is up
> Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s2 is up
> Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0 is up
> Dec 24 16:37:45 blacklamb /kernel: vinum: ftp is up
>
> After running fsck, I can mount and access the volume.  The system
> continues to
> run fine until the next reboot where the same problems start all over
> again.  I
> also have the same problem with another volume I wish to run called
> "backup".
> How can I fix my vinum volumes so they survive reboots?
>
> Thanks for your help and Happy Holidays!
>
> Drew
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe@freebsd.org"
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?LOBBIFDAGNMAMLGJJCKNAEBEFAAA.tedm>