Date: Mon, 29 Oct 2007 13:23:15 -0400 (EDT) From: "Kevin Thompson" <antiduh@csh.rit.edu> To: "Felipe Neuwald" <felipe@neuwald.biz> Cc: freebsd-geom@freebsd.org Subject: Re: Raid 0 + 1 Message-ID: <4414.147.177.192.113.1193678595.squirrel@angst.csh.rit.edu>
next in thread | raw e-mail | index | archive | help
> Hi Folks,
>
> I talked with my customer, and we decided to implement a 0 + 1 RAID,
> with 4 disks of 250Gb each.
>
> Here is how my RAID is working now:
I have some rough instructions I wrote up about doing a bootable 0+1 Geom
RAID, in wiki format. I don't have a public place to post them, so I've
just copied them below. Feedback is welcome.
Once you get it setup, I would highly recommend that you experiment with
the setup before relying on it - make sure you know how to handle failed
disk replacements, etc. VMWare/camcontrol/atacontrol is very handy in this
regard.
--Kevin Thompson
==Introduction==
This article describes one method to build a RAID 0+1 array with 4 logical
disks using the FreeBSD GEOM framework, and being able to boot off of such
an array.
Building a GEOM mirror and being able to directly boot off of it is a
relatively simple task - this is due to the fact that when mirroring is
done, all FreeBSD slices maintain their disk-level structure - GEOM stores
its metadata at the end of the disk/partition/etc. Most importantly, in
such a configuration the MBR and boot file system is untouched.
The latter is not the case in RAID 0, RAID 0+1, or RAID 1+0 setups.
In striping configurations, the boot filesystem is interleaved between the
various disks (usually two), and as such, the filesystem appears to be
corrupt if read from only one disk.
Since the MBR program on the boot disk is incapable of understanding this
physical block layout, it is unable to find the kernel in order to start
FreeBSD.
However, should the MBR program be able to read the boot filesystem, it
can then load the kernel and related kernel modules, which would then
allow the system to 'boot' from the raidset.
Technically speaking, the only file system that has to remain untouched is
the /boot file system. root (/), /usr, /var, /tmp, et cetera may all then
be mounted from a raidset.
In this example, for simplicity's sake, I protect all of the typical root
file system, not just /boot. A more enterprising user may want to modify
their approach.
==Instructions==
Boot with a FreeBSD install disk, then start the fixit console. Next
you'll need to clean up the environment (the fixit console needs a little
updating)
ln -s /dist/boot/kernel /boot/kernel
ln -s /dist/lib /lib
EDITOR=/mnt2/usr/bin/vi; export EDITOR
PATH=$PATH':'/mnt2/sbin':'/mnt2/usr/bin':'/mnt2/usr/sbin
export PATH
Now load the kernel modules for the geom modules we're going to be using.
glabel load
gstripe load
gmirror load
Next, we're going to label each of the drives. The 'da0' drive name is
typically assigned by the bios on boot, by way of some sort of metric such
as chain location. The geom label module writes a fixed label to each
drive, so that no matter where each drive goes, or if new devices are
inserted and the numbering is reordered, the raid set will always work the
same.
glabel label geom0 da0
glabel label geom1 da1
glabel label geom2 da2
glabel label geom3 da3
Now install a mbr and basic partioning on each drive. This will create a
single partition taking the entire drive for each drive.
fdisk -vBI /dev/label/geom0
fdisk -vBI /dev/label/geom1
fdisk -vBI /dev/label/geom2
fdisk -vBI /dev/label/geom3
On the first disk slice of the first drive, install a simple disk label
and bootstrap code.
bsdlabel -wB /dev/label/geom0s1
Now edit the generic label on that disk, setting it as you please. 'a' is
commonly root, 'b' the swap partition and 'd' the rest. Don't create any
more partitions other than a, b and d (d will be used as the provider for
a future geom consumer).
slave(/u9/antiduh) # bsdlabel -e /dev/label/geom0s1
# size offset fstype [fsize bsize bps/cpg]
a: 500M 16 4.2BSD
b: 500M * swap
c: # leave as is
d: * * 4.2BSD
The '*' for size means use whatever is left, and the '*' for offset means
use the next logical offset.
Once you're finished labeling the first drive, write the label for the
drive out to a file, then use it to initialize the other three disks:
bsdlabel /dev/label/geom0s1 > /file
bsdlabel -R /dev/label/geom1s1 /file
bsdlabel -R /dev/label/geom2s1 /file
bsdlabel -R /dev/label/geom3s1 /file
Now, create a GEOM mirror out of the 'a' partition of each drive. This
will eventually be the root partition. The new device will be called
'''boot''', and will enumerate in FreeBSD as '''/dev/mirror/boot'''
gmirror label -vh boot /dev/label/geom0s1a /dev/label/geom1s1a
/dev/label/geom2s1a /dev/label/geom3s1a
Now we're going to pull together the d partion on each drive, create
pairwise stripes, then mirror the new stripes together.
gstripe label -vh -s 131072 st0 /dev/label/geom0s1d /dev/geom1s1d
gstripe label -vh -s 131072 st1 /dev/label/geom2s1d /dev/geom3s1d
We should now have two new devices '''/dev/stripe/st0''' and
'''/dev/stripe/st1'''. Now mirror those two devices to create our final
device that will next be used for the rest of our filesystems:
gmirror label -vh gm0 /dev/stripe/st0 /dev/stripe/st1
We now have our final device '''/dev/mirror/gm0''' that is going to serve
as the base for our regular filesystems.
Create a basic slicing of the new disk, then edit to taste. Note that we
don't fdisk the raw gm0 device - we create slices directly on the raw
device.
slave(/u9/antiduh) # bsdlabel -wB /dev/mirror/gm0
slave(/u9/antiduh) # bsdlabel -e /dev/mirror/gm0
# size offset fstype [fsize bsize bps/cpg]
a: 1 16 unused # that is just a bare '1', not '1M'. This is to get
around a bug in bsdlabel
c: #-- leave as is --
e: 1000M * 4.2BSD # /var
f: 500M * 4.2BSD # /tmp
d: * * 4.2BSD # /usr, which just gets the rest
Now create our filesystems on associated slices (the -U option enables
soft updates)
newfs /dev/mirror/boot
newfs -U /dev/mirror/gm0d
newfs -U /dev/mirror/gm0e
newfs -U /dev/mirror/gm0f
Now we finally get down to mounting the disks, installing the OS on it,
setting up the OS to boot, and finally booting.
Mount the root disk as /mnt
mount /dev/mirror/boot /mnt
Create mount points for our other file systems, then mount them:
mkdir /mnt/usr
mkdir /mnt/var
mkdir /mnt/tmp
mount /dev/mirror/gm0d /mnt/usr
mount /dev/mirror/gm0e /mnt/var
mount /dev/mirror/gm0f /mnt/tmp
Now we're going to do a install the fun way:
DESTDIR=/mnt
export DESTDIR
cd /dist/6.2-RELEASE/base; ./install.sh
cd ../ports; ./install.sh
cd ../manpages; ./install.sh
cd ../kernels; ./install.sh GENERIC
mv /mnt/boot/GENERIC/* /mnt/boot/kernel
You now have a basic, blank, sorta bootable unconfigured FreeBSD install
on the machine.
Setup the boot loader to load the needed kernel modules, so that we can
mount root from the mirror.
echo 'geom_label_load="YES"' >> /mnt/boot/loader.conf
echo 'geom_stripe_load="YES"' >> /mnt/boot/loader.conf
echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf
Set a kernel option to use more memory but make the stripe layer faster,
otherwise stripes are unbearably slow.
echo 'kern.geom.stripe.fast=1"' >> /mnt/boot/loader.conf
Now setup our initial fstab file:
echo '/dev/label/geom0s1b none swap sw 0 0' >> /mnt/etc/fstab
echo '/dev/label/geom1s1b none swap sw 0 0' >> /mnt/etc/fstab
echo '/dev/label/geom2s1b none swap sw 0 0' >> /mnt/etc/fstab
echo '/dev/label/geom3s1b none swap sw 0 0' >> /mnt/etc/fstab
echo '/dev/mirror/boot / ufs rw 1 1' >> /mnt/etc/fstab
echo '/dev/mirror/gm0d /usr ufs rw 2 2' >> /mnt/etc/fstab
echo '/dev/mirror/gm0e /var ufs rw 2 2' >> /mnt/etc/fstab
echo '/dev/mirror/gm0f /tmp ufs rw 2 2' >> /mnt/etc/fstab
Reboot and the machine should start the raid sets automatically and mount
off of the raid array automatically. Keep that install CD handy in case
you messed up.
Login, change the root password, setup rc.conf (hostname, interfaces, ssh,
linux binary compat... ), start installing stuff, etc.
Enjoy.
==Block Diagram==
<code>
da0 --label-->geom0
da1 --label-->geom1
da2 --label-->geom2
da3 --label-->geom3
label/geom0s1a --|--mirror-->/dev/mirror/boot
label/geom1s1a --|
label/geom2s1a --|
label/geom3s1a --|
label/geom0s1d --|--stripe-->/dev/stripe/st0 --|--mirror-->/dev/mirror/gm0
label/geom1s1d --| |
|
label/geom2s1d --|--stripe-->/dev/stripe/st1 --|
label/geom3s1d --|
mirror/boot --> /
mirror/gm0s1d --> /usr
mirror/gm0s1e --> /var
mirror/gm0s1f --> /tmp
label/geom0s1b --> swap
label/geom1s1b --> swap
label/geom2s1b --> swap
label/geom3s1b --> swap
</code>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4414.147.177.192.113.1193678595.squirrel>