Date: Fri, 19 Jul 1996 16:33:12 -0700 (PDT) From: asami@cs.berkeley.edu (Satoshi Asami) To: njensen@salsa.habaneros.com Cc: questions@freebsd.org Subject: Re: 2.1.5R and ccd questions Message-ID: <199607192333.QAA13479@baloon.mimi.com> In-Reply-To: <01BB74CA.1F813D80@jalapeno.habaneros.com> (njensen@salsa.habaneros.com)
next in thread | previous in thread | raw e-mail | index | archive | help
* Just finished installing 2.1.5R (went very smoothly, thanks
* Team FreeBSD!) and thought I'd try disk mirroring using
* ccd. I want to mirror the contents of my entire file system
* so that I can reboot from one of the disk if the other fails.
*
* I have read the documentation; ccd home page and man
* pages and am still unclear on exactly what I need to do.
*
* I have two idential SCSI drives, sd0 & sd1. Right now the OS
* is installed on sd0 and I have done nothing to sd1.
I'm afraid you are somewhat misunderstanding the way ccd mirroring
works. Despite the name, ccd actually concatenates disk partitions,
not disks themselves. The result of concatenation is a meta-device
that looks like a disk, which you can further partition up by editing
the disklabel.
As far as I can tell, there are two ways to accomplish what you want
to do. One is to mirror every partition you want as a separate ccd,
and the other is to first create a mega-partition that is the size of
everything except root & swap (more on this later) and divide this
thing up into pieces.
I'll give you an example on how to do the latter. The strategy here
is to first create a 1-disk ccd on sd1, partition it up, copy
everything you need onto it from sd0, then set up a 2-disk ccd with
mirroring using both disks.
Beware, I'm not sure at all if this will work, so make a backup of
your system before you try this!
* 1. recompile a new kernel with 'pseudo-device ccd 4' ?
Yes. The ccd lkm unfortunately doesn't work right now.
* 2. use dd to copy the contents of sd0 to sd1? (do I need
* to use disklabel,partition first?)
If we were just interested in copying things from one disk to another
identical one, I'd say "dd if=/dev/rsd0 of=/dev/rsd1" should probably
work, although I have never copied anything larger than a single
partition myself.
However, there is a little problem here: a filesystem on a regular
partition is not the same as that on a ccd on the same partition. The
ccd partition needs its own disklabel and such, so the start of the
filesystem is shifted back somewhat.
What I recommend would be like:
(1) dd if=/dev/rsd0 of=/dev/rsd1 bs=512 count=100
This should copy the fdisk table and disklabel to sd1. The number
"100" is arbirtary, I'm not sure how many is needed here.
(2) disklabel -e sd1
Edit the disklabel, subtract the size of "c" from the offset of "e"
(or "d" or whatever the first partition after "c" is), and use that as
the size of the new "e" partition.
For instance, if your partition table looked like this:
7 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 65536 0 4.2BSD 1024 8192 0 # (Cyl. 0 - 4*)
b: 262144 65536 swap # (Cyl. 4*- 20*)
c: 4199697 0 unused 0 0 # (Cyl. 0 - 261*)
e: 32768 327680 4.2BSD 1024 8192 0 # (Cyl. 20*- 22*)
f: 196608 360448 4.2BSD 1024 8192 0 # (Cyl. 22*- 34*)
g: 3642641 557056 4.2BSD 1024 8192 0 # (Cyl. 34*- 261*)
You should change it to:
5 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 65536 0 4.2BSD 1024 8192 0 # (Cyl. 0 - 4*)
b: 262144 65536 swap # (Cyl. 4*- 20*)
c: 4199697 0 unused 0 0 # (Cyl. 0 - 261*)
e: 3872017 327680 4.2BSD 1024 8192 0 # (Cyl. 20*- 22*)
^^^^^^^
= 4199697 - 327680
(The number of partitions is actually the index of the last
partition.) Also, don't worry about the cylinder numbers, they are
comments and disklabel will fix them up for you.
* 3. create the /etc/ccd.conf file with CCDF_MIRROR flag?
Yes. This is what it eventually should look like:
ccd0 128 CCDF_MIRROR /dev/sd0e /dev/sd1e
^^^
The interleave size shouldn't really matter, we are mirroring only two
disks....
* 4. now run ccdconfig, creating device ccd0?
ccdconfig won't create the devices. Use /dev/MAKEDEV to create them.
"sh ./MAKEDEV ccd0" will create /dev/ccd0* and /dev/rccd0*, etc.
ccdconfig is automatically run from /etc/rc if you have a
/etc/ccd.conf file on your system.
Now, this is what you need to do now. Remember we need to copy
everything onto the 1-disk ccd on sd1 first.
(1) create /etc/ccd.conf with the entry
ccd0 128 0 /dev/sd1e
(2) create a 1-disk ccd
ccdconfig -Cv
(3) edit the disklabel
disklabel -wr ccd0 auto
disklabel -e ccd0
Now, remember how the disklabel looked like before?
# size offset fstype [fsize bsize bps/cpg]
...
e: 32768 327680 4.2BSD 1024 8192 0 # (Cyl. 20*- 22*)
f: 196608 360448 4.2BSD 1024 8192 0 # (Cyl. 22*- 34*)
g: 3642641 557056 4.2BSD 1024 8192 0 # (Cyl. 34*- 261*)
Take these numbers, subtract "327680" from all the offsets, and write
that as the new disklabel:
# size offset fstype [fsize bsize bps/cpg]
...
e: 32768 0 4.2BSD 1024 8192 0 # (Cyl. 20*- 22*)
f: 196608 32768 4.2BSD 1024 8192 0 # (Cyl. 22*- 34*)
g: 3642641 229376 4.2BSD 1024 8192 0 # (Cyl. 34*- 261*)
You will also need to decrement the size of the last partition to make
the whole thing fit into the size of "c". I'm not exactly sure what
it's going to be, it will be a few dozen sectors less than the 3872017
we gave as the size of the whole thing. So, make sure that size(g) +
offset(g) = size(c).
(4) Create filesystems
newfs /dev/ccd0e
newfs /dev/ccd0f
newfs /dev/ccd0g
(5) Mount them
Assuming these three partitions are mounted as /var, /usr and /a on sd0:
mkdir /mnt/var /mnt/usr /mnt/a
mount -o async /dev/ccd0e /mnt/var
mount -o async /dev/ccd0f /mnt/usr
mount -o async /dev/ccd0g /mnt/a
The "-o async" mounts the drives asynchronously, which is dangerous
for normal use but should be ok for this situation. It's much faster.
(6) Copy the stuff over
cd /var; find -x . | cpio -pmV /mnt/var
cd /usr; find -x . | cpio -pmV /mnt/usr
cd /a; find -x . | cpio -pmV /mnt/a
* 5. is ccd0 now my device with the original filesystem installed,
* except now I am mirrored?
Not yet. :) Now we need to boot with root/swap from sd0 and the rest from
ccd0 (sd1), and then copy the stuff back to sd0. It will be like
this:
(1) vi /etc/fstab
Change "/dev/sd0e" to "/dev/ccd0e", etc.
Don't touch the references to sd0a and sd0b! Actually, you can add a
new line so that it says
/dev/sd0b none swap sw 0 0
/dev/sd1b none swap sw 0 0
if you want to use the "b" partition of both disks as swap.
(2) reboot
The system should now come up with something like this:
% mount
/dev/sd0a on / (local)
/dev/ccd0e on /var (local)
/dev/ccd0f on /usr (local)
/dev/ccd0g on /a (local)
procfs on /proc (local)
In addition, "pstat -s" will show both sd0b and sd1b if you added sd1b
as swap.
(3) edit sd0's disklabel
Do a "disklabel -e sd0" and make it look identical to sd1. Now it
should have sd0a, sd0b, sd0c (the whole disk) and one big partition
(sd0e in our example).
(4) copy stuff over to sd0
dd if=/dev/rsd1e of=/dev/rsd0e bs=65536
(5) vi /etc/ccd.conf
Change it to
ccd0 128 CCDF_MIRROR /dev/sd0e /dev/sd1e
(6) reboot
Pray it works. ;)
* 6. by default, I think the system will boot from sd0. If sd0 fails,
* how do I get the system to boot from sd1, will it automatically
* check for a valid system there?
Unfortunately, I don't think you can boot from a concatenated disk.
The ccdconfig utility is run from /etc/rc, after the root partition is
mounted (read-only), so any filesystem other than root should work
fine. (You can even swap to a ccd if you want, the "swapon -a" is
after ccdconfig in /etc/rc. ;)
Of course you probably want to keep a copy of the root partition as
well, so maybe you can run a find -x | cpio from cron or something.
I think the easiest way to be able to boot from sd1 when sd0 goes bad
would be to install the boot manager. That way you can select "disk
2" to boot from the second disk easily. Assuming you have maintained
a copy of sd0a in sd1a, you should be able to boot from there. Of
course, if sd0 goes totally bad and you have to take it out, sd1 will
now become sd0 so it will boot from there. ;)
Beware, ccd doesn't automatically check if a disk is valid, so you
will have to boot the system with -s and edit the ccd.conf file to use
only one disk if one of the two goes bad. Also, if you did the find
-x | cpio from cron, fstab will say root is sd0a so if the (bad) sd0
is still in the system and you are booting from sd1, you have to
change it to sd1a (and take out the sd0b reference as well).
* 7. how do I test if ccd is working?
Write something to /usr/tmp, and see lights on both disks flash. :)
* Thanks in advance for your help. I will compile any responses I
* get for possible inclusion in the handbook.
Well, I'm not sure if this was the kind of answer you wanted, but this
is what I *think* should work. Again, no guarantees, and if you don't
understand the phrase "boot the system with -s", you may not want to
go this far. ;)
Satoshi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199607192333.QAA13479>