From owner-freebsd-geom@FreeBSD.ORG Sat Oct 8 01:35:26 2005 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3B84116A423 for ; Sat, 8 Oct 2005 01:35:26 +0000 (GMT) (envelope-from aw1@stade.co.uk) Received: from lon-mail-1.gradwell.net (lon-mail-1.gradwell.net [193.111.201.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6DE5643D46 for ; Sat, 8 Oct 2005 01:35:25 +0000 (GMT) (envelope-from aw1@stade.co.uk) Received: from alsager-adsl.stade.co.uk ([81.6.222.119] helo=access2.hanley.stade.co.uk) by lon-mail-1.gradwell.net with esmtp (Gradwell gwh-smtpd 1.190) id 4347225b.41f.1b9c for freebsd-geom@freebsd.org; Sat, 8 Oct 2005 02:35:23 +0100 (envelope-sender ) Received: from titus.hanley.stade.co.uk (titus [192.168.1.5]) by access2.hanley.stade.co.uk (8.13.4/8.13.3) with ESMTP id j981ZN7i012000 for ; Sat, 8 Oct 2005 02:35:23 +0100 (BST) (envelope-from aw1@titus.hanley.stade.co.uk) Received: from titus.hanley.stade.co.uk (localhost [127.0.0.1]) by titus.hanley.stade.co.uk (8.12.11/8.12.11) with ESMTP id j981ZMQR056946 for ; Sat, 8 Oct 2005 02:35:22 +0100 (BST) (envelope-from aw1@titus.hanley.stade.co.uk) Received: (from aw1@localhost) by titus.hanley.stade.co.uk (8.12.11/8.12.11/Submit) id j981ZLxB056945 for freebsd-geom@freebsd.org; Sat, 8 Oct 2005 02:35:21 +0100 (BST) (envelope-from aw1) Date: Sat, 8 Oct 2005 02:35:21 +0100 From: Adrian Wontroba To: freebsd-geom@freebsd.org Message-ID: <20051008023521.D55180@titus.hanley.stade.co.uk> References: <20051006125233.754dd00e.josh@oplink.net> <86r7aywju7.fsf@xps.des.no> <20051006200802.GL564@ratchet.nebcorp.com> <20051006201616.GE26614@garage.freebsd.pl> <20051006202700.GM564@ratchet.nebcorp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <20051006202700.GM564@ratchet.nebcorp.com>; from dannyman@toldme.com on Thu, Oct 06, 2005 at 01:27:00PM -0700 X-Operating-System: FreeBSD 4.10-STABLE Organization: Oh dear, I've joined one again. X-Virus-Scanned: ClamAV 0.86.1/1121/Fri Oct 7 18:38:02 2005 on access2.hanley.stade.co.uk X-Virus-Status: Clean Subject: Re: GEOM's RAID level support X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: aw1@stade.co.uk List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Oct 2005 01:35:26 -0000 On Thu, Oct 06, 2005 at 01:27:00PM -0700, Danny Howard wrote: > Well, I have RAID1 systems ... I've just reworked one of my home systems, doubling its disk capacity. I started with a pretty standard gmirrored two disk system, added another Promise controller and another pair of (nearly) identical disks, setting up the new disks as: Mirrored root (gm0). Single plex mirror of swap (gm1) or Single plex mirror of /var and /usr (gm2) Rest of the space striped (gs0), underneath a single plex mirror of /home (gm3) When I'd got the new mirrors working, I manually amended /etc/fstab etc, booted from them and setup the other stripes/ plexes on the original disks, ending up with four plexes for /root and two for everything else. With the ATA write cache enabled it flies. With the write cache disabled it crawls. Must buy a UPS. The end result looks like: bash-2.05b# gmirror status Name Status Components mirror/gm0 COMPLETE ad4a ad6a ad8a ad10a mirror/gm1 COMPLETE ad4d ad8d mirror/gm2 COMPLETE ad6d ad10d mirror/gm3 COMPLETE stripe/gs1 stripe/gs0 bash-2.05b# gstripe status Name Status Components stripe/gs1 UP ad4e ad6e stripe/gs0 UP ad8e ad10e bash-2.05b# df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/mirror/gm0 1012972 56370 875566 6% / devfs 1 1 0 100% /dev /dev/mirror/gm2d 1012974 18558 913380 2% /var /dev/mirror/gm2e 3045004 1398126 1403278 50% /usr /dev/mirror/gm3 292588412 74542894 194638446 28% /home /dev/md0 1012974 284 931654 0% /tmp bash-2.05b# swapinfo Device 1K-blocks Used Avail Capacity /dev/mirror/gm1 4194300 148 4194152 0% I set the system up with a Makefile and some initial disk labels (below my sig) and lots of trial and error, executing make zap, make label_disks, etc. No doubt less than perfect, but it worked for me. At work, I've developed a makefile based version of RSE's "single disk to two disk" approach. If there is any interest, I'll ask my boss if I can release it. Is anybody doing anything about: Making systinstall GEOM aware? Putting RSE's excellent instructions in the handbook? -- Adrian Wontroba ==> Makefile <== clear: # stop and clear anything left over from last attempt @logger -p crit "clear" -gmirror stop -v gm3 -gmirror clear -v stripe/gs0 -gstripe stop -v gs0 -gstripe clear -v ad8e ad10e -gmirror stop -v gm2 -gmirror clear -v ad10d -gmirror stop -v gm1 -gmirror clear -v ad8d -gmirror stop -v gm0 -gmirror clear -v ad8a ad10a zap: # zap the front of the disks @logger -p crit "zap" dd if=/dev/zero of=/dev/ad8 count=100 dd if=/dev/zero of=/dev/ad10 count=100 label_disks: # label disks @logger -p crit "label disks" fdisk -v -B -I /dev/ad8 fdisk -v -B -I /dev/ad10 bsdlabel -w -B ad8 bsdlabel -w -B ad10 label_partitions: @logger -p crit "label slices" bsdlabel -R ad8 initial_ad8.label bsdlabel ad8 > ad8.label bsdlabel -R ad10 initial_ad10.label bsdlabel ad10 > ad10.label mirrors: # bring on the mirrors @logger -p crit "gm0" gmirror label -v -h -n -b round-robin gm0 ad8a ad10a @logger -p crit "gm1" gmirror label -v -h -n -b round-robin gm1 ad8d @logger -p crit "gm2" gmirror label -v -h -n -b round-robin gm2 ad10d @logger -p crit "gs0" gstripe label -v -h gs0 ad8e ad10e @logger -p crit "gm3" gmirror label -v -h -n -b round-robin gm3 stripe/gs0 label_mirrors: # label the mirrors which contain more than one filesystem # gm2 contains d: var and e: usr bsdlabel -w mirror/gm2 bsdlabel -R mirror/gm2 initial_gm2.label bsdlabel mirror/gm2 > gm2.label newfs: # newfs the filesystems # gm1 is swap, doesn't need newfs newfs -U /dev/mirror/gm0 | tee gm0.newfs newfs -U /dev/mirror/gm2d | tee gm2d.newfs newfs -U /dev/mirror/gm2e | tee gm2e.newfs newfs -U /dev/mirror/gm3 | tee gm3.newfs restore_root: # dump and restore root filesystem mount /dev/mirror/gm0 /mnt; dump -L -C 32 -0 -f- / | (cd /mnt; restore -r -f-) restore_rest: # dump and restore other filesystem mount /dev/mirror/gm2d /mnt/var; dump -L -C 32 -0 -f- /var | (cd /mnt/var; restore -r -f-) mount /dev/mirror/gm2e /mnt/usr; dump -L -C 32 -0 -f- /usr | (cd /mnt/usr; restore -r -f-) mount /dev/mirror/gm3 /mnt/home; dump -L -C 32 -0 -f- /home | (cd /mnt/home; restore -r -f-) zap1: # zap the front of the disks @logger -p crit "zap" dd if=/dev/zero of=/dev/ad4 count=100 dd if=/dev/zero of=/dev/ad6 count=100 label_disks1: # label disks @logger -p crit "label disks" fdisk -v -B -I /dev/ad4 fdisk -v -B -I /dev/ad6 bsdlabel -w -B ad4 bsdlabel -w -B ad6 label_partitions1: @logger -p crit "label slices" bsdlabel -R ad4 initial_ad4.label bsdlabel ad4 > ad4.label bsdlabel -R ad6 initial_ad6.label bsdlabel ad6 > ad6.label mirrors1: # bring on the mirror plexes @logger -p crit "gm0" gmirror insert -v -h gm0 ad4a ad6a @logger -p crit "gm1" gmirror insert -v -h gm1 ad4d @logger -p crit "gm2" gmirror insert -v -h gm2 ad6d @logger -p crit "gs1" gstripe label -v -h gs1 ad4e ad6e @logger -p crit "gm3" gmirror insert -v -h gm3 stripe/gs1 auto1: # configure mirrors for automatic recovery gmirror configure -v -h -a gm0 gmirror configure -v -h -a gm1 gmirror configure -v -h -a gm2 gmirror configure -v -h -a gm3 ==> initial_ad10.label <== # /dev/ad8: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 312581808 0 unused 0 0 # "raw" part, don't edit a: 1G * 4.2BSD 0 0 d: 4G * 4.2BSD 0 0 e: * * 4.2BSD 0 0 ==> initial_ad10s1.label <== # /dev/ad8s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1G * 4.2BSD 0 0 c: 312581745 0 unused 0 0 # "raw" part, don't edit d: 4G * 4.2BSD 0 0 e: * * 4.2BSD 0 0 ==> initial_ad4.label <== # /dev/ad8: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 312581808 0 unused 0 0 # "raw" part, don't edit a: 1G * 4.2BSD 0 0 d: 4G * 4.2BSD 0 0 e: * * 4.2BSD 0 0 ==> initial_ad6.label <== # /dev/ad8: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 312581808 0 unused 0 0 # "raw" part, don't edit a: 1G * 4.2BSD 0 0 d: 4G * 4.2BSD 0 0 e: * * 4.2BSD 0 0 ==> initial_ad8.label <== # /dev/ad8: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 312581808 0 unused 0 0 # "raw" part, don't edit a: 1G * 4.2BSD 0 0 d: 4G * 4.2BSD 0 0 e: * * 4.2BSD 0 0 ==> initial_ad8s1.label <== # /dev/ad8s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1G * 4.2BSD 0 0 c: 312581745 0 unused 0 0 # "raw" part, don't edit d: 4G * 4.2BSD 0 0 e: * * 4.2BSD 0 0 ==> initial_gm1.label~ <== # /dev/mirror/gm1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 8388591 16 unused 0 0 c: 8388607 0 unused 0 0 # "raw" part, don't edit ==> initial_gm2.label <== # /dev/mirror/gm1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 8388607 0 unused 0 0 # "raw" part, don't edit d: 1G * 4.2BSD 0 0 e: * * 4.2BSD 0 0