From owner-freebsd-doc@FreeBSD.ORG Tue Jul 7 11:33:12 2009 Return-Path: Delivered-To: freebsd-doc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35A3D106566C for ; Tue, 7 Jul 2009 11:33:12 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from dirj.bris.ac.uk (dirj.bris.ac.uk [137.222.10.78]) by mx1.freebsd.org (Postfix) with ESMTP id DE7C08FC15 for ; Tue, 7 Jul 2009 11:33:10 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from isis.bris.ac.uk ([137.222.10.63]) by dirj.bris.ac.uk with esmtp (Exim 4.69) (envelope-from ) id 1MO8eh-00023F-5Y for freebsd-doc@freebsd.org; Tue, 07 Jul 2009 12:16:14 +0100 Received: from mech-cluster238.men.bris.ac.uk ([137.222.187.238]) by isis.bris.ac.uk with esmtp (Exim 4.67) (envelope-from ) id 1MO8eg-0004UY-C9 for freebsd-doc@freebsd.org; Tue, 07 Jul 2009 12:16:10 +0100 Received: from mech-cluster238.men.bris.ac.uk (localhost.men.bris.ac.uk [127.0.0.1]) by mech-cluster238.men.bris.ac.uk (8.14.3/8.14.3) with ESMTP id n67BG9vM045690 for ; Tue, 7 Jul 2009 12:16:09 +0100 (BST) (envelope-from mexas@bristol.ac.uk) Received: (from mexas@localhost) by mech-cluster238.men.bris.ac.uk (8.14.3/8.14.3/Submit) id n67BG9Ha045689 for freebsd-doc@freebsd.org; Tue, 7 Jul 2009 12:16:09 +0100 (BST) (envelope-from mexas@bristol.ac.uk) X-Authentication-Warning: mech-cluster238.men.bris.ac.uk: mexas set sender to mexas@bristol.ac.uk using -f Date: Tue, 7 Jul 2009 12:16:09 +0100 From: Anton Shterenlikht To: freebsd-doc@freebsd.org Message-ID: <20090707111609.GA45627@mech-cluster238.men.bris.ac.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) X-Spam-Score: -1.4 X-Spam-Level: - Subject: handbook incorrect in 19.4 RAID1 for ia64 X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2009 11:33:12 -0000 After careful reading section 19.4 RAID1 of the handbook, and after some discussion on ia64 list, I believe this section is incorrect when it comes to ia64 platform. I'd like to write a correction, but not sure if I should aim for a subsection, e.g "19.4.x RAID1 on ia64", or just some notes and examples in the main 19.4 article. The major problems are: (a) ia64 uses GPT instead of MBR, and secondary GPT is stored in the last 34 sectors of the disk, so that it is damaged by gmirror; (b) because of (a) gmirror per partition is the only possible way forward. (c) gmirror per partition ivolves serveral extra steps because only unmounted partition can be mirrored. My experience is briefly detailed below, and I was thinking of submitting something along these lines. Please advise anton ******************* Some steps are related to install on my particular box, i.e. not generic ia64 requirements. It seems quite complicated, requiring two reboots. Please see if you can shorten or simplify the procedure. I'd like to write a section on RAID-1 on ia64 for the handbook based on this, because existing advice is incorrect when it comes to ia64. So, I welcome any suggestions on these quidelines. This example is based on setting up gmirror on rx2600 with two directly attached scsi disks, da0 and da1. 1. preliminary: read man pages for gpart(8), gmirror(8), geom(4), geom(8), dump(8), restore(8). use # gpart show # gmirror status # gmirror list between the steps to see partitions and mirror state. 2. fresh FBSD install on da0, as default, but: a) /efi is 400MB b) set up at least one network interface c) create at least one user and add him to group wheel (b) and (c) are necessary because /etc/ttys by default does not open getty on console via MP, so ssh connection is required. 3. (optional, possibly for my box only): add hw.ata.atapi_dma=0 to /boot/device.hints 4. (optional) enable extra debug messages: # sysctl kern.geom.debugflags=17 5. partition a spare disk, da1, exactly as the boot disk: # gpart create -s gpt da1 # gpart add ... so that the results look like: # gpart show => 34 143374671 da0 GPT (68G) 34 819200 1 efi (400M) 819234 1048576 2 freebsd-ufs (512M) 1867810 4194304 3 freebsd-swap (2.0G) 6062114 2097152 4 freebsd-ufs (1.0G) 8159266 2097152 5 freebsd-ufs (1.0G) 10256418 133118287 6 freebsd-ufs (63G) => 34 143374671 da1 GPT (68G) 34 819200 1 efi (400M) 819234 1048576 2 freebsd-ufs (512M) 1867810 4194304 3 freebsd-swap (2.0G) 6062114 2097152 4 freebsd-ufs (1.0G) 8159266 2097152 5 freebsd-ufs (1.0G) 10256418 133118287 6 freebsd-ufs (63G) # 6. load gmirror kernel module # gmirror load 7. create mirror for EFI partition: a) unmount /efi because GEOM manipulations can be performed only on unmounted, not in use, partition: # umount /efi b) create efi mirror on the boot(!) disk: # gmirror label -vb round-robin efi da0p1 c) add spare disk's efi partition to the mirror: # gmirror insert efi da1p1 rebuilding should start, check progress with # gmirror status after rebuilding is complete you should see: # gmirror status efi Name Status Components mirror/efi COMPLETE da0p1 da1p1 # d) mount mirror efi: # mount -t msdosfs /dev/mirror/efi /efi # df Filesystem 1K-blocks Used Avail Capacity Mounted on ... /dev/mirror/efi 409504 163264 246240 40% /efi # 8. create mirror for root (/) partition. This involves extra steps since / cannot be unmounted. a) create mirror on the spare disk: # gmirror label -vb round-robin root da1p2 b) create ufs filesystem on the mirror: # newfs /dev/mirror/root c) mount root mirror temporarily, say under /mnt: # mount /dev/mirror/root /mnt d) now copy / onto /mnt (actually onto the root mirror, /dev/mirror/root). Use a combination of dump(8) and restore(8). No other copying tool will do it right: # cd /mnt # dump 0aLf - / | restore rf - (check the man pages for more details on the options) 9. update fstab on the mirror(!) Edit /mnt/etc/fstab and change da0p1 into mirror/efi and da0p2 into mirror/root. 10. update /boot/loader.conf a) put this line at the very beginning: geom_mirror_load="YES" f) In this line: vfs.root.mountfrom="ufs:/dev/da0p2" replace da0p2 with mirror/root, so that it is: vfs.root.mountfrom="ufs:/dev/mirror/root" 11. reboot into single user(!) mode on shutdown you should see root and efi mirrors destroyed. 12. on boot you should see gmirror loaded, /dev/mirror/root and /dev/mirror/efi started and /dev/mirror/root is used as a boot device, 13. now that you have booted from /dev/da1p2, da0p2 is not mounted, so it can be inserted into root mirror: # gmirror insert root da0p2 14. create mirrors for all other partitions of da0, which are now not mounted: # gmirror label -vb round-robin swap da0p3 # gmirror label -vb round-robin var da0p4 # gmirror label -vb round-robin tmp da0p5 # gmirror label -vb round-robin usr da0p6 15. edit /etc/fstab and change da0p3 into mirrior/swap da0p4 into mirrior/var da0p5 into mirrior/tmp da0p6 into mirrior/usr 16. reboot 17. on startup add spare disks to mirrors: # gmirror insert swap da1p3 # gmirror insert var da1p4 # gmirror insert tmp da1p5 # gmirror insert usr da1p6 18. done! -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 928 8233 Fax: +44 (0)117 929 4423