From owner-freebsd-stable@FreeBSD.ORG Wed Sep 3 12:28:42 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21B61106567E for ; Wed, 3 Sep 2008 12:28:42 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id A29BD8FC17 for ; Wed, 3 Sep 2008 12:28:41 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from [192.168.0.10] by mainframe.kkip.pl with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KarTR-000KTo-J1 for freebsd-stable@freebsd.org; Wed, 03 Sep 2008 14:28:39 +0200 Message-ID: <48BE82F6.9070103@kkip.pl> Date: Wed, 03 Sep 2008 14:28:38 +0200 From: Bartosz Stec User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 8bit X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.69 (build at 26-Jun-2008 18:19:28) X-Date: 2008-09-03 14:28:39 X-Connected-IP: 192.168.0.10:4288 X-Message-Linecount: 140 X-Body-Linecount: 129 X-Message-Size: 5665 X-Body-Size: 5225 X-Received-Count: 1 X-Recipient-Count: 1 X-Local-Recipient-Count: 1 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Subject: bugged sysinstall, bsdlabel, zfs, gmirror - recept for disaster :) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Sep 2008 12:28:42 -0000 Hello there! Here's my story, hopefully some of you won't follow my steps and avoid some troubles :) Yesterday I've decided that's about time to test zfs functionality on my home server PC (i386 FreeBSD 7.1-pre) . A couple of weeks ago I bought new desktop PC (with SATA), so I had a bunch of PATA disks from old one to use in server. Lucky me - there was 3 HDD at size 40GB - RAIDZ was on approch! So after a thirty minutes I had a plan, and my server had 4 disks connected - one 20GB with actual system (ad1), and three 40GB to replace actual system (ad[023]). Plan was simple: 1. csup freebsd-stable 2. follow the tuning guide for zfs, rebuild world, kernel, and follow system upgrade 3. Reboot in single user mode 4. fdisk new disks with sysinstall using one big slice for every disk 5. bsdlabel every new disk with sysinstall using: 1GB for /, 512MB for swap, and rest unused (for ZFS) 6. gmirror -n -v -b round-robin boot ad0s1a ad2s1a ad3s1a 7. newfs /dev/mirror/boot 8. mount /dev/mirror/boot /mnt && cd /mnt 9. dump -h 0 -L -f - -C 32 / | restore rf - 10. zpool create tank raidz ad0s1d ad2s1d ad3s1d 11. zfs create new cool filesystems :) 12. dump | restore old ufs2 filesystem to new cool zfs filesystems :) 13. changing mount points from tank/foo to /foo 14. edit new fstab on mirror by replacing root mount point by "boot" mirror, adding new swaps and remove ald ones and all fs now placed on zpool 15. power off system, detach ad1 and power on new system in mixed gmirror - raidz environment. Yay! Well...it has almost works. Sysinstall screw it up. I was always too lazy to read man bsdlabel, that's why I've been using this "nice" tool for disk related tasks. Such a mistake! Problem with labels created with sysinstal, is that it aks for a mount point for every partition in slice. Well, in my case it was unwanted behaviour, so on every disk I created first: a: / b: swap c: none d: /foo Then by using "M" key I removed mount points and saved changes with "W". At this point everything seems ok. So I've added gmirror to loader.conf and run "gmirror label -n -v -b round-robin boot ad0s1a ad2s1a ad3s1a". Still ok until now. Next step - kldload geom_mirror. Here's disaster! System became unresponsible and hangs after a while. Reboot didn't help, just after gmirror module was loaded by kernel, screen was flooded with messages: WARNING: Expected rawoffset 0, found 63 andy didn't boot. I've made system start only because an old drive ad1 has no gmirror module added to loader.conf. So after reboot I've cleared metadata on providers and made some another attempts, but results were always the same. Finally I have found explanation for this issue. Man bsdlabel says: /offset/ The offset of the start of the partition from the beginning of the drive in sectors, or *** to have *bsdlabel* calculate the correct offset to use (the end of the previous partition plus one, ignor- ing partition `c'). For partition `c', *** will be interpreted as an offset of 0. The first partition should start at offset 16, because the first 16 sectors are reserved for metadata. So proper labels for disks should be (and they are now): # /dev/ad0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2097152 16 4.2BSD 0 0 0 b: 1048576 2097168 swap c: 78156162 0 unused 0 0 # "raw" part, don't edit d: 75010418 3145744 unused 0 0 Problem was - Sysinstall has placed partition "a:" starting with offset 0! This is what happens when you don't RTFM :) I assume that this bug occured because I created mount point for root on ad[023]s1a and removed it after, than saved label. It seems that GEOM framework didn't expect this, neither maual for bsdlabel. I think that should be fixed somehow. Fortunately manually editing labels by "bsdlabel -e" wasn't so hard as I expected. This is how I made everything back to normal: a: 1024M * 4.2BSD 0 0 0 b: 512M * swap c: 78156162 0 unused 0 0 # "raw" part, don't edit d: * * unused 0 0 After that, gmirror has stopped pissing me off, and I finished my plan, as below: # zpool status pool: tank state: ONLINE scrub: scrub completed with 0 errors on Wed Sep 3 10:10:07 2008 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1d ONLINE 0 0 0 ad2s1d ONLINE 0 0 0 ad3s1d ONLINE 0 0 0 errors: No known data errors # gmirror status Name Status Components mirror/boot COMPLETE ad0s1a ad2s1a ad3s1a Good luck with ZFS everyone! :) And RTFM ;) -- Bartosz Stec - specjalista ds. IT AUXILIA Spółka z o.o.