From owner-freebsd-geom@FreeBSD.ORG Wed Mar 7 19:41:19 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 920EF16A404 for ; Wed, 7 Mar 2007 19:41:19 +0000 (UTC) (envelope-from clayton@bitheaven.net) Received: from secure.bikemonkey.org (bluefish.bikemonkey.org [69.80.211.101]) by mx1.freebsd.org (Postfix) with ESMTP id 7E94B13C48D for ; Wed, 7 Mar 2007 19:41:19 +0000 (UTC) (envelope-from clayton@bitheaven.net) Received: from [192.168.33.103] (75-162-137-20.slkc.qwest.net [75.162.137.20]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by secure.bikemonkey.org (Postfix) with ESMTP id 6925D17020 for ; Wed, 7 Mar 2007 11:16:54 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <0B1A704D-A455-4741-BC11-A2019BFB4B22@bitheaven.net> Content-Transfer-Encoding: 7bit From: Clayton F Date: Wed, 7 Mar 2007 12:16:51 -0700 To: freebsd-geom@freebsd.org X-Mailer: Apple Mail (2.752.2) Subject: Problems simulating gvinum raid5 rebuild X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bo3behemoth List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2007 19:41:19 -0000 Howdy GEOM group, I've been using gvinum raid5 for the past year, but never had to replace a drive. One just failed, and I backed up my files from the degraded raid5 array to an external drive before attempting to rebuild with a replacement drive. I botched the rebuild attempt, so now am trying to better understand how to go about restoring a raid5 under gvinum. Unfortunately, it is behaving strangely, and I haven't been able to find anything in the man pages or GEOM archives that seem to address my problem. My new configuration has 7 drives instead of 5, all partitioned in dangerously dedicated mode. I'm running 6.2 STABLE. The gvinum config is as follows: [root@alcor /export]# gvinum l 7 drives: D disk6 State: up /dev/ad14 A: 0/194480 MB (0%) D disk5 State: up /dev/ad12 A: 0/194480 MB (0%) D disk4 State: up /dev/ad10 A: 0/194480 MB (0%) D disk3 State: up /dev/ad8 A: 0/194480 MB (0%) D disk2 State: up /dev/ad6 A: 0/194480 MB (0%) D disk1 State: up /dev/ad4 A: 0/194480 MB (0%) D disk0 State: up /dev/ad2 A: 0/194480 MB (0%) 1 volume: V raid State: up Plexes: 1 Size: 1139 GB 1 plex: P raid.p0 R5 State: up Subdisks: 7 Size: 1139 GB 7 subdisks: S raid.p0.s0 State: up D: disk0 Size: 189 GB S raid.p0.s1 State: up D: disk1 Size: 189 GB S raid.p0.s2 State: up D: disk2 Size: 189 GB S raid.p0.s3 State: up D: disk3 Size: 189 GB S raid.p0.s4 State: up D: disk4 Size: 189 GB S raid.p0.s5 State: up D: disk5 Size: 189 GB S raid.p0.s6 State: up D: disk6 Size: 189 GB I am able to create a filesystem on the array, mount it and read/ write without problems. Next, I attempt to simulate a hardware drive failure by rebooting with one drive in the array unplugged, expecting to see that the array is available, but degraded due to the loss of a drive. Instead, I get the following report, showing the subdisk of the missing drive (drive4, or subdisk raid.p0.s4) as 'up,' but reporting one less drive and a volume size 189 GB smaller than 7 drive array. The array will mount, but has no data. Listing the mounted filesystems using df shows the original 1.1 terabyte array size, not the smaller value reported in gvinum. Output of gvinum and df -h are below: [root@alcor /export]# shutdown -h now (power down and unplug disk4) [root@alcor ~]# gvinum l 6 drives: D disk6 State: up /dev/ad14 A: 0/194480 MB (0%) D disk5 State: up /dev/ad12 A: 0/194480 MB (0%) D disk3 State: up /dev/ad8 A: 0/194480 MB (0%) D disk2 State: up /dev/ad6 A: 0/194480 MB (0%) D disk1 State: up /dev/ad4 A: 0/194480 MB (0%) D disk0 State: up /dev/ad2 A: 0/194480 MB (0%) 1 volume: V raid State: up Plexes: 1 Size: 949 GB 1 plex: P raid.p0 R5 State: up Subdisks: 6 Size: 949 GB 7 subdisks: S raid.p0.s0 State: up D: disk0 Size: 189 GB S raid.p0.s1 State: up D: disk1 Size: 189 GB S raid.p0.s2 State: up D: disk2 Size: 189 GB S raid.p0.s3 State: up D: disk3 Size: 189 GB S raid.p0.s4 State: up D: disk4 Size: 189 GB S raid.p0.s5 State: up D: disk5 Size: 189 GB S raid.p0.s6 State: up D: disk6 Size: 189 GB [root@alcor ~]# df -h Filesystem Size Used Avail Capacity Mounted on /dev/ad0s1a 496M 60M 396M 13% / devfs 1.0K 1.0K 0B 100% /dev /dev/ad0s1g 95G 1.7G 86G 2% /jail /dev/ad0s1e 496M 18K 456M 0% /tmp /dev/ad0s1f 9.7G 2.6G 6.3G 30% /usr /dev/ad0s1d 1.4G 64M 1.3G 5% /var /dev/gvinum/raid 1.1T 3.5G 1.0T 0% /export If I plug the drive back in (powering down first - I don't have hot swappable hardware), the array comes up normally with its data still intact. It is obvious to me that I could never rebuild the array in the event of losing a drive, which is my intent in configuring a raid5 in the first place. The behavior seems more like a jbod config than a raid5. Any suggestions? Is using 7 drives exceeding the number that gvinum raid5 will allow? Should I be labeling the drives differently? Is my method for simulating a drive failure/replacement flawed? Any help would be most appreciated! Thanks, Clayton