From owner-freebsd-questions@FreeBSD.ORG Thu Mar 7 14:03:21 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 10538D4C for ; Thu, 7 Mar 2013 14:03:21 +0000 (UTC) (envelope-from doug@polands.org) Received: from mailgate2.plexus.com (mailgate2.plexus.com [64.211.22.15]) by mx1.freebsd.org (Postfix) with ESMTP id B7C28BE8 for ; Thu, 7 Mar 2013 14:03:20 +0000 (UTC) Received: from mailgate2.plexus.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 696791A1B67; Thu, 7 Mar 2013 07:33:20 -0600 (CST) Received: from Neen-mail-009.na.plexus.com (unknown [10.255.48.136]) by mailgate2.plexus.com (Postfix) with ESMTP id 5B0F91A23B1; Thu, 7 Mar 2013 07:33:20 -0600 (CST) Received: from lystra.neenah.na.plexus.com (206.209.255.11) by Neen-mail-009.na.plexus.com (10.255.48.136) with Microsoft SMTP Server id 14.2.328.9; Thu, 7 Mar 2013 07:33:20 -0600 Date: Thu, 7 Mar 2013 07:33:20 -0600 From: Doug Poland To: Shane Ambler Subject: Re: Booting from an aribrary disk in ZFS RAIDZ on 8.x Message-ID: <20130307133319.GA71468@lystra.neenah.na.plexus.com> References: <20130305184446.GA81297@polands.org> <5136B047.3000700@ShaneWare.Biz> <20130306042456.GA92710@polands.org> <51381A79.2070305@ShaneWare.Biz> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <51381A79.2070305@ShaneWare.Biz> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: No X-TM-AS-Product-Ver: IMSVA-8.2.0.1597-7.0.0.1014-19694.007 X-TM-AS-Result: No--20.569-4.0-31-10 X-imss-scan-details: No--20.569-4.0-31-10 X-TMASE-Version: IMSVA-8.2.0.1597-7.0.1014-19694.007 X-TMASE-Result: 10--20.569100-5.000000 X-TMASE-MatchedRID: 0dFPYP4mu5QOwH4pD14DsPHkpkyUphL91KoSW5Ji1XuzTLPEogYO4EXm qHshxrcjRXJR/yrOMl+bqzDH9G7eTPatsGs9O72pgMKLAwXrj8FAg1IWH8kbY5UMECmkBP+l/F1 Brea7byKfI9eqmP8Wu7pukerCcV5JErPe+EcoU4udtRmRhPNchl3KZkFy4YZE9ItC2dmXn119qT NwMiKD+qPnTZgndIdNZyrb9KuymCZoRvAKYVTCorMjW/sniEQKH45GXKADbKLruHLA2CJi3eXoA zgBreAWdkps8KxOe2xpob5Q6RKb6RZK05xEkCkCsK+WWVTsOXUYR+gKWoGXzoxsO/peQ4cNy7dO fKnHiUYH9WrAO77U/rHn/HPzhGPjlHVBLjBDrjkSWCj0fkOcnDpVtQQHm78w3XgCp7wTMXyEsNW kHG22z58wAJkwWx+vWaNzC4DdSNCI0KPyMNrNUiSxIFlMYKvCyJ1gFgOMhOn6APa9i04WGCq2rl 3dzGQ1A/3R8k/14e0= Cc: FreeBSD-questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 14:03:21 -0000 On Thu, Mar 07, 2013 at 03:11:29PM +1030, Shane Ambler wrote: > On 06/03/2013 14:54, Doug Poland wrote: > >On Wed, Mar 06, 2013 at 01:26:07PM +1030, Shane Ambler wrote: > >>On 06/03/2013 05:14, Doug Poland wrote: > > >>>I have 6 disks in a RAIDZ configuration. All disks were sliced the > >>>same with gpart (da(n)p1,p2,p3) with bootcode written to index 1, > >>>swap on index 2 and freebsd-zfs on index 3. > >>> > >>>Given this configuration, I should be able to boot from any of the > >>>6 disks in the RAIDZ. If this is a true statement, how do I make > >>>that happen from the loader prompt? > >> > >>You don't boot from an individual disk you boot from a zpool - all > >>disks are linked together making one zpool "disk". > >> > >Something has to pick a physical device from which to boot, does it > >not?. All the HP Smart Array 6i controller knows is I have 6 RAID 0 > >disks to present to the OS. > > I meant to add if the bootcode is installed on each disk then pointing > the bios to any individual disk as the primary boot device will lead > to the boot process loading the zpool. Installing it on each disk > gives the redundancy to match the raid in the zpool. If you only have > one disk with bootcode and it is the one that needs replacing then you > can't boot. Then having 100 disks in a pool with bootcode would be > overkill, but the consistency may be easier to maintain. > So in my case, the HP SmartArray doesn't allow me to choose an individual boot disk. So it's up to the controller to keep trying to boot from the next configured disk. I believe I'm going to craft a test to prove this out. > >I've had issues with this RAID controller in the past where it won't > >present the new disk to the OS. I've had to reboot, go into the > >RAID config and tell it it's a single RAID 0 device (stupid, I > >know). > > When you think about it, as a raid controller it shouldn't make > assumptions as to how to use the new disk, should it add it to an > existing raid set, replace a missing drive or show it as a new single > drive? Being able to specify per socket as permanently jbod could be > useful feature though. > One would think. I've been testing this on a similarly configured machine and the controller eventually presents a new drive to the OS. It takes a couple of minutes, but appears to work on this test box. > >The roll of /boot/zfs/zpool.cache is a mystery to me. I belive it > >somehow tells ZFS what devices are in use. What if a disk goes > >offline or is removed? > > > > As I understand it the zpool.cache contains the zpools mounted by the > system. After reboot it then re-imports each zpool in the cache. I > believe a recent commit enabled the vfs.root.mountfrom zpool to be > imported even if there was no cache available. > > From what I have heard and seen the data about the zpool it belongs to > and the role the disk plays in the zpool is stored on each disk and > duplicated at the beginning and end of the disk. In my early > experiments after starting clean even after gparting and zeroing out > the start of the disks, zpool still says it belongs to a pool. > If that's the case, I wonder about the wisdom of re-using a drive from my test configuration? My plan has been to prove this out on test and use the same disk from test and insert it into production. One would think ZFS is smart enough to recognize a "different" drive has been inserted, even if it has the same gpart structure and came from a pool with the same name. Thanks for your help. -- Regards, Doug