From owner-freebsd-fs@FreeBSD.ORG Mon Jul 22 12:28:54 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7CD13560; Mon, 22 Jul 2013 12:28:54 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4312F2AFF; Mon, 22 Jul 2013 12:28:53 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.17]) by ltcfislmsgpa03.fnfis.com (8.14.5/8.14.5) with ESMTP id r6MCSqTI029435 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 22 Jul 2013 07:28:52 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT06.FNFIS.com ([10.132.206.17]) with mapi id 14.02.0309.002; Mon, 22 Jul 2013 07:28:43 -0500 From: "Teske, Devin" To: Andriy Gapon Subject: Re: r253070 and "disappearing" zpool Thread-Topic: r253070 and "disappearing" zpool Thread-Index: AQHOhq1kEkFm6CmVtUilt1UjZinLKZlw9HiA Date: Mon, 22 Jul 2013 12:28:43 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201FD4633@ltcfiswmsgmb21> References: <20130710180548.GA2151@glenbarber.us> <51ECDF64.2020704@FreeBSD.org> In-Reply-To: <51ECDF64.2020704@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.121] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8794, 1.0.431, 0.0.0000 definitions=2013-07-22_02:2013-07-19,2013-07-22,1970-01-01 signatures=0 Cc: "freebsd-fs@freebsd.org" , Glen Barber , FreeBSD Current , Devin Teske X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 12:28:54 -0000 On Jul 22, 2013, at 12:29 AM, Andriy Gapon wrote: >=20 > [turning this into a public discussion with Glen's permission] >=20 > on 10/07/2013 21:05 Glen Barber said the following: >> Hi, >>=20 >> My setup is like this: >>=20 >> root@nucleus:/usr/src # zpool list >> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> zboot0 9.94G 379M 9.57G 3% 1.00x ONLINE - >> zroot0 159G 113G 46.2G 70% 1.00x ONLINE - >>=20 >> root@nucleus:/usr/src # zpool get bootfs >> NAME PROPERTY VALUE SOURCE >> zboot0 bootfs - default >> zroot0 bootfs - default >>=20 >> root@nucleus:/usr/src # zfs list zboot0 >> NAME USED AVAIL REFER MOUNTPOINT >> zboot0 379M 9.41G 281M /bootdir >>=20 >> root@nucleus:/usr/src # zfs list zroot0 >> NAME USED AVAIL REFER MOUNTPOINT >> zroot0 113G 43.7G 147M / >>=20 >> 'zroot0' is a GELI-backed pool, so I have this to fix the boot process: >>=20 >> root@nucleus:/usr/src # ll /boot >> lrwxr-xr-x 1 root wheel 12 Aug 25 2012 /boot@ -> bootdir/boot >>=20 >> I upgraded from head/ on July 1 to r253159, and when I rebooted the >> system, I could correctly boot from the "/bootdir/boot". Once I enter >> the GELI passphrase, "/" (from zroot0) is mounted. Normally, everything >> would be okay at that point, but since the upgrade, "/bootdir/boot" >> disappears because the zboot0 pool is not imported as it was before. >>=20 >> Any thoughts? >=20 > I think that this setup (on ZFS level) is quite untypical, although not > impossible on FreeBSD (and perhaps only FreeBSD). > It's untypical because you have separate boot pool (where loader, loader.= conf > and kernel are taken from) and root pool (where "/" is mounted from). >=20 > There is this "magic" zpool.cache file that essentially tells what pools = should > be automatically imported. On FreeBSD this file lives in /boot/zfs direc= tory > while originally (in Solaris and its descendants) its home is /etc/zfs. >=20 > Until recently FreeBSD could use only zpool.cache from a boot pool and, i= n fact, > if a root pool was different from a boot pool the root pool had to be in > zpool.cache. > I changed things a little bit and now a root pool does not have to be in > zpool.cache. Also, now zpool.cache is taken from the root > pool, or to be more precise from a root filesystem (whatever happens to be > /boot/zfs/zpool.cache after "/" is mounted). I am considering if perhaps= now we > should move zpool.cache back to /etc/zfs/. >=20 > So, I see three ways of resolving the problem that my changes caused for = your > configuration. >=20 > 1. [the easiest] Put zpool.cache loading instructions that used to be in > defaults/loader.conf into your loader.conf. This way everything should w= ork as > before -- zpool.cache would be loaded from your boot pool. >=20 > 2. Somehow (I don't want to go into any technical details here) arrange t= hat > your root pool has /boot/zfs/zpool.cache that describes your boot pool. = This is > probably hard given that your /boot is a symlink at the moment. This pro= bably > would be easier to achieve if zpool.cache lived in /etc/zfs. >=20 > 3. [my favorite] Remove an artificial difference between your boot and r= oot > pools, so that they are a single root+boot pool (as zfs gods intended). = As far > as I understand your setup, you use GELI to protect some sensitive data. > Apparently your kernel is not sensitive data, so I wonder if your /bin/sh= or > /sbin/init are really sensitive either. > So perhaps you can arrange your unencrypted pool to hold all of the base = system > (boot + root) and put all your truly sensitive filesystems (like e.g. /ho= me or > /var/data or /opt/xyz) onto your encrypted pool. > ZFS is really flexible, you can use mountpoint and canmount properties to= place > your filesystems from same or different pools into whatever file namespace > hierarchy you desire. Remember that your filesystem hierarchy in the mou= nt > namespace does not always have to be the same as your ZFS dataset hierarc= hy. >=20 > I hope that this makes sense to you. > If you have any additional questions, please do not hesitate. >=20 > P.S. > ZFS/FreeBSD boot process is extremely flexible. For example zfsboot can = take > zfsloader from pool1/fsA, zfsloader can boot kernel from pool2/fsB and ke= rnel > can mount / from pool3/fsC. Of these 3 filesystems from where should > zpool.cache be taken? > My firm opinion is that it should be taken from / (pool3/fsC in the examp= le > above). Because it is the root filesystem that defines what a system is = going > to do ultimately: what daemons are started, with what configurations, etc. > And thus it should also determine what pools to auto-import. > We can say that zpool.cache is analogous to /etc/fstab in this respect. >=20 > So I understand that my change causes a problem for a setup like yours, b= ut I > believe that the change is correct. >=20 +1 for zpool.cache on / (pool3/fsC in last example) --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.