Date: Mon, 22 Jul 2013 12:28:43 +0000 From: "Teske, Devin" <Devin.Teske@fisglobal.com> To: Andriy Gapon <avg@FreeBSD.org> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@FreeBSD.org>, Glen Barber <gjb@FreeBSD.org>, FreeBSD Current <freebsd-current@FreeBSD.org>, Devin Teske <dteske@freebsd.org> Subject: Re: r253070 and "disappearing" zpool Message-ID: <13CA24D6AB415D428143D44749F57D7201FD4633@ltcfiswmsgmb21> In-Reply-To: <51ECDF64.2020704@FreeBSD.org> References: <20130710180548.GA2151@glenbarber.us> <51ECDF64.2020704@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 22, 2013, at 12:29 AM, Andriy Gapon wrote: >=20 > [turning this into a public discussion with Glen's permission] >=20 > on 10/07/2013 21:05 Glen Barber said the following: >> Hi, >>=20 >> My setup is like this: >>=20 >> root@nucleus:/usr/src # zpool list >> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> zboot0 9.94G 379M 9.57G 3% 1.00x ONLINE - >> zroot0 159G 113G 46.2G 70% 1.00x ONLINE - >>=20 >> root@nucleus:/usr/src # zpool get bootfs >> NAME PROPERTY VALUE SOURCE >> zboot0 bootfs - default >> zroot0 bootfs - default >>=20 >> root@nucleus:/usr/src # zfs list zboot0 >> NAME USED AVAIL REFER MOUNTPOINT >> zboot0 379M 9.41G 281M /bootdir >>=20 >> root@nucleus:/usr/src # zfs list zroot0 >> NAME USED AVAIL REFER MOUNTPOINT >> zroot0 113G 43.7G 147M / >>=20 >> 'zroot0' is a GELI-backed pool, so I have this to fix the boot process: >>=20 >> root@nucleus:/usr/src # ll /boot >> lrwxr-xr-x 1 root wheel 12 Aug 25 2012 /boot@ -> bootdir/boot >>=20 >> I upgraded from head/ on July 1 to r253159, and when I rebooted the >> system, I could correctly boot from the "/bootdir/boot". Once I enter >> the GELI passphrase, "/" (from zroot0) is mounted. Normally, everything >> would be okay at that point, but since the upgrade, "/bootdir/boot" >> disappears because the zboot0 pool is not imported as it was before. >>=20 >> Any thoughts? >=20 > I think that this setup (on ZFS level) is quite untypical, although not > impossible on FreeBSD (and perhaps only FreeBSD). > It's untypical because you have separate boot pool (where loader, loader.= conf > and kernel are taken from) and root pool (where "/" is mounted from). >=20 > There is this "magic" zpool.cache file that essentially tells what pools = should > be automatically imported. On FreeBSD this file lives in /boot/zfs direc= tory > while originally (in Solaris and its descendants) its home is /etc/zfs. >=20 > Until recently FreeBSD could use only zpool.cache from a boot pool and, i= n fact, > if a root pool was different from a boot pool the root pool had to be in > zpool.cache. > I changed things a little bit and now a root pool does not have to be in > zpool.cache. Also, now zpool.cache is taken from the root > pool, or to be more precise from a root filesystem (whatever happens to be > /boot/zfs/zpool.cache after "/" is mounted). I am considering if perhaps= now we > should move zpool.cache back to /etc/zfs/. >=20 > So, I see three ways of resolving the problem that my changes caused for = your > configuration. >=20 > 1. [the easiest] Put zpool.cache loading instructions that used to be in > defaults/loader.conf into your loader.conf. This way everything should w= ork as > before -- zpool.cache would be loaded from your boot pool. >=20 > 2. Somehow (I don't want to go into any technical details here) arrange t= hat > your root pool has /boot/zfs/zpool.cache that describes your boot pool. = This is > probably hard given that your /boot is a symlink at the moment. This pro= bably > would be easier to achieve if zpool.cache lived in /etc/zfs. >=20 > 3. [my favorite] Remove an artificial difference between your boot and r= oot > pools, so that they are a single root+boot pool (as zfs gods intended). = As far > as I understand your setup, you use GELI to protect some sensitive data. > Apparently your kernel is not sensitive data, so I wonder if your /bin/sh= or > /sbin/init are really sensitive either. > So perhaps you can arrange your unencrypted pool to hold all of the base = system > (boot + root) and put all your truly sensitive filesystems (like e.g. /ho= me or > /var/data or /opt/xyz) onto your encrypted pool. > ZFS is really flexible, you can use mountpoint and canmount properties to= place > your filesystems from same or different pools into whatever file namespace > hierarchy you desire. Remember that your filesystem hierarchy in the mou= nt > namespace does not always have to be the same as your ZFS dataset hierarc= hy. >=20 > I hope that this makes sense to you. > If you have any additional questions, please do not hesitate. >=20 > P.S. > ZFS/FreeBSD boot process is extremely flexible. For example zfsboot can = take > zfsloader from pool1/fsA, zfsloader can boot kernel from pool2/fsB and ke= rnel > can mount / from pool3/fsC. Of these 3 filesystems from where should > zpool.cache be taken? > My firm opinion is that it should be taken from / (pool3/fsC in the examp= le > above). Because it is the root filesystem that defines what a system is = going > to do ultimately: what daemons are started, with what configurations, etc. > And thus it should also determine what pools to auto-import. > We can say that zpool.cache is analogous to /etc/fstab in this respect. >=20 > So I understand that my change causes a problem for a setup like yours, b= ut I > believe that the change is correct. >=20 +1 for zpool.cache on / (pool3/fsC in last example) --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?13CA24D6AB415D428143D44749F57D7201FD4633>