Date: Wed, 01 Oct 2014 23:50:26 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Encrypted (GELI) root on ZFS troubles Message-ID: <542CD992.7050509@denninger.net> In-Reply-To: <d4f3fecfcd8785c41e3658ff34123088@dweimer.net> References: <542C71C9.1050907@denninger.net> <d4f3fecfcd8785c41e3658ff34123088@dweimer.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 10/1/2014 8:15 PM, dweimer wrote: > On 10/01/2014 4:27 pm, Karl Denninger wrote: >> So here's the fun part of what I'm trying to do (and getting frustrated >> with) >> >> I have set up a GPT disk with the following setup: >> >> => 34 625142381 da2 GPT (298G) >> 34 6 - free - (3.0K) >> 40 1024 1 freebsd-boot (512K) >> 1064 4194304 2 freebsd-zfs [bootme] (2.0G) >> 4195368 134217728 3 freebsd-swap (64G) >> 138413096 486729312 4 freebsd-zfs (232G) >> 625142408 7 - free - (3.5K) >> >> Then on freebsd-boot I have written the bootloaders. >> >> The "bootme" filesystem has *only* the /boot directory copied over from >> the rest of the system's root directory (that is, the kernel, loadables, >> /boot/loader.conf, etc); that pool is called "zboot" >> >> Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt. >> I have initialized that with geli, set the boot option flag (that is, >> prompt on boot) and created a pool called "root" on the resulting .eli >> device and then put the system on that. That's all ok. >> >> Finally, I set the bootfs on that latter pool. There is no bootfs set >> on /zboot: >> >> # zpool get bootfs zboot >> NAME PROPERTY VALUE SOURCE >> zboot bootfs - default >> >> It is set on the root pool to the proper filesystem: >> >> # zpool get bootfs root >> NAME PROPERTY VALUE SOURCE >> root bootfs root/R/10.1-CLEAN local >> >> The problem is that when the system boots geli "finds" the raw device >> (in this case /dev/da0p4), prompts for the password and attaches there >> instead of in /dev/gpt. The gpt label is missing --- and equally bad >> the "root" pool does not appear to import at boot time either. >> >> As a result the system tries to mount root from /zboot (even though it's >> not been told to, and HAS been told where to mount off the root pool), >> but there's no init in there (or anything else other than the boot >> filesystem itself) and as a result I get an immediate panic. >> >> If I boot off a different (working) zfs-based system the probe still >> finds the "prompt during boot" flag on that gpt partition and asks for >> the password on the device. I can see the pool; zpool import shows it: >> >> pool: root >> id: 17719633931604198170 >> state: ONLINE >> action: The pool can be imported using its name or numeric identifier. >> config: >> >> root ONLINE >> da2p4.eli ONLINE >> >> Not so good. >> >> If I detach that the device reappears in /dev/gpt; I can then attach >> geli and import the pool in either location. Putting the cache file >> from the previous imported state in the zboot/boot/zfs directory doesn't >> help (nor does removing the cache file entirely) >> >> More-interestingly if I reboot the cloned system with the root pool >> imported it does come back up, even though the device is the base >> (da2p4.eli) rather than in the /dev/gpt directory. >> >> Anyone know what's going on here? And is there a way to have geli >> attach during boot-time off the /dev/gpt directory instead of on the >> base device partition name? > > On my work laptop (not turned on so I am going by memory on this), I > have a similar setup using a USB thumb drive for the boot volume. My > setup is as follows and works quite well, perhaps this will help you. > > Thumb Drive > da0 > > Disk Drive > ada0 > > da0 has a GPT table of > da0 GPT (8G) > 1 freebsd-boot (512k) -- /dev/gpt/usbboot > 2 freebsd-zfs (8G) -- /dev/gpt/usbzfs > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 > > ada0 has a GPT table of > 1 freebsd-swap (8G) -- /dev/gpt/swap > 2 freebsd-zfs (222G) -- /dev/gpt/zroot > > > I used geli init -b /dev/gpt/zroot > when attached /dev/gpt/zroot.eli > > swap is auto encrypted at boot using the fstab line > /dev/gpt/swap.eli none none swap sw 0 0 > > I believe they devices only show up as /dev/gpt/... if the -l ... > option is used to set a label on the partition at creation time. > > 2 configured zpools > usbzfs > gpt/usbzfs > > zroot > gpt/zroot.eli > > zpool set bootfs=usbzfs/boot usbzfs > zpool set bootfs=zroot/ROOT/installation zroot (not sure if this does > anything, I just set it) > > usbzfs/boot has a mountpoint of /zfsboot > > loader.conf: > zfs_load="YES" > vfs.root.mountfrom="zfs:zroot/ROOT/install" > > copied /boot to /zfsboot/boot > > zpool export usbzfs > > It will still boot after the zpool has been exported if the devices is > found, just doesn't get mounted, in my case this means I can remove > the USB thumb drive as soon as root is remounted from the geli > partition, after entering the password without causing any issues. > > I can send you the full gpt output and zpool status information > tomorrow morning when I am back in the office on my laptop if you > still need help getting yours working. > I got it working but..... 1. Having the kernel able to cross pools and examine the bootfs property would be fairly useful. I use beadm, but of course there are lots of risks associated with it and this sort of setup; if you forget to update the unencrypted boot area you could find yourself in pretty-serious trouble with an unbootable system -- if your emergency media is also out of date vis-a-vis zfs flags you've got trouble. 2. Not having the system look at the /dev/gpt entries (yes, I use "-l" when I create the partitions) is troublesome. Specifically, it's trouble when you have a lot of drives and an adapter that doesn't play nice with coherent unit numbering related to the cable point the disk is on -- which is true of most of the mps-driven adapters (and especially those with a SAS expander or three hooked to them.) If you've never pulled the wrong disk from a running machine by thinking the numbering is different than it actually is then this probably doesn't mean much to you :-) Not being able to be SURE which disk failed so you can make sure you yank the right one is a quite serious annoyance. -- Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ [-- Attachment #2 --] 0 *H 01 0 + 0 *H O0K030 *H 01 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 130824190344Z 180823190344Z0[1 0 UUS10UFlorida10UKarl Denninger1!0 *H karl@denninger.net0"0 *H 0 bi՞] MNԿawx?`)'ҴcWgR@BlWh+ u}ApdCF JVй~FOL }EW^bچYp3K& ׂ(R lxڝ.xz?6 &nsJ +1v9 v/( kqĪp[vjcK%fϻe?iq]z lyzFO'ppdX//Lw(3 JI A*S#՟H[f|CGqJKoo y.oEuOw$ /섀$삻J9b|A P~8]D1YI<"""Y^T2iQ2 b yH)] Ƶ0y$ _N 6XqMC 9 XgώjGTP"#nˋ"Bk1 00 U 0 0 `HB0 U 0, `HB OpenSSL Generated Certificate0 U |8 ˴d[20 U #0]Af4 U3x&^"408 `HB+)https://cudasystems.net:11443/revoked.crl0 *H gBwH] j\x`( &gW32"Uf^. ^Iϱ k!DQA g{(w /)\N'[ oRW @C HO>)XrTNɘ!u`xt5(=f\-l3<@C6mnhv##1ŃbH͍_Nq aʷ?rk$^9TIa!kh,D -ct1 0001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 + ;0 *H 1 *H 0 *H 1 141002045026Z0# *H 1l] e19I0l *H 1_0]0 `He*0 `He0 *H 0*H 0 *H @0+0 *H (0 +71001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H 101 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H N8JЂss 杮i$"X$:jLh ӇR} ='羟~OT:$]4} 2G>%S>(| P&1*\Hx@Q z :cOGW1{/xٕx/_f9}֠QeN Ծ'!/XxOi^N]j\ &$ y Gߟ#wkU b:Kob7!l ^4O:,tS m.6v4Å0 E^=l^Pqװx\_hp(OpcQؒK z] d*C+ wS9k UGZ.Z=P8ҸDU c@24~ʃlaQR$MyŨ&8Űn.8| 1RtJ*D^{Ax;$ehHMf_rk
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?542CD992.7050509>