From owner-freebsd-stable@FreeBSD.ORG Thu Oct 2 01:15:45 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 43564D81; Thu, 2 Oct 2014 01:15:45 +0000 (UTC) Received: from webmail.dweimer.net (24-240-198-187.static.stls.mo.charter.com [24.240.198.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.dweimer.local", Issuer "webmail2.dweimer.local" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B1FDB2D; Thu, 2 Oct 2014 01:15:43 +0000 (UTC) Received: from www.dweimer.net (webmail [192.168.5.2]) by webmail.dweimer.net (8.14.7/8.14.7) with ESMTP id s921FSr1015116 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 1 Oct 2014 20:15:28 -0500 (CDT) (envelope-from dweimer@dweimer.net) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 01 Oct 2014 20:15:28 -0500 From: dweimer To: Karl Denninger Subject: Re: Encrypted (GELI) root on ZFS troubles Organization: dweimer.net Reply-To: dweimer@dweimer.net Mail-Reply-To: dweimer@dweimer.net In-Reply-To: <542C71C9.1050907@denninger.net> References: <542C71C9.1050907@denninger.net> Message-ID: X-Sender: dweimer@dweimer.net User-Agent: Roundcube Webmail/1.0.3 Cc: owner-freebsd-stable@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Oct 2014 01:15:45 -0000 On 10/01/2014 4:27 pm, Karl Denninger wrote: > So here's the fun part of what I'm trying to do (and getting frustrated > with) > > I have set up a GPT disk with the following setup: > > => 34 625142381 da2 GPT (298G) > 34 6 - free - (3.0K) > 40 1024 1 freebsd-boot (512K) > 1064 4194304 2 freebsd-zfs [bootme] (2.0G) > 4195368 134217728 3 freebsd-swap (64G) > 138413096 486729312 4 freebsd-zfs (232G) > 625142408 7 - free - (3.5K) > > Then on freebsd-boot I have written the bootloaders. > > The "bootme" filesystem has *only* the /boot directory copied over from > the rest of the system's root directory (that is, the kernel, > loadables, > /boot/loader.conf, etc); that pool is called "zboot" > > Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt. > I have initialized that with geli, set the boot option flag (that is, > prompt on boot) and created a pool called "root" on the resulting .eli > device and then put the system on that. That's all ok. > > Finally, I set the bootfs on that latter pool. There is no bootfs set > on /zboot: > > # zpool get bootfs zboot > NAME PROPERTY VALUE SOURCE > zboot bootfs - default > > It is set on the root pool to the proper filesystem: > > # zpool get bootfs root > NAME PROPERTY VALUE SOURCE > root bootfs root/R/10.1-CLEAN local > > The problem is that when the system boots geli "finds" the raw device > (in this case /dev/da0p4), prompts for the password and attaches there > instead of in /dev/gpt. The gpt label is missing --- and equally bad > the "root" pool does not appear to import at boot time either. > > As a result the system tries to mount root from /zboot (even though > it's > not been told to, and HAS been told where to mount off the root pool), > but there's no init in there (or anything else other than the boot > filesystem itself) and as a result I get an immediate panic. > > If I boot off a different (working) zfs-based system the probe still > finds the "prompt during boot" flag on that gpt partition and asks for > the password on the device. I can see the pool; zpool import shows it: > > pool: root > id: 17719633931604198170 > state: ONLINE > action: The pool can be imported using its name or numeric identifier. > config: > > root ONLINE > da2p4.eli ONLINE > > Not so good. > > If I detach that the device reappears in /dev/gpt; I can then attach > geli and import the pool in either location. Putting the cache file > from the previous imported state in the zboot/boot/zfs directory > doesn't > help (nor does removing the cache file entirely) > > More-interestingly if I reboot the cloned system with the root pool > imported it does come back up, even though the device is the base > (da2p4.eli) rather than in the /dev/gpt directory. > > Anyone know what's going on here? And is there a way to have geli > attach during boot-time off the /dev/gpt directory instead of on the > base device partition name? On my work laptop (not turned on so I am going by memory on this), I have a similar setup using a USB thumb drive for the boot volume. My setup is as follows and works quite well, perhaps this will help you. Thumb Drive da0 Disk Drive ada0 da0 has a GPT table of da0 GPT (8G) 1 freebsd-boot (512k) -- /dev/gpt/usbboot 2 freebsd-zfs (8G) -- /dev/gpt/usbzfs gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 ada0 has a GPT table of 1 freebsd-swap (8G) -- /dev/gpt/swap 2 freebsd-zfs (222G) -- /dev/gpt/zroot I used geli init -b /dev/gpt/zroot when attached /dev/gpt/zroot.eli swap is auto encrypted at boot using the fstab line /dev/gpt/swap.eli none none swap sw 0 0 I believe they devices only show up as /dev/gpt/... if the -l ... option is used to set a label on the partition at creation time. 2 configured zpools usbzfs gpt/usbzfs zroot gpt/zroot.eli zpool set bootfs=usbzfs/boot usbzfs zpool set bootfs=zroot/ROOT/installation zroot (not sure if this does anything, I just set it) usbzfs/boot has a mountpoint of /zfsboot loader.conf: zfs_load="YES" vfs.root.mountfrom="zfs:zroot/ROOT/install" copied /boot to /zfsboot/boot zpool export usbzfs It will still boot after the zpool has been exported if the devices is found, just doesn't get mounted, in my case this means I can remove the USB thumb drive as soon as root is remounted from the geli partition, after entering the password without causing any issues. I can send you the full gpt output and zpool status information tomorrow morning when I am back in the office on my laptop if you still need help getting yours working. -- Thanks, Dean E. Weimer http://www.dweimer.net/