Date: Sat, 17 Nov 2018 23:08:09 +0100 From: Marco Steinbach <coco@executive-computing.de> To: freebsd-geom@freebsd.org Subject: Re: eli encrypted providers for zfs raidz1 Message-ID: <20181117230809.428ed59a@bsdbuch.c0c0.intra> In-Reply-To: <CAOc73CDsY0CtuZxgkH0HEBrxQ%2BSS8gGHTt4MpjUWJLm3h-wBMA@mail.gmail.com> References: <20181116231809.40a8f74c@bsdbuch.c0c0.intra> <CAOc73CDsY0CtuZxgkH0HEBrxQ%2BSS8gGHTt4MpjUWJLm3h-wBMA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 17 Nov 2018 08:42:18 +0800 Ben Woods <woodsb02@gmail.com> wrote: > On Sat, 17 Nov 2018 at 06:36, Marco Steinbach > <coco@executive-computing.de> wrote: >=20 > > I'm trying to wrap my head around on how to avoid a zpool resilver > > on a non-booting ZFS raidz1 of off four equally sized (GPT) > > partitions on four distinct drives using eli for encyption. > > > > IOW: I do struggle with finding a way to attach all the > > providers such, that ZFS does not initiate a resilver due to the > > providers being attached sequentially. > > =20 >=20 >=20 > ZFS doesn't auto-mount by itself - there are other elements at work > doing this. > 1. During boot, the /etc/rc.d/zfs script will call "zfs mount -va" to > automount any available datasets. Note this only happens once during > boot (not continuously after boot), and it happens *after* > the /etc/rc.d/geli script has already attached any geli providers. > 2. The daemon zfsd(8) was newly introduced in FreeBSD 11.0 will > online any vdevs as new GEOM devices appear. It also handles other > devctl(4) events. This daemon is not enabled by default - you have to > specify zfsd_enable=3D"YES" in /etc/rc.conf if you want this behaviour. > >=20 > I suspect your problem is the second items here. Have you enabled it? I think what I was seeing was the result of me not being aware of the geli rc script, and the rc order at boot with regards to zfs. I didn't use zfsd -- I am not even sure I was aware of its existance. =46rom the man page of zfsd it looks like it'll help with what until now was my main worry with ZFS on FreeBSD: Commands not returning, forcing a reboot, due to devices dying or erroring out. I need to check that with pulling a (USB) drive from under zfs's feet :) > I've created and initialized the partitions as follows (generic > > notation, comments on chosen encryption algo welome, since this > > testing setup lacks AES-NI): > > # gpart create -s gpt /dev/ada[2-5] > > # gpart add -t freebsd-zfs /dev/ada[2-5] > > # geli init -e AES-CBC -l 128 /dev/ada[2-5]p1 > > =20 >=20 > I suspect you are just using the [2-5] as shorthand in your email > here, and not running this exact command on FreeBSD 11.2. Yes, that was shorthand notation. > For reference, geli has been updated on FreeBSD 12.0 such that you > could run this exact command to init multiple providers in a single > command (they will all use the same password/file with unique salt). >=20 >=20 > > Then I attached the geli partitions like so: > > # geli attach /dev/ada[2-5]p1 > > =20 >=20 > Same for geli attach - this could all be done in the one command like > this on FreeBSD 12.0. >=20 >=20 > > And finally created a raidz1 spanning all four partitions: > > # zpool create u0001 raidz1 /dev/ada[2-5]p1.eli > > > > That works flawlessly. And naturally, after a reboot none of the > > encrypted devices is available to the zpool then, unless I attach > > them.=20 >=20 > You can configure the /etc/rc.d/geli script to attach additional > devices (e.g. non-root ZFS) during userland boot. You can add the > following to /etc/rc.conf: > geli_devices=3D"ada2p1 ada3p1 ada4p1 ada5p1" > This will prompt you to type your password in 4 times during userland > boot, and each would be attached one at a time. >=20 > In FreeBSD 12.0, you can make this only prompt you for your password > once by putting the following in /etc/rc.conf: > geli_groups=3D"u0001" > geli_u0001_devices=3D"ada2p1 ada3p1 ada4p1 ada5p1" >=20 Pure gold. I just tried this back and forth on 12-RC1 (and 11.2 minus the neat provider name expansion), and it does exactly what I'm after. =20 > > > > Doing so using geli attach requires me to attach them sequentially, > > which then results in ZFS resilvering the pool. > > =20 >=20 > This would only occur if there was something automatically onlining > the vdevs and mounting the datasets, such as zfsd. >=20 >=20 > > So, here's my questions: > > > > 1. Is the inavoidable resilver intended behaviour based on current > > implementation, or am I missing something ? > > =20 >=20 > I suspect this is zfsd working against you in this instance. >=20 > 2. In case I use a bootable zfsroot (cudos to allanjude@, I highly > > recommend his BSDCan presentations on the matter), is there a way to > > hand over the zfsroot passphrase to eli for automatically attaching > > other providers ? > > =20 >=20 > The geli attach for root ZFS devices would be handled by the boot > loader, whilst the geli attach for non-root ZFS devices should be > handled by the userland /etc/rc.d/geli script. > Whilst you will need to type in your password separately for root and > non-root devices, FreeBSD 12.0 at least makes it so you don't have to > type the password multiple times for non-root devices. > There might be a way to get the boot loader to handle the non-root > devices also - I don't have experience with this. I think I'll start by looking at how encrpyted swap is mounted -- since, oddly, I have 11.2 occassionally ask me again for the eli swap providers passphrase during boot from a encrypted zroot. The system was installed using the encrypted swap and zroot option of the installer.=20 > > Please note, that I'd like to stick as close as possible to what the > > base system offers for this use-case. > > > > MfG CoCo > > =20 >=20 >=20 > Good luck! Thank you for your explanations -- much appreciated. I don't think that stack of new SMR drives likes the look on my face -- time for some encrypted cold storage testing ... MfG CoCo
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20181117230809.428ed59a>