From owner-freebsd-geom@freebsd.org Sun Nov 18 02:25:06 2018 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE21F11237AC for ; Sun, 18 Nov 2018 02:25:05 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (mail.moehre.org [195.96.35.7]) by mx1.freebsd.org (Postfix) with ESMTP id E19668DA78 for ; Sun, 18 Nov 2018 02:25:04 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (unknown [195.96.35.7]) by mail.moehre.org (Postfix) with ESMTP id CE42526722 for ; Sat, 17 Nov 2018 23:08:00 +0100 (CET) X-Spam-Flag: NO X-Spam-Score: -100.863 X-Spam-Level: X-Spam-Status: No, score=-100.863 tagged_above=-999 required=5 tests=[ALL_TRUSTED=-1, AWL=-0.017, TW_GP=0.077, TW_ZF=0.077, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mail.moehre.org ([195.96.35.7]) by mail.moehre.org (mail.moehre.org [195.96.35.7]) (amavisd-new, port 10024) with ESMTP id nQl_x56O1ZLA for ; Sat, 17 Nov 2018 23:08:00 +0100 (CET) Received: from bsdbuch.c0c0.intra (p54BEC3C8.dip0.t-ipconnect.de [84.190.195.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: coco@executive-computing.de) by mail.moehre.org (Postfix) with ESMTPSA id 068452671A for ; Sat, 17 Nov 2018 23:07:59 +0100 (CET) Date: Sat, 17 Nov 2018 23:08:09 +0100 From: Marco Steinbach To: freebsd-geom@freebsd.org Subject: Re: eli encrypted providers for zfs raidz1 Message-ID: <20181117230809.428ed59a@bsdbuch.c0c0.intra> In-Reply-To: References: <20181116231809.40a8f74c@bsdbuch.c0c0.intra> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: E19668DA78 X-Spamd-Result: default: False [-0.92 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[executive-computing.de]; MX_GOOD(-0.01)[mail.moehre.org]; NEURAL_HAM_SHORT(-0.71)[-0.706,0]; RCVD_NO_TLS_LAST(0.10)[]; RECEIVED_SPAMHAUS_PBL(0.00)[200.195.190.84.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; IP_SCORE(-0.00)[country: DE(-0.01)]; ASN(0.00)[asn:8354, ipnet:195.96.32.0/19, country:DE]; FROM_EQ_ENVFROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 02:25:06 -0000 On Sat, 17 Nov 2018 08:42:18 +0800 Ben Woods wrote: > On Sat, 17 Nov 2018 at 06:36, Marco Steinbach > wrote: >=20 > > I'm trying to wrap my head around on how to avoid a zpool resilver > > on a non-booting ZFS raidz1 of off four equally sized (GPT) > > partitions on four distinct drives using eli for encyption. > > > > IOW: I do struggle with finding a way to attach all the > > providers such, that ZFS does not initiate a resilver due to the > > providers being attached sequentially. > > =20 >=20 >=20 > ZFS doesn't auto-mount by itself - there are other elements at work > doing this. > 1. During boot, the /etc/rc.d/zfs script will call "zfs mount -va" to > automount any available datasets. Note this only happens once during > boot (not continuously after boot), and it happens *after* > the /etc/rc.d/geli script has already attached any geli providers. > 2. The daemon zfsd(8) was newly introduced in FreeBSD 11.0 will > online any vdevs as new GEOM devices appear. It also handles other > devctl(4) events. This daemon is not enabled by default - you have to > specify zfsd_enable=3D"YES" in /etc/rc.conf if you want this behaviour. > >=20 > I suspect your problem is the second items here. Have you enabled it? I think what I was seeing was the result of me not being aware of the geli rc script, and the rc order at boot with regards to zfs. I didn't use zfsd -- I am not even sure I was aware of its existance. =46rom the man page of zfsd it looks like it'll help with what until now was my main worry with ZFS on FreeBSD: Commands not returning, forcing a reboot, due to devices dying or erroring out. I need to check that with pulling a (USB) drive from under zfs's feet :) > I've created and initialized the partitions as follows (generic > > notation, comments on chosen encryption algo welome, since this > > testing setup lacks AES-NI): > > # gpart create -s gpt /dev/ada[2-5] > > # gpart add -t freebsd-zfs /dev/ada[2-5] > > # geli init -e AES-CBC -l 128 /dev/ada[2-5]p1 > > =20 >=20 > I suspect you are just using the [2-5] as shorthand in your email > here, and not running this exact command on FreeBSD 11.2. Yes, that was shorthand notation. > For reference, geli has been updated on FreeBSD 12.0 such that you > could run this exact command to init multiple providers in a single > command (they will all use the same password/file with unique salt). >=20 >=20 > > Then I attached the geli partitions like so: > > # geli attach /dev/ada[2-5]p1 > > =20 >=20 > Same for geli attach - this could all be done in the one command like > this on FreeBSD 12.0. >=20 >=20 > > And finally created a raidz1 spanning all four partitions: > > # zpool create u0001 raidz1 /dev/ada[2-5]p1.eli > > > > That works flawlessly. And naturally, after a reboot none of the > > encrypted devices is available to the zpool then, unless I attach > > them.=20 >=20 > You can configure the /etc/rc.d/geli script to attach additional > devices (e.g. non-root ZFS) during userland boot. You can add the > following to /etc/rc.conf: > geli_devices=3D"ada2p1 ada3p1 ada4p1 ada5p1" > This will prompt you to type your password in 4 times during userland > boot, and each would be attached one at a time. >=20 > In FreeBSD 12.0, you can make this only prompt you for your password > once by putting the following in /etc/rc.conf: > geli_groups=3D"u0001" > geli_u0001_devices=3D"ada2p1 ada3p1 ada4p1 ada5p1" >=20 Pure gold. I just tried this back and forth on 12-RC1 (and 11.2 minus the neat provider name expansion), and it does exactly what I'm after. =20 > > > > Doing so using geli attach requires me to attach them sequentially, > > which then results in ZFS resilvering the pool. > > =20 >=20 > This would only occur if there was something automatically onlining > the vdevs and mounting the datasets, such as zfsd. >=20 >=20 > > So, here's my questions: > > > > 1. Is the inavoidable resilver intended behaviour based on current > > implementation, or am I missing something ? > > =20 >=20 > I suspect this is zfsd working against you in this instance. >=20 > 2. In case I use a bootable zfsroot (cudos to allanjude@, I highly > > recommend his BSDCan presentations on the matter), is there a way to > > hand over the zfsroot passphrase to eli for automatically attaching > > other providers ? > > =20 >=20 > The geli attach for root ZFS devices would be handled by the boot > loader, whilst the geli attach for non-root ZFS devices should be > handled by the userland /etc/rc.d/geli script. > Whilst you will need to type in your password separately for root and > non-root devices, FreeBSD 12.0 at least makes it so you don't have to > type the password multiple times for non-root devices. > There might be a way to get the boot loader to handle the non-root > devices also - I don't have experience with this. I think I'll start by looking at how encrpyted swap is mounted -- since, oddly, I have 11.2 occassionally ask me again for the eli swap providers passphrase during boot from a encrypted zroot. The system was installed using the encrypted swap and zroot option of the installer.=20 > > Please note, that I'd like to stick as close as possible to what the > > base system offers for this use-case. > > > > MfG CoCo > > =20 >=20 >=20 > Good luck! Thank you for your explanations -- much appreciated. I don't think that stack of new SMR drives likes the look on my face -- time for some encrypted cold storage testing ... MfG CoCo