From owner-freebsd-fs@freebsd.org Mon Jul 6 13:10:26 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7D8B366DD0 for ; Mon, 6 Jul 2020 13:10:26 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mx1.cksoft.de (mx1.cksoft.de [IPv6:2001:67c:24f8:1::25:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.cksoft.de", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B0mC54vQRz41w3 for ; Mon, 6 Jul 2020 13:10:25 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from m.cksoft.de (m.cksoft.de [IPv6:2001:67c:24f8:2003::25:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.cksoft.de (Postfix) with ESMTPSA id 37AC51EA983 for ; Mon, 6 Jul 2020 15:10:17 +0200 (CEST) Received: from amavisfra1.cksoft.de (unknown [IPv6:2001:67c:24f8:2003::25:a1]) by m.cksoft.de (Postfix) with ESMTP id D4FAF63027 for ; Mon, 6 Jul 2020 15:10:16 +0200 (CEST) X-Virus-Scanned: amavisd-new at cksoft.de Received: from m.cksoft.de ([192.168.35.42]) by amavisfra1.cksoft.de (amavisfra1.cksoft.de [192.168.35.41]) (amavisd-new, port 10051) with ESMTP id 1Z05Cp9bBgsa for ; Mon, 6 Jul 2020 15:10:14 +0200 (CEST) Received: from nocfra1.cksoft.de (nocfra1.cksoft.de [IPv6:2001:67c:24f8:2001::53:1]) by m.cksoft.de (Postfix) with ESMTP id 16CCF63026 for ; Mon, 6 Jul 2020 15:10:16 +0200 (CEST) Received: by nocfra1.cksoft.de (Postfix, from userid 1000) id 174D713E92; Mon, 6 Jul 2020 15:10:16 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by nocfra1.cksoft.de (Postfix) with ESMTP id 12D3B13E3F for ; Mon, 6 Jul 2020 15:10:16 +0200 (CEST) Date: Mon, 6 Jul 2020 15:10:16 +0200 (CEST) From: Christian Kratzer X-X-Sender: ck@nocfra1.cksoft.de Reply-To: Christian Kratzer To: freebsd-fs Subject: gptzfsboot targeting wrong vdev Message-ID: User-Agent: Alpine 2.22 (BSF 395 2020-01-19) X-NCC-RegID: de.cksoft X-Spammer-Kill-Ratio: 75% MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 4B0mC54vQRz41w3 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ck-lists@cksoft.de designates 2001:67c:24f8:1::25:1 as permitted sender) smtp.mailfrom=ck-lists@cksoft.de X-Spamd-Result: default: False [-2.48 / 15.00]; HAS_REPLYTO(0.00)[ck@cksoft.de]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.98)[-0.978]; DMARC_NA(0.00)[cksoft.de]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.20)[-0.205]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:57407, ipnet:2001:67c:24f8::/48, country:DE]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2020 13:10:26 -0000 Hi, I have a couple of freebsd based zfs servers on 12.1-RELEASE-p6 that have zfs root and separate geli procted zfs data pools. I have been booting these off usb sticks or sdcards with a copy of /boot and the geli keys and then subsequently mounting root from the zfsroot pool. root@zfs1:/home/ck # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zp1 3.62T 242G 3.39T - - 3% 6% 1.00x ONLINE - zp2 21.8T 13.2T 8.51T - - 13% 60% 1.00x ONLINE - zroot 127G 112G 15.1G - - 30% 88% 1.00x ONLINE - root@zfs1:/home/ck # In above setup zroot is an unencrypted two ssd vdev with just the OS on it. The disks for zp1 and zp2 are encrypted using geli via keys provided in loader.conf. Above setup is working fine whilst I keep /boot on the usb stick. I have tried again to get rid of the usb stick and booting directly from the mirror on zroot. I have following in my loader.conf # zfs zfs_load="YES" vfs.root.mountfrom="zfs:zroot/ROOT/default" vfs.zfs.boot.primary_pool="17582686064334413803" vfs.zfs.boot.primary_vdev="9317482241260982593" I got the guid values using root@zfs1:/usr/src # zpool get guid zroot NAME PROPERTY VALUE SOURCE zroot guid 17582686064334413803 default root@zfs1:/usr/src # zfs get guid zroot/ROOT/default NAME PROPERTY VALUE SOURCE zroot/ROOT/default guid 9317482241260982593 - I also have following setup on the devices that are part of zroot root@zfs1:/home/ck # gpart show ada0 => 40 468862048 ada0 GPT (224G) 40 1024 1 freebsd-boot (512K) 1064 134217728 2 freebsd-swap (64G) 134218792 33554432 3 freebsd-zfs (16G) 167773224 33554432 4 freebsd-zfs (16G) 201327656 267534424 5 freebsd-zfs (128G) 468862080 8 - free - (4.0K) root@zfs1:/home/ck # When booting from ada0 I get following: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zp1 gptzfsboot: failed to mount default pool zp1 FreeBSD/x86 boot At least it seems that my bios is loading the zfs loader from the correct gpt partition. But I cannot figure out why it is still trying to boot from zp1 and not zroot. I have not been able to break out into the loader via serial console to inspect if those vfs values are really there. I do see them if I boot via my usb stick copy of /boot root@zfs1:/home/ck # kenv | grep vfs vfs.root.mountfrom="zfs:zroot/ROOT/default" vfs.zfs.boot.primary_pool="17582686064334413803" vfs.zfs.boot.primary_vdev="9317482241260982593" root@zfs1:/home/ck # This should all be unrelated to geli as I only need the zroot pool mounted at this point in time which is unencrypted. I am a bit lost at this stage and would have following questions if anybody can help me: 1. Am I getting the correct guid values for the primary_pool and _vdev or am I misundestanding the requirements ? 2. How can I break into the loader to inspect or set kenv values ? 3. Should I perhaps try exporting zp1 and zp2 so as only to have zroot visible ? 4. Can I try setting the vfs variables from /config ? How would the syntax for that be ? Anything else I can do or check to get this to work ? Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/ From owner-freebsd-fs@freebsd.org Tue Jul 7 00:07:16 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C79C834D30C for ; Tue, 7 Jul 2020 00:07:16 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [IPv6:2001:470:1:474::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B12mz6CyRz3W0G for ; Tue, 7 Jul 2020 00:07:15 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 5AE21276B9; Tue, 7 Jul 2020 00:07:09 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 5AE21276B9 From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: Date: Mon, 6 Jul 2020 20:07:05 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO" X-Rspamd-Queue-Id: 4B12mz6CyRz3W0G X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 00:07:16 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO Content-Type: multipart/mixed; boundary="3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> In-Reply-To: <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> --3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable In my continuing effort to complete the integration of ZSTD into OpenZFS, here is my third weekly status report: https://github.com/allanjude/zfs/commit/87bb538bdbc4bb8848ed5791b7b0de84a= 026ebbe - Completed the rewrite of the way the compression property is handled, moving away from the initial approach of storing the compression property (enum zio_compress) and the level (uint64_t) separately. Previously we exposed the list of compression algorithms and levels to userland as the corresponding value from the enum in the lower 7 bits, and the level in the remaining upper bits. Then, as part of the property GET and SET IOCTLs, we read the separate compression=3D and compress_level=3D properties from the ZAP and returned the combined value= , or split the combined value into those two separate properties. This worked but caused a lot of headache around property inheritance. Instead I've changed to doing the combine/split when reading/writing from the dataset properties ZAP, via the compression_changed_cb() function. So the properties ZAP contains the combined value (lower 7 bits are the compression algorithm, as defined in the enum zio_compress, and the upper bits are the compression level). Elsewhere in ZFS we keep the two values separate (os_compress and os_complevel, and related variables in all of the different parts of ZFS). So now, inheritance of the property is handled correctly, and avoids issues where a dataset with compression=3Dzstd-12, would say 'inherited from' a dataset with zstd at some other compression level (since both actually just had compression=3Dzstd, but different compress_level=3D val= ues). I have also further extended zdb to inspect the compression settings when looking at an object: https://github.com/allanjude/zfs/commit/3fef3c83b8ce90149110ed989bd9fd3e2= 89798e0 I am still working on a solution for setting the zstd feature flag to 'active' as soon as it is set, rather than only once a block is born. Additionally, I am investigating how to best handle the fact that embedded block pointers compressed with ZSTD will make 'zfs send -e' streams backwards incompatible, without a way for the user to opt-out of sending a stream that contains zstd compressed blocks that the receiving side may not be able to read. The same can be said for 'zfs send -c' as well. I am open to ideas on how best to handle this. I have thought about only sending ZSTD compressed blocks if the user specifies the -Z (--zstd) flag, but this can lead to confusion where using -c without -Z would have to either error out, or send the ZSTD compressed blocks uncompressed. This project is sponsored by the FreeBSD Foundation. --=20 Allan Jude --3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P-- --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfA7ysAAoJEBmVNT4SmAt+6vIP/A0FQvM5pFtur2ReGoFt0HxC Hj58PfJCBIPDfcwrdkBN4LsfRzb2jymlF8PyVqzvHkC0scGyaDiOSWu2EjEgtYlD xpCKByvfguEiME2vZ0Dlh1571SA0go0jJdS76RkehvKDX6i6EekWksCyUbIX4LDs ft2am63KFz9WKAG5PxOfZn94mKLODbfxzwf9KCpUAz0zwvz5ej07kyFblhJCiR35 cGvQOPH9XlOibos97XZsL2XEWt2gsxJv9qFslvAjOzK6XAtQfjLw52XaOSOp3Abe iEFxMLWm8i0s14BkgPh/bHmv7wB/TN/nwgIwoNeLMNQpCPZLYLg069sBY2gwNaWo yZ9LK836fzbJ/DbBRlaZzrI+Nrs8Bp28ZSmCGDD6WX/wpt2gyPKLJnrAKd9XNm0E WpqD1omGS1pCU/NiKoXpFbbid5S1O40bF4NNF0hIAVaPYlEES/M3mcdhg4ONwxa9 LdLcALGsyGrAzXwG6UmZQfcr80Tf6IGwiNsGEtnV4+FOK2yZu6MB1KA3v3FpLjqZ uASOqHxLMwILWWoHtAD7ggeX2BelfFfkDU3PVMhPs75SNEBNtAD6eouIxKzrIpOG /RMHbXEFr1Grba7qJ96f9avGvqBSndYUbfL4+3C9L9Rr+uhwiUBju0VQBZtT3IP5 PvEjG3n4Ty/q0/jJApDF =BenT -----END PGP SIGNATURE----- --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO-- From owner-freebsd-fs@freebsd.org Tue Jul 7 06:21:57 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADFC535705C for ; Tue, 7 Jul 2020 06:21:57 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1C5J1H6Yz47Mq for ; Tue, 7 Jul 2020 06:21:55 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f174.google.com with SMTP id e8so14259334ljb.0 for ; Mon, 06 Jul 2020 23:21:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=DsirsDn8YG/0c1K+u2/z4t2EEHGQoZEXasJS6/Iqc/M=; b=B998lPhqEnF0uJHY1Y4LDq7AGq9SurRsgHUzBF+iSK+pj+rMKSs7FjAFYmT4EmIrNr eMhz/Gd1r23tnVez69YzvTRz5O+Ar0++wG2lrrxOtSYGtlCgjjyolt5nF2f9nf1dzFVi A5y4vyTwKF62Rf0fBReeOeO8umWQM0+LeAtTpEo2hWhi9eWJOFwmI0vPUUNgGK1yUW+X +i/WnSLuHpyytQlTrDEfbxAAiaZ5UUP/Ws62C3IAp/0GAiV5VFWzYFKee6Hk0rbBpLiU 9InRjQl1bjLYifZ7HDhFSMK0XvA5FODn+FxKiC7iUHgye4xH+87C7MX85/LVOztX2syD NLDQ== X-Gm-Message-State: AOAM5316DnjmF/Kzjbg4myfOV81PfQTyBltad5d1bevyyt0JyCESLca9 q3ZIljp5FUwwbffTNaP90kgmOtSpMuk= X-Google-Smtp-Source: ABdhPJyceKJ8B+rFvF+ShAJZ6EqDvHL22/KadQ4BIK3AlWROKREWVF8rmwvop4A1I3U2718zqU1E6w== X-Received: by 2002:a2e:7c07:: with SMTP id x7mr18566921ljc.166.1594102913793; Mon, 06 Jul 2020 23:21:53 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id s8sm6598322ljh.74.2020.07.06.23.21.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Jul 2020 23:21:52 -0700 (PDT) Subject: Re: gptzfsboot targeting wrong vdev To: Christian Kratzer , freebsd-fs References: From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> Date: Tue, 7 Jul 2020 09:21:50 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4B1C5J1H6Yz47Mq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.174 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.16 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.97)[-0.971]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.24)[-0.242]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.174:from]; NEURAL_HAM_MEDIUM(-0.95)[-0.951]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.174:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[93.72.151.96:received] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 06:21:57 -0000 On 06/07/2020 16:10, Christian Kratzer wrote: > Hi, > > I have a couple of freebsd based zfs servers on 12.1-RELEASE-p6 that have zfs > root and separate geli procted zfs data pools. > > I have been booting these off usb sticks or sdcards with a copy of /boot > and the geli keys and then subsequently mounting root from the zfsroot pool. > >     root@zfs1:/home/ck # zpool list >     NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  > ALTROOT >     zp1    3.62T   242G  3.39T        -         -     3%     6%  1.00x  ONLINE  - >     zp2    21.8T  13.2T  8.51T        -         -    13%    60%  1.00x  ONLINE  - >     zroot   127G   112G  15.1G        -         -    30%    88%  1.00x  ONLINE  - >     root@zfs1:/home/ck # > > In above setup zroot is an unencrypted two ssd vdev with just the OS on it. > > The disks for zp1 and zp2 are encrypted using geli via keys provided in > loader.conf. > > Above setup is working fine whilst I keep /boot on the usb stick. > > I have tried again to get rid of the usb stick and booting directly from the > mirror on zroot. > > I have following in my loader.conf > >     # zfs >     zfs_load="YES" >     vfs.root.mountfrom="zfs:zroot/ROOT/default" >     vfs.zfs.boot.primary_pool="17582686064334413803" >     vfs.zfs.boot.primary_vdev="9317482241260982593" > > I got the guid values using > >     root@zfs1:/usr/src # zpool get guid zroot >     NAME   PROPERTY  VALUE  SOURCE >     zroot  guid      17582686064334413803  default > >     root@zfs1:/usr/src # zfs get guid zroot/ROOT/default >     NAME                PROPERTY  VALUE  SOURCE >     zroot/ROOT/default  guid      9317482241260982593  - > > > I also have following setup on the devices that are part of zroot > >     root@zfs1:/home/ck # gpart show ada0 >     =>       40  468862048  ada0  GPT  (224G) >          40       1024     1  freebsd-boot  (512K) >            1064  134217728     2  freebsd-swap  (64G) >       134218792   33554432     3  freebsd-zfs  (16G) >       167773224   33554432     4  freebsd-zfs  (16G) >       201327656  267534424     5  freebsd-zfs  (128G) >       468862080          8        - free -  (4.0K) > >     root@zfs1:/home/ck # > > When booting from ada0 I get following: > >     ZFS: i/o error - all block copies unavailable >     ZFS: can't read MOS of pool zp1 >     gptzfsboot: failed to mount default pool zp1 > >     FreeBSD/x86 boot > > At least it seems that my bios is loading the zfs loader from the correct gpt > partition. > > But I cannot figure out why it is still trying to boot from zp1 and not zroot. > > I have not been able to break out into the loader via serial console to inspect > if those vfs values are really there. > > I do see them if I boot via my usb stick copy of /boot > >     root@zfs1:/home/ck # kenv | grep vfs >     vfs.root.mountfrom="zfs:zroot/ROOT/default" >     vfs.zfs.boot.primary_pool="17582686064334413803" >     vfs.zfs.boot.primary_vdev="9317482241260982593" >     root@zfs1:/home/ck # > > This should all be unrelated to geli as I only need the zroot pool mounted at > this point in time which is unencrypted. > > I am a bit lost at this stage and would have following questions if anybody can > help me: > > 1. Am I getting the correct guid values for the primary_pool and _vdev or am I > misundestanding the requirements ? You are. Your vdev guid is obviously incorrect. I am not sure how you decided that a dataset guid is a vdev guid. Also, those variables are for internal use only. They are a part of communication between an early boot block and loader. > 2. How can I break into the loader to inspect or set kenv values ? That's an interesting question. I always see a menu and simply press a button to go the loader prompt. Maybe you wanted to ask something else? Do you see anything from loader on serial console at all? Do you have loader serial console support enabled? > 3. Should I perhaps try exporting zp1 and zp2 so as only to have zroot visible ? gptzfsboot(8) describes the pool discovery algorithm. > 4. Can I try setting the vfs variables from /config ? How would the syntax for > that be ? Assuming you mean /boot.config or /boot/config -- no. You can read boot(8) about all supported options. It could be helpful with respect to serial console as well. > Anything else I can do or check to get this to work ? There are no that many things that can be controlled with respect to ZFS boot. BIOS boot device priority and order of partitions is what controls which pool should be used for boot. You provided output of gpart show ada0, there are more than zfs partitions in it. You did not explain which one is which. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Jul 7 14:59:32 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 430A7364637 for ; Tue, 7 Jul 2020 14:59:32 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1QZW6N4Xz3S0J for ; Tue, 7 Jul 2020 14:59:31 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 276B92878A for ; Tue, 7 Jul 2020 14:59:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 276B92878A To: freebsd-fs@freebsd.org References: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> From: Allan Jude Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: gptzfsboot targeting wrong vdev Message-ID: <78024f0d-4889-713e-15a5-56ec6d8d82b3@freebsd.org> Date: Tue, 7 Jul 2020 10:59:24 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX" X-Rspamd-Queue-Id: 4B1QZW6N4Xz3S0J X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 14:59:32 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX Content-Type: multipart/mixed; boundary="zSXW4eehrhC2Lu0NvwCB4OfT5wmUMX1M4"; protected-headers="v1" From: Allan Jude To: freebsd-fs@freebsd.org Message-ID: <78024f0d-4889-713e-15a5-56ec6d8d82b3@freebsd.org> Subject: Re: gptzfsboot targeting wrong vdev References: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> In-Reply-To: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> --zSXW4eehrhC2Lu0NvwCB4OfT5wmUMX1M4 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2020-07-07 02:21, Andriy Gapon wrote: > On 06/07/2020 16:10, Christian Kratzer wrote: >> Hi, >> >> I have a couple of freebsd based zfs servers on 12.1-RELEASE-p6 that h= ave zfs >> root and separate geli procted zfs data pools. >> >> I have been booting these off usb sticks or sdcards with a copy of /bo= ot >> and the geli keys and then subsequently mounting root from the zfsroot= pool. >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # zpool list >> =C2=A0=C2=A0=C2=A0=C2=A0NAME=C2=A0=C2=A0=C2=A0 SIZE=C2=A0 ALLOC=C2=A0=C2= =A0 FREE=C2=A0 CKPOINT=C2=A0 EXPANDSZ=C2=A0=C2=A0 FRAG=C2=A0=C2=A0=C2=A0 = CAP=C2=A0 DEDUP=C2=A0 HEALTH=C2=A0 >> ALTROOT >> =C2=A0=C2=A0=C2=A0=C2=A0zp1=C2=A0=C2=A0=C2=A0 3.62T=C2=A0=C2=A0 242G=C2= =A0 3.39T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0 3%=C2=A0=C2=A0=C2=A0= =C2=A0 6%=C2=A0 1.00x=C2=A0 ONLINE=C2=A0 - >> =C2=A0=C2=A0=C2=A0=C2=A0zp2=C2=A0=C2=A0=C2=A0 21.8T=C2=A0 13.2T=C2=A0 = 8.51T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0 13%=C2=A0=C2=A0=C2=A0 60%=C2= =A0 1.00x=C2=A0 ONLINE=C2=A0 - >> =C2=A0=C2=A0=C2=A0=C2=A0zroot=C2=A0=C2=A0 127G=C2=A0=C2=A0 112G=C2=A0 = 15.1G=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0 30%=C2=A0=C2=A0=C2=A0 88%=C2= =A0 1.00x=C2=A0 ONLINE=C2=A0 - >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # >> >> In above setup zroot is an unencrypted two ssd vdev with just the OS o= n it. >> >> The disks for zp1 and zp2 are encrypted using geli via keys provided i= n >> loader.conf. >> >> Above setup is working fine whilst I keep /boot on the usb stick. >> >> I have tried again to get rid of the usb stick and booting directly fr= om the >> mirror on zroot. >> >> I have following in my loader.conf >> >> =C2=A0=C2=A0=C2=A0=C2=A0# zfs >> =C2=A0=C2=A0=C2=A0=C2=A0zfs_load=3D"YES" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.root.mountfrom=3D"zfs:zroot/ROOT/default" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_pool=3D"17582686064334413= 803" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_vdev=3D"93174822412609825= 93" >> >> I got the guid values using >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/usr/src # zpool get guid zroot >> =C2=A0=C2=A0=C2=A0=C2=A0NAME=C2=A0=C2=A0 PROPERTY=C2=A0 VALUE=C2=A0 SO= URCE >> =C2=A0=C2=A0=C2=A0=C2=A0zroot=C2=A0 guid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 17582686064334413803=C2=A0 default >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/usr/src # zfs get guid zroot/ROOT/d= efault >> =C2=A0=C2=A0=C2=A0=C2=A0NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 PROPERTY=C2=A0 VALUE=C2=A0= SOURCE >> =C2=A0=C2=A0=C2=A0=C2=A0zroot/ROOT/default=C2=A0 guid=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 9317482241260982593=C2=A0 - >> >> >> I also have following setup on the devices that are part of zroot >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # gpart show ada0 >> =C2=A0=C2=A0=C2=A0=C2=A0=3D>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 40=C2= =A0 468862048=C2=A0 ada0=C2=A0 GPT=C2=A0 (224G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 40=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 1024=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0 freebsd-boot=C2=A0 (= 512K) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1064=C2=A0= 134217728=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0 freebsd-swap=C2=A0 (64G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 134218792=C2=A0=C2=A0 33554432=C2=A0=C2= =A0=C2=A0=C2=A0 3=C2=A0 freebsd-zfs=C2=A0 (16G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 167773224=C2=A0=C2=A0 33554432=C2=A0=C2= =A0=C2=A0=C2=A0 4=C2=A0 freebsd-zfs=C2=A0 (16G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 201327656=C2=A0 267534424=C2=A0=C2=A0=C2= =A0=C2=A0 5=C2=A0 freebsd-zfs=C2=A0 (128G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 468862080=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - fr= ee -=C2=A0 (4.0K) >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # >> >> When booting from ada0 I get following: >> >> =C2=A0=C2=A0=C2=A0=C2=A0ZFS: i/o error - all block copies unavailable >> =C2=A0=C2=A0=C2=A0=C2=A0ZFS: can't read MOS of pool zp1 >> =C2=A0=C2=A0=C2=A0=C2=A0gptzfsboot: failed to mount default pool zp1 >> >> =C2=A0=C2=A0=C2=A0=C2=A0FreeBSD/x86 boot >> So, just to be clear, at this point you have not loaded the boot loader yet. You are in the bootstrap (gptzfsboot), and it is unable to load the loader. I think it just looks at the first 'freebsd-zfs' type'd partition. However, if zp1 is GELI encrypted, it shouldn't be able to even tell the name of the pool. You might try changing the partition type of the boots you are not booting from, to 'freebsd-vinum' or something other than 'freebsd-zfs' so that gptzfsboot only sees 1 'freebsd-zfs' to boot from. >> At least it seems that my bios is loading the zfs loader from the corr= ect gpt >> partition. >> >> But I cannot figure out why it is still trying to boot from zp1 and no= t zroot. >> >> I have not been able to break out into the loader via serial console t= o inspect >> if those vfs values are really there. >> >> I do see them if I boot via my usb stick copy of /boot >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # kenv | grep vfs >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.root.mountfrom=3D"zfs:zroot/ROOT/default" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_pool=3D"17582686064334413= 803" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_vdev=3D"93174822412609825= 93" >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # >> >> This should all be unrelated to geli as I only need the zroot pool mou= nted at >> this point in time which is unencrypted. >> >> I am a bit lost at this stage and would have following questions if an= ybody can >> help me: >> >> 1. Am I getting the correct guid values for the primary_pool and _vdev= or am I >> misundestanding the requirements ? >=20 > You are. Your vdev guid is obviously incorrect. I am not sure how you= decided > that a dataset guid is a vdev guid. > Also, those variables are for internal use only. They are a part of > communication between an early boot block and loader. >=20 >> 2. How can I break into the loader to inspect or set kenv values ? >=20 > That's an interesting question. > I always see a menu and simply press a button to go the loader prompt. > Maybe you wanted to ask something else? > Do you see anything from loader on serial console at all? > Do you have loader serial console support enabled? >=20 >> 3. Should I perhaps try exporting zp1 and zp2 so as only to have zroot= visible ? >=20 > gptzfsboot(8) describes the pool discovery algorithm. >=20 >> 4. Can I try setting the vfs variables from /config ? How would the sy= ntax for >> that be ? >=20 > Assuming you mean /boot.config or /boot/config -- no. You can read boo= t(8) > about all supported options. It could be helpful with respect to seria= l console > as well. >=20 >> Anything else I can do or check to get this to work ? >=20 > There are no that many things that can be controlled with respect to ZF= S boot. > BIOS boot device priority and order of partitions is what controls whic= h pool > should be used for boot. >=20 > You provided output of gpart show ada0, there are more than zfs partiti= ons in > it. You did not explain which one is which. >=20 --=20 Allan Jude --zSXW4eehrhC2Lu0NvwCB4OfT5wmUMX1M4-- --BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfBI3MAAoJEBmVNT4SmAt+sFEP/R66uLtT3NDh3a5m5nOSagMp jFPvlC5wE+p3D0B4OnWLcXZuMTaYLr5bEZW/kgthxsPHbp3LdbxsqubudRziIcrr VxAyzTWW5K3PET6qhaRx15lQGRActx0A8bZ4VnffZWLaBxd3k4lC4/bc3plvvxrM 83a3fgBPGo1EjC/xdL5tEtM83GqXOi054LBCQUJcySHjCxqucpE0J3NxRdIhFJlQ SPPpshPtSEKUnMk+qa39WUnM37XTiWsFFFXCr80pyV6MNlbuQlb+EAXSD5GVPFL6 yLv0QeAzlXOW2ixT4oQ+Dnkj5ovRn24mNGuIc/+7fhbspdtiqkifTb0MZrQjb4AV 9W+A3tlAyW96+MjxMBoPGLit1vzS1AnKnUeAI6N30Bmq7ULw701LQo0A/KWFlT72 96x/QCiVYY1kflNHznV3DjJMe9Aix3ZxGIiFBt+9LO3w4SF6ddRtANNSvW1tetSr B/u1HmU+Ofy+guCnZrtRuho7JM35PxFIgChL7N7KxBM1WjOype1FbCo/c2+Hu++I Ew+LYZiyGhgaR8FiG56N1t6KHz5y/kp9HDuGZV0Wn06hMg5sgVIWOqZWjoSuX0mB wjd8jUNktfSu8WTL9UoWhYVYj+ESQO4eJMS+G1s4Zb5tyjKq9O3B70l7f6fdL0/h OWwoihrtKgGuv4e38sJX =pzwv -----END PGP SIGNATURE----- --BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX-- From owner-freebsd-fs@freebsd.org Wed Jul 8 21:31:24 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 687C935531F; Wed, 8 Jul 2020 21:31:24 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B2CDD2BsLz45sn; Wed, 8 Jul 2020 21:31:24 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 295171BA9C; Wed, 8 Jul 2020 21:31:24 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-lj1-f181.google.com with SMTP id s9so55869586ljm.11; Wed, 08 Jul 2020 14:31:24 -0700 (PDT) X-Gm-Message-State: AOAM5309htfwCuYFXMlXk3RpsEF0VT9y/jIEZJ7c3twKLlgFQ+MDdkGo 0SyrtMgpjC2mjeWE/DgJRR3CaUwfU5/7CrPN0Ss= X-Google-Smtp-Source: ABdhPJzkPsOrt2Tp54Rq3g/9QQ8qqMdK0dEW7USpjDSngAx8RvmZQ5CXlDzQyEcuoybib73aQ7E0al6TIJhpnuBfg3A= X-Received: by 2002:a2e:943:: with SMTP id 64mr9555237ljj.445.1594243882646; Wed, 08 Jul 2020 14:31:22 -0700 (PDT) MIME-Version: 1.0 From: Matthew Macy Date: Wed, 8 Jul 2020 14:31:11 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: CFT for vendor openzfs To: freebsd-fs , freebsd-hackers@freebsd.org, freebsd-current Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2020 21:31:24 -0000 Checkout updated HEAD: % git clone https://github.com/mattmacy/networking.git -b projects/openzfs_vendor freebsd Checkout updated openzfs in to sys/contrib: % git clone https://github.com/zfsonfreebsd/ZoF.git -b projects/openzfs_vendor freebsd/sys/contrib/openzfs Build world and kernel with whatever your usual configuration is. Where possible the openzfs kmod is backward compatible with the cmd utils in HEAD so common operations work with existing tools and the new kmod. In the projects/openzfs_vendor branch of ZoF ozfs libraries are backward compatible with the zfs kmod in HEAD. Although ideally one would test this in a separate boot environment, the interoperability should allow one to rollback without too much difficulty. Thanks in advance for your time. -M From owner-freebsd-fs@freebsd.org Wed Jul 8 21:39:01 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 15E9D355B84; Wed, 8 Jul 2020 21:39:01 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B2CP06swrz4698; Wed, 8 Jul 2020 21:39:00 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id C9E481B97A; Wed, 8 Jul 2020 21:39:00 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-lj1-f173.google.com with SMTP id e8so13866ljb.0; Wed, 08 Jul 2020 14:39:00 -0700 (PDT) X-Gm-Message-State: AOAM533oPBHjaOhWxUQDUAkAC8m8IiXzWr5Fdj+NmkiCesIo2BaXA8gr +jfM8iGj9Kh0RXC8xY6tAR0JVBTFs84wZbSWFhE= X-Google-Smtp-Source: ABdhPJwL4BO4hAzjODcNu461p3QculXkL1tFgcDVJ0yVuQNmP6PstV61cERF9UhP1rNOWpZzYuZikcnz3+/5eAHIFnI= X-Received: by 2002:a05:651c:307:: with SMTP id a7mr33532041ljp.297.1594244339383; Wed, 08 Jul 2020 14:38:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Wed, 8 Jul 2020 14:38:48 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Further note - Re: CFT for vendor openzfs To: freebsd-fs , freebsd-hackers@freebsd.org, freebsd-current Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2020 21:39:01 -0000 Do NOT zpool upgrade unless you are willing to live without the ability to ever rollback to the legacy zfs kmod. On Wed, Jul 8, 2020 at 2:31 PM Matthew Macy wrote: > > Checkout updated HEAD: > % git clone https://github.com/mattmacy/networking.git -b > projects/openzfs_vendor freebsd > > Checkout updated openzfs in to sys/contrib: > % git clone https://github.com/zfsonfreebsd/ZoF.git -b > projects/openzfs_vendor freebsd/sys/contrib/openzfs > > Build world and kernel with whatever your usual configuration is. > Where possible the openzfs kmod is backward compatible with the cmd > utils in HEAD so common operations work with existing tools and the > new kmod. In the projects/openzfs_vendor branch of ZoF ozfs libraries > are backward compatible with the zfs kmod in HEAD. Although ideally > one would test this in a separate boot environment, the > interoperability should allow one to rollback without too much > difficulty. > > Thanks in advance for your time. > -M