From owner-freebsd-fs@freebsd.org Tue Jul 7 14:59:32 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 430A7364637 for ; Tue, 7 Jul 2020 14:59:32 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1QZW6N4Xz3S0J for ; Tue, 7 Jul 2020 14:59:31 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 276B92878A for ; Tue, 7 Jul 2020 14:59:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 276B92878A To: freebsd-fs@freebsd.org References: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> From: Allan Jude Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: gptzfsboot targeting wrong vdev Message-ID: <78024f0d-4889-713e-15a5-56ec6d8d82b3@freebsd.org> Date: Tue, 7 Jul 2020 10:59:24 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX" X-Rspamd-Queue-Id: 4B1QZW6N4Xz3S0J X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 14:59:32 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX Content-Type: multipart/mixed; boundary="zSXW4eehrhC2Lu0NvwCB4OfT5wmUMX1M4"; protected-headers="v1" From: Allan Jude To: freebsd-fs@freebsd.org Message-ID: <78024f0d-4889-713e-15a5-56ec6d8d82b3@freebsd.org> Subject: Re: gptzfsboot targeting wrong vdev References: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> In-Reply-To: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> --zSXW4eehrhC2Lu0NvwCB4OfT5wmUMX1M4 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2020-07-07 02:21, Andriy Gapon wrote: > On 06/07/2020 16:10, Christian Kratzer wrote: >> Hi, >> >> I have a couple of freebsd based zfs servers on 12.1-RELEASE-p6 that h= ave zfs >> root and separate geli procted zfs data pools. >> >> I have been booting these off usb sticks or sdcards with a copy of /bo= ot >> and the geli keys and then subsequently mounting root from the zfsroot= pool. >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # zpool list >> =C2=A0=C2=A0=C2=A0=C2=A0NAME=C2=A0=C2=A0=C2=A0 SIZE=C2=A0 ALLOC=C2=A0=C2= =A0 FREE=C2=A0 CKPOINT=C2=A0 EXPANDSZ=C2=A0=C2=A0 FRAG=C2=A0=C2=A0=C2=A0 = CAP=C2=A0 DEDUP=C2=A0 HEALTH=C2=A0 >> ALTROOT >> =C2=A0=C2=A0=C2=A0=C2=A0zp1=C2=A0=C2=A0=C2=A0 3.62T=C2=A0=C2=A0 242G=C2= =A0 3.39T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0 3%=C2=A0=C2=A0=C2=A0= =C2=A0 6%=C2=A0 1.00x=C2=A0 ONLINE=C2=A0 - >> =C2=A0=C2=A0=C2=A0=C2=A0zp2=C2=A0=C2=A0=C2=A0 21.8T=C2=A0 13.2T=C2=A0 = 8.51T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0 13%=C2=A0=C2=A0=C2=A0 60%=C2= =A0 1.00x=C2=A0 ONLINE=C2=A0 - >> =C2=A0=C2=A0=C2=A0=C2=A0zroot=C2=A0=C2=A0 127G=C2=A0=C2=A0 112G=C2=A0 = 15.1G=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0 30%=C2=A0=C2=A0=C2=A0 88%=C2= =A0 1.00x=C2=A0 ONLINE=C2=A0 - >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # >> >> In above setup zroot is an unencrypted two ssd vdev with just the OS o= n it. >> >> The disks for zp1 and zp2 are encrypted using geli via keys provided i= n >> loader.conf. >> >> Above setup is working fine whilst I keep /boot on the usb stick. >> >> I have tried again to get rid of the usb stick and booting directly fr= om the >> mirror on zroot. >> >> I have following in my loader.conf >> >> =C2=A0=C2=A0=C2=A0=C2=A0# zfs >> =C2=A0=C2=A0=C2=A0=C2=A0zfs_load=3D"YES" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.root.mountfrom=3D"zfs:zroot/ROOT/default" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_pool=3D"17582686064334413= 803" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_vdev=3D"93174822412609825= 93" >> >> I got the guid values using >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/usr/src # zpool get guid zroot >> =C2=A0=C2=A0=C2=A0=C2=A0NAME=C2=A0=C2=A0 PROPERTY=C2=A0 VALUE=C2=A0 SO= URCE >> =C2=A0=C2=A0=C2=A0=C2=A0zroot=C2=A0 guid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 17582686064334413803=C2=A0 default >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/usr/src # zfs get guid zroot/ROOT/d= efault >> =C2=A0=C2=A0=C2=A0=C2=A0NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 PROPERTY=C2=A0 VALUE=C2=A0= SOURCE >> =C2=A0=C2=A0=C2=A0=C2=A0zroot/ROOT/default=C2=A0 guid=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 9317482241260982593=C2=A0 - >> >> >> I also have following setup on the devices that are part of zroot >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # gpart show ada0 >> =C2=A0=C2=A0=C2=A0=C2=A0=3D>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 40=C2= =A0 468862048=C2=A0 ada0=C2=A0 GPT=C2=A0 (224G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 40=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 1024=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0 freebsd-boot=C2=A0 (= 512K) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1064=C2=A0= 134217728=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0 freebsd-swap=C2=A0 (64G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 134218792=C2=A0=C2=A0 33554432=C2=A0=C2= =A0=C2=A0=C2=A0 3=C2=A0 freebsd-zfs=C2=A0 (16G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 167773224=C2=A0=C2=A0 33554432=C2=A0=C2= =A0=C2=A0=C2=A0 4=C2=A0 freebsd-zfs=C2=A0 (16G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 201327656=C2=A0 267534424=C2=A0=C2=A0=C2= =A0=C2=A0 5=C2=A0 freebsd-zfs=C2=A0 (128G) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 468862080=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - fr= ee -=C2=A0 (4.0K) >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # >> >> When booting from ada0 I get following: >> >> =C2=A0=C2=A0=C2=A0=C2=A0ZFS: i/o error - all block copies unavailable >> =C2=A0=C2=A0=C2=A0=C2=A0ZFS: can't read MOS of pool zp1 >> =C2=A0=C2=A0=C2=A0=C2=A0gptzfsboot: failed to mount default pool zp1 >> >> =C2=A0=C2=A0=C2=A0=C2=A0FreeBSD/x86 boot >> So, just to be clear, at this point you have not loaded the boot loader yet. You are in the bootstrap (gptzfsboot), and it is unable to load the loader. I think it just looks at the first 'freebsd-zfs' type'd partition. However, if zp1 is GELI encrypted, it shouldn't be able to even tell the name of the pool. You might try changing the partition type of the boots you are not booting from, to 'freebsd-vinum' or something other than 'freebsd-zfs' so that gptzfsboot only sees 1 'freebsd-zfs' to boot from. >> At least it seems that my bios is loading the zfs loader from the corr= ect gpt >> partition. >> >> But I cannot figure out why it is still trying to boot from zp1 and no= t zroot. >> >> I have not been able to break out into the loader via serial console t= o inspect >> if those vfs values are really there. >> >> I do see them if I boot via my usb stick copy of /boot >> >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # kenv | grep vfs >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.root.mountfrom=3D"zfs:zroot/ROOT/default" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_pool=3D"17582686064334413= 803" >> =C2=A0=C2=A0=C2=A0=C2=A0vfs.zfs.boot.primary_vdev=3D"93174822412609825= 93" >> =C2=A0=C2=A0=C2=A0=C2=A0root@zfs1:/home/ck # >> >> This should all be unrelated to geli as I only need the zroot pool mou= nted at >> this point in time which is unencrypted. >> >> I am a bit lost at this stage and would have following questions if an= ybody can >> help me: >> >> 1. Am I getting the correct guid values for the primary_pool and _vdev= or am I >> misundestanding the requirements ? >=20 > You are. Your vdev guid is obviously incorrect. I am not sure how you= decided > that a dataset guid is a vdev guid. > Also, those variables are for internal use only. They are a part of > communication between an early boot block and loader. >=20 >> 2. How can I break into the loader to inspect or set kenv values ? >=20 > That's an interesting question. > I always see a menu and simply press a button to go the loader prompt. > Maybe you wanted to ask something else? > Do you see anything from loader on serial console at all? > Do you have loader serial console support enabled? >=20 >> 3. Should I perhaps try exporting zp1 and zp2 so as only to have zroot= visible ? >=20 > gptzfsboot(8) describes the pool discovery algorithm. >=20 >> 4. Can I try setting the vfs variables from /config ? How would the sy= ntax for >> that be ? >=20 > Assuming you mean /boot.config or /boot/config -- no. You can read boo= t(8) > about all supported options. It could be helpful with respect to seria= l console > as well. >=20 >> Anything else I can do or check to get this to work ? >=20 > There are no that many things that can be controlled with respect to ZF= S boot. > BIOS boot device priority and order of partitions is what controls whic= h pool > should be used for boot. >=20 > You provided output of gpart show ada0, there are more than zfs partiti= ons in > it. You did not explain which one is which. >=20 --=20 Allan Jude --zSXW4eehrhC2Lu0NvwCB4OfT5wmUMX1M4-- --BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfBI3MAAoJEBmVNT4SmAt+sFEP/R66uLtT3NDh3a5m5nOSagMp jFPvlC5wE+p3D0B4OnWLcXZuMTaYLr5bEZW/kgthxsPHbp3LdbxsqubudRziIcrr VxAyzTWW5K3PET6qhaRx15lQGRActx0A8bZ4VnffZWLaBxd3k4lC4/bc3plvvxrM 83a3fgBPGo1EjC/xdL5tEtM83GqXOi054LBCQUJcySHjCxqucpE0J3NxRdIhFJlQ SPPpshPtSEKUnMk+qa39WUnM37XTiWsFFFXCr80pyV6MNlbuQlb+EAXSD5GVPFL6 yLv0QeAzlXOW2ixT4oQ+Dnkj5ovRn24mNGuIc/+7fhbspdtiqkifTb0MZrQjb4AV 9W+A3tlAyW96+MjxMBoPGLit1vzS1AnKnUeAI6N30Bmq7ULw701LQo0A/KWFlT72 96x/QCiVYY1kflNHznV3DjJMe9Aix3ZxGIiFBt+9LO3w4SF6ddRtANNSvW1tetSr B/u1HmU+Ofy+guCnZrtRuho7JM35PxFIgChL7N7KxBM1WjOype1FbCo/c2+Hu++I Ew+LYZiyGhgaR8FiG56N1t6KHz5y/kp9HDuGZV0Wn06hMg5sgVIWOqZWjoSuX0mB wjd8jUNktfSu8WTL9UoWhYVYj+ESQO4eJMS+G1s4Zb5tyjKq9O3B70l7f6fdL0/h OWwoihrtKgGuv4e38sJX =pzwv -----END PGP SIGNATURE----- --BgBbXekksRH5Jr7ZT4zeWdIStmoAGKzkX--