From owner-freebsd-fs@freebsd.org Tue Jul 7 06:21:57 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADFC535705C for ; Tue, 7 Jul 2020 06:21:57 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1C5J1H6Yz47Mq for ; Tue, 7 Jul 2020 06:21:55 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f174.google.com with SMTP id e8so14259334ljb.0 for ; Mon, 06 Jul 2020 23:21:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=DsirsDn8YG/0c1K+u2/z4t2EEHGQoZEXasJS6/Iqc/M=; b=B998lPhqEnF0uJHY1Y4LDq7AGq9SurRsgHUzBF+iSK+pj+rMKSs7FjAFYmT4EmIrNr eMhz/Gd1r23tnVez69YzvTRz5O+Ar0++wG2lrrxOtSYGtlCgjjyolt5nF2f9nf1dzFVi A5y4vyTwKF62Rf0fBReeOeO8umWQM0+LeAtTpEo2hWhi9eWJOFwmI0vPUUNgGK1yUW+X +i/WnSLuHpyytQlTrDEfbxAAiaZ5UUP/Ws62C3IAp/0GAiV5VFWzYFKee6Hk0rbBpLiU 9InRjQl1bjLYifZ7HDhFSMK0XvA5FODn+FxKiC7iUHgye4xH+87C7MX85/LVOztX2syD NLDQ== X-Gm-Message-State: AOAM5316DnjmF/Kzjbg4myfOV81PfQTyBltad5d1bevyyt0JyCESLca9 q3ZIljp5FUwwbffTNaP90kgmOtSpMuk= X-Google-Smtp-Source: ABdhPJyceKJ8B+rFvF+ShAJZ6EqDvHL22/KadQ4BIK3AlWROKREWVF8rmwvop4A1I3U2718zqU1E6w== X-Received: by 2002:a2e:7c07:: with SMTP id x7mr18566921ljc.166.1594102913793; Mon, 06 Jul 2020 23:21:53 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id s8sm6598322ljh.74.2020.07.06.23.21.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Jul 2020 23:21:52 -0700 (PDT) Subject: Re: gptzfsboot targeting wrong vdev To: Christian Kratzer , freebsd-fs References: From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <9400f5f0-e267-932c-b1ce-8436748cf2c0@FreeBSD.org> Date: Tue, 7 Jul 2020 09:21:50 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4B1C5J1H6Yz47Mq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.174 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.16 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.97)[-0.971]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.24)[-0.242]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.174:from]; NEURAL_HAM_MEDIUM(-0.95)[-0.951]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.174:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[93.72.151.96:received] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 06:21:57 -0000 On 06/07/2020 16:10, Christian Kratzer wrote: > Hi, > > I have a couple of freebsd based zfs servers on 12.1-RELEASE-p6 that have zfs > root and separate geli procted zfs data pools. > > I have been booting these off usb sticks or sdcards with a copy of /boot > and the geli keys and then subsequently mounting root from the zfsroot pool. > >     root@zfs1:/home/ck # zpool list >     NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  > ALTROOT >     zp1    3.62T   242G  3.39T        -         -     3%     6%  1.00x  ONLINE  - >     zp2    21.8T  13.2T  8.51T        -         -    13%    60%  1.00x  ONLINE  - >     zroot   127G   112G  15.1G        -         -    30%    88%  1.00x  ONLINE  - >     root@zfs1:/home/ck # > > In above setup zroot is an unencrypted two ssd vdev with just the OS on it. > > The disks for zp1 and zp2 are encrypted using geli via keys provided in > loader.conf. > > Above setup is working fine whilst I keep /boot on the usb stick. > > I have tried again to get rid of the usb stick and booting directly from the > mirror on zroot. > > I have following in my loader.conf > >     # zfs >     zfs_load="YES" >     vfs.root.mountfrom="zfs:zroot/ROOT/default" >     vfs.zfs.boot.primary_pool="17582686064334413803" >     vfs.zfs.boot.primary_vdev="9317482241260982593" > > I got the guid values using > >     root@zfs1:/usr/src # zpool get guid zroot >     NAME   PROPERTY  VALUE  SOURCE >     zroot  guid      17582686064334413803  default > >     root@zfs1:/usr/src # zfs get guid zroot/ROOT/default >     NAME                PROPERTY  VALUE  SOURCE >     zroot/ROOT/default  guid      9317482241260982593  - > > > I also have following setup on the devices that are part of zroot > >     root@zfs1:/home/ck # gpart show ada0 >     =>       40  468862048  ada0  GPT  (224G) >          40       1024     1  freebsd-boot  (512K) >            1064  134217728     2  freebsd-swap  (64G) >       134218792   33554432     3  freebsd-zfs  (16G) >       167773224   33554432     4  freebsd-zfs  (16G) >       201327656  267534424     5  freebsd-zfs  (128G) >       468862080          8        - free -  (4.0K) > >     root@zfs1:/home/ck # > > When booting from ada0 I get following: > >     ZFS: i/o error - all block copies unavailable >     ZFS: can't read MOS of pool zp1 >     gptzfsboot: failed to mount default pool zp1 > >     FreeBSD/x86 boot > > At least it seems that my bios is loading the zfs loader from the correct gpt > partition. > > But I cannot figure out why it is still trying to boot from zp1 and not zroot. > > I have not been able to break out into the loader via serial console to inspect > if those vfs values are really there. > > I do see them if I boot via my usb stick copy of /boot > >     root@zfs1:/home/ck # kenv | grep vfs >     vfs.root.mountfrom="zfs:zroot/ROOT/default" >     vfs.zfs.boot.primary_pool="17582686064334413803" >     vfs.zfs.boot.primary_vdev="9317482241260982593" >     root@zfs1:/home/ck # > > This should all be unrelated to geli as I only need the zroot pool mounted at > this point in time which is unencrypted. > > I am a bit lost at this stage and would have following questions if anybody can > help me: > > 1. Am I getting the correct guid values for the primary_pool and _vdev or am I > misundestanding the requirements ? You are. Your vdev guid is obviously incorrect. I am not sure how you decided that a dataset guid is a vdev guid. Also, those variables are for internal use only. They are a part of communication between an early boot block and loader. > 2. How can I break into the loader to inspect or set kenv values ? That's an interesting question. I always see a menu and simply press a button to go the loader prompt. Maybe you wanted to ask something else? Do you see anything from loader on serial console at all? Do you have loader serial console support enabled? > 3. Should I perhaps try exporting zp1 and zp2 so as only to have zroot visible ? gptzfsboot(8) describes the pool discovery algorithm. > 4. Can I try setting the vfs variables from /config ? How would the syntax for > that be ? Assuming you mean /boot.config or /boot/config -- no. You can read boot(8) about all supported options. It could be helpful with respect to serial console as well. > Anything else I can do or check to get this to work ? There are no that many things that can be controlled with respect to ZFS boot. BIOS boot device priority and order of partitions is what controls which pool should be used for boot. You provided output of gpart show ada0, there are more than zfs partitions in it. You did not explain which one is which. -- Andriy Gapon