From owner-freebsd-stable@freebsd.org Fri Feb 19 18:03:13 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1139E535042 for ; Fri, 19 Feb 2021 18:03:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dhzvg6PgDz4nd0 for ; Fri, 19 Feb 2021 18:03:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x729.google.com with SMTP id x14so6415121qkm.2 for ; Fri, 19 Feb 2021 10:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=x6KmsYdKRX6hHxyyoDW1lS8Udt7X//8d5+MlvR5LP+E=; b=MIq+hUePONXkeM9A/1rEJSBP7n+QPv6N5nkZ6VKFfndI8yq+vorLhNVeifFl/cRvMD aC6u1m3MrdfXD0/xkAdhBZMFVW8LGkF+TK99WjFs0TlFHHss3banAEaY9DIaNVnYbXIq fqawEZZUmZ4xImyC4iqThHrf4IFKDS2EekO2dK/uTWX3gBBAxdY4L3zRyx6mkOj8XC8p bTCrBh9EKRHNMyTe7oR12IDKrNhEaCTxMu+8o2gqJjA7fVxdZteSCwvAcGtIMzar/L/f h9m/7/Ep0PHv/vqeG07SE6cO4OzF1a0OhmoVTFqNM7Ouv6pfkrOlwPYwFO1+QcHdliPk z/GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=x6KmsYdKRX6hHxyyoDW1lS8Udt7X//8d5+MlvR5LP+E=; b=q2R/QmvtgVI2e+TQiY0jqEupouU9l865kZjxSX3lOlmf4/MGqbiqllp1IVxZBV/VBw PEm56rAuXB/iLRlZLuZkgMDrajbq9NjnxYsN9z/UxLgAOKXr/qtdJez5KvK2QzVwYrSw 7ccLOIm1ZcBk+pputHa6ztePWsuNHHn7vf61dgPTK6BEFAD8H/QLw66vSPmyAOQdnIZe mE9pGgj1g84nXIL6Zyiwg07s3lBGff6FdqsG/MGnK1+SS6oADzpFDeV8Rvy2CwlbLpnU dt7XrSJtbclV+5wuuHGv1pMIXQ2XiPrs6aYXThDWajsix0T627Xs3IHnfix6u6y7Xx3m 28vg== X-Gm-Message-State: AOAM533EJRScCdmu8kCJBlW4+JjLlqPdmZ0ffGujpGq29yuC0iGVySN4 oxDD0JGKMjXq1ZdHCyJkGZikphq1s+a5BMSp6BfHSwN1dwgZ8A== X-Google-Smtp-Source: ABdhPJzOxVLldr6vW63Z81wH0OYXfz1DGQ0wZxaG2agj+w6H57jAK4AZ53irb3v0ycg7P6an0uSX+pTEe5zYS8J/hnM= X-Received: by 2002:a05:620a:12d1:: with SMTP id e17mr10379641qkl.89.1613757790345; Fri, 19 Feb 2021 10:03:10 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 19 Feb 2021 11:02:59 -0700 Message-ID: Subject: Re: FreeBSD 13/stable and zpool upgrade To: David Marec Cc: freebsd-stable X-Rspamd-Queue-Id: 4Dhzvg6PgDz4nd0 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=MIq+hUeP; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::729) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; MAILMAN_DEST(0.00)[freebsd-stable]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::729:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::729:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::729:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2021 18:03:13 -0000 On Fri, Feb 19, 2021 at 8:13 AM David Marec wrote: > I have just upgrade one machine from 12-stable to 13-stable. > > Everything runs fine until the main ZFS pool was upgraded. > > Then the box stopped booting. > > Thanks to a FreeBSD-13 Beta2 usb stick, I was able to fix the issue by > copying `BOOTx64.efi` from the stick to the hard-drive. > > Looking to `/boot` to check out for a new EFI boot-loader available, I > noticed that `boot1.efifat` was removed, but `gptboot.efifat` is still > there. > > Unfortunately it contains an old version of the boot loader: > > root@machine:/boot # mdconfig -f gptboot.efifat -u 0 > root@machine:/boot # mount -t msdosfs /dev/md0 /mnt > root@machine:/boot # ll /mnt/efi/boot/ > total 385 > -rwxr-xr-x 1 root wheel 393216 Apr 16 2018 BOOTx64.efi* > -rwxr-xr-x 1 root wheel 12 Apr 16 2018 startup.nsh* > > The boot firmware I got from the 13-Beta2 installer is different: > > -rwxr-xr-x 1 root wheel 894464 Feb 18 21:07 BOOTx64.efi* > > What did I missed ? An error message would be more helpful :) However, I think I know what's going on. The old boot blocks have a filter on the enabled features of the zpool. If anything that's not in the allowed list is enabled, it won't boot off that pool. I suspect that your 'zpool upgrade' enabled things that weren't enabled before. This caused the old boot blocks to no longer work. We should be better about upgrading boot blocks, but EFI is kinda new and kinda different than the other out-of-root-filesystem boot blocks we've had in the past, so there's still some rough edges. You should be able to update bootx64.efi to the one that's in the 13-beta2 installer and the system should boot off your zpool. Warner