From owner-freebsd-fs@freebsd.org Mon Nov 11 06:10:38 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1B2BB1AC2E2 for ; Mon, 11 Nov 2019 06:10:38 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BL8X4Shwz3K1X for ; Mon, 11 Nov 2019 06:10:36 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id xAB6AKnf020621 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 11 Nov 2019 17:10:26 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id xAB6AFwS069202 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Nov 2019 17:10:15 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id xAB6AFvv069201; Mon, 11 Nov 2019 17:10:15 +1100 (AEDT) (envelope-from peter) Date: Mon, 11 Nov 2019 17:10:15 +1100 From: Peter Jeremy To: Jonathan Anderson Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111061015.GA50716@server.rulingia.com> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47BL8X4Shwz3K1X X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-7.71 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[rulingia.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-3.31)[ip: (-9.91), ipnet: 2001:19f0:5800::/38(-4.95), asn: 20473(-1.65), country: US(-0.05)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 06:10:38 -0000 --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2019-Nov-10 22:16:26 -0330, Jonathan Anderson = wrote: >I=E2=80=99ve gone and done it: I upgraded a key ZFS-on-root machine from 1= 1.2 to >12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on >this list might be able to help me figure out what's wrong and what I can >do about it? Based on your symptoms, it sounds like you might have a corrupt zpool.cache. /boot/zfs/zpool.cache should be rewritten on every boot but I had one system where that wasn't occurring and a FreeBSD upgrade (I don't currently recall the actual versions) resulted in more thorough validation checks, which failed. Can you share your actual layout ("gpart show", "zpool status", details of the non-partitioned disks, etc) - that might help us identify a problem. >It looks like the ZFS code in the bootloader can't find anything in my root >directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If >I boot a rescue image from USB I can mount everything in the pool (`zpool >import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs >zroot/ROOT/default` from the loader prompt it gives me an empty result (so, >e.g., no /boot). Booting fails with messages such as, "i/o error - all blo= ck >copies unavailable". If you boot from a rescue image and either delete (or rename) your existing zpool.cache or run # zpool set cachefile=3D/mnt/boot/zfs/zpool.cache zroot (where the path cachefile path maps to /boot/zfs/zpool.cache at boot), does that help? >My pool consists of three mirrored vdevs, in which the first mirror uses G= PT > partitioning (for the boot partitions) and the other two mirrors use >whole disks. Whole disks are not recommended for anything other than building partitions in. Ideally, you want all the disks to have GPT (or similar) partitions. If you don't need anything else on the disk, just create a single partition occupying the entire disk[1]. (I'd also recommend having a small root zpool that is a single, preferably mirrored, vdev, rather than a large root spread over multiple vdevs). >I recall reading somewhere that the bootloader ZFS code doesn't like >non-partition-based >vdevs... is that true? If so, perhaps the issue is that my upgrade caused >/boot to live on one of the newer whole-disk-based mirrors, hiding it from >the bootloader's view? That might be possible though it's not clear why it wouldn't have caused a problem in the past. Note that the bootloader is performing I/O via the BIOS so if the BIOS can't see the disks, the bootloader won't be able to read them. >> partitions with bootcode and file systems needed for booting can be >added. This allows booting from disks that are also members of a pool. >There is no performance penalty on FreeBSD when using a partition rather >than a whole disk Actually, Solaris might insist that you add a whole disk but, under the hood, ZFS actually partitions the disk with a single partition occupying the entire disk. >The Handbook suggests that it's possible to break disks into multiple >partitions that are added separately to a vdev, but is this a sensible >thing to do? Is there any other ZFS lore that hasn't made it to the >Handbook but that ought to be kept in mind from the outset? You can have multiple partitions on a disk and put different partitions into different vdevs but there's no point in having different partitions on the same disk in the same vdev - that will reduce both performance and resilience. [1] Though leaving a free GB or so can help if you need to replace a disk a= nd the replacement is slightly smaller. --=20 Peter Jeremy --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAl3I+0BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQ6yhAAj/hYwvytdS2IdtZfQtBcX29j2h5FRfIBMwBBkZJnB6hekCiOgGhPAuL0 D+FNAi1GY50NSe2qcgxvVOGqKlGcz5f8xoRUWlYg4u1y+cb1XwEXU87ujW4BoDWd kxnZaYozeB9IANa5FZwl9BdFcYTNIUzWA9Z2hc3ofCCEgG9Ckl+cnjlQNRXdbQAv cZEMof/mqNYfRtEXPQboAOcB5MVdZu3ytn/s5tlI2Nk9EiWyJ+zue6UZfBuTfraI 13Zt6AVWgXedyD1zDToAl5bfexlVuE8gjVZH3FOcGNfQgYvMUnnkzrK2/25vH3cJ 0rvSN53DU5zTIdyoI3xP80ck416gF/sVK/v5CyPmHdNkS1cqIHsCLfEOCfh+9PtP fNULo3LQf/cuY4SG8CkCh097NJ3Fq6X+3dHqlRFJbIkoShhCZNvx1XNXmy2KOI6M wlGi52M+PdDcU8hrUGYURlwbhn1dtMxSQTMIqLHq5tq15ivPaCi4lOLlDwBwZBE9 BfIV331gm8KcfRkW3H4qcbS8gIEsubuKTsIyJibLkxfNT0XUPNT8SrRaT9rKsdyN rsxTTg177E/PjE9awLSnOWVzhUjFy6e4Ml+YIZnSKWAudi6xVjyPO5MjUIdP+IYP +eJ+4ddRGjPIC0nfyQqx57InXyoQvDdOv8VdfSgC+mxVcWJB98k= =mbBZ -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh--