Date: Mon, 11 Nov 2019 17:10:15 +1100 From: Peter Jeremy <peter@rulingia.com> To: Jonathan Anderson <jonathan.anderson@mun.ca> Cc: "freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org> Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111061015.GA50716@server.rulingia.com> In-Reply-To: <CAP8WKbJWSHzhFCKijRVxydKEwgD_4NX2gmA-QVEVZPuotFCGvQ@mail.gmail.com> References: <CAP8WKbJWSHzhFCKijRVxydKEwgD_4NX2gmA-QVEVZPuotFCGvQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2019-Nov-10 22:16:26 -0330, Jonathan Anderson <jonathan.anderson@mun.ca>= wrote: >I=E2=80=99ve gone and done it: I upgraded a key ZFS-on-root machine from 1= 1.2 to >12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on >this list might be able to help me figure out what's wrong and what I can >do about it? Based on your symptoms, it sounds like you might have a corrupt zpool.cache. /boot/zfs/zpool.cache should be rewritten on every boot but I had one system where that wasn't occurring and a FreeBSD upgrade (I don't currently recall the actual versions) resulted in more thorough validation checks, which failed. Can you share your actual layout ("gpart show", "zpool status", details of the non-partitioned disks, etc) - that might help us identify a problem. >It looks like the ZFS code in the bootloader can't find anything in my root >directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If >I boot a rescue image from USB I can mount everything in the pool (`zpool >import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs >zroot/ROOT/default` from the loader prompt it gives me an empty result (so, >e.g., no /boot). Booting fails with messages such as, "i/o error - all blo= ck >copies unavailable". If you boot from a rescue image and either delete (or rename) your existing zpool.cache or run # zpool set cachefile=3D/mnt/boot/zfs/zpool.cache zroot (where the path cachefile path maps to /boot/zfs/zpool.cache at boot), does that help? >My pool consists of three mirrored vdevs, in which the first mirror uses G= PT > partitioning (for the boot partitions) and the other two mirrors use >whole disks. Whole disks are not recommended for anything other than building partitions in. Ideally, you want all the disks to have GPT (or similar) partitions. If you don't need anything else on the disk, just create a single partition occupying the entire disk[1]. (I'd also recommend having a small root zpool that is a single, preferably mirrored, vdev, rather than a large root spread over multiple vdevs). >I recall reading somewhere that the bootloader ZFS code doesn't like >non-partition-based >vdevs... is that true? If so, perhaps the issue is that my upgrade caused >/boot to live on one of the newer whole-disk-based mirrors, hiding it from >the bootloader's view? That might be possible though it's not clear why it wouldn't have caused a problem in the past. Note that the bootloader is performing I/O via the BIOS so if the BIOS can't see the disks, the bootloader won't be able to read them. >> partitions with bootcode and file systems needed for booting can be >added. This allows booting from disks that are also members of a pool. >There is no performance penalty on FreeBSD when using a partition rather >than a whole disk Actually, Solaris might insist that you add a whole disk but, under the hood, ZFS actually partitions the disk with a single partition occupying the entire disk. >The Handbook suggests that it's possible to break disks into multiple >partitions that are added separately to a vdev, but is this a sensible >thing to do? Is there any other ZFS lore that hasn't made it to the >Handbook but that ought to be kept in mind from the outset? You can have multiple partitions on a disk and put different partitions into different vdevs but there's no point in having different partitions on the same disk in the same vdev - that will reduce both performance and resilience. [1] Though leaving a free GB or so can help if you need to replace a disk a= nd the replacement is slightly smaller. --=20 Peter Jeremy --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAl3I+0BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQ6yhAAj/hYwvytdS2IdtZfQtBcX29j2h5FRfIBMwBBkZJnB6hekCiOgGhPAuL0 D+FNAi1GY50NSe2qcgxvVOGqKlGcz5f8xoRUWlYg4u1y+cb1XwEXU87ujW4BoDWd kxnZaYozeB9IANa5FZwl9BdFcYTNIUzWA9Z2hc3ofCCEgG9Ckl+cnjlQNRXdbQAv cZEMof/mqNYfRtEXPQboAOcB5MVdZu3ytn/s5tlI2Nk9EiWyJ+zue6UZfBuTfraI 13Zt6AVWgXedyD1zDToAl5bfexlVuE8gjVZH3FOcGNfQgYvMUnnkzrK2/25vH3cJ 0rvSN53DU5zTIdyoI3xP80ck416gF/sVK/v5CyPmHdNkS1cqIHsCLfEOCfh+9PtP fNULo3LQf/cuY4SG8CkCh097NJ3Fq6X+3dHqlRFJbIkoShhCZNvx1XNXmy2KOI6M wlGi52M+PdDcU8hrUGYURlwbhn1dtMxSQTMIqLHq5tq15ivPaCi4lOLlDwBwZBE9 BfIV331gm8KcfRkW3H4qcbS8gIEsubuKTsIyJibLkxfNT0XUPNT8SrRaT9rKsdyN rsxTTg177E/PjE9awLSnOWVzhUjFy6e4Ml+YIZnSKWAudi6xVjyPO5MjUIdP+IYP +eJ+4ddRGjPIC0nfyQqx57InXyoQvDdOv8VdfSgC+mxVcWJB98k= =mbBZ -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20191111061015.GA50716>