From owner-freebsd-fs@freebsd.org Mon Nov 11 01:46:43 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7477E1BF9A1 for ; Mon, 11 Nov 2019 01:46:43 +0000 (UTC) (envelope-from p15jra@mun.ca) Received: from esa06.ucs.mun.ca (esa06.ucs.mun.ca [134.153.136.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BDJ24DfDz4cLY for ; Mon, 11 Nov 2019 01:46:42 +0000 (UTC) (envelope-from p15jra@mun.ca) IronPort-SDR: CV17VZTCHizXiLJ6joelu8VA0eQtUujITNPNTdz6ES2BPfJwtsforsaV+MIKIaWbO+/xbm3J78 EGTEl30YTD+/zz8QDZVNyBSoXaxvpQ45pJFJ9Ld0okGexeTw60hux2t+gPcxFSEHKoALVFh/a0 P+cuJ5K3XOtHu4J0pRhJUBWTs94xsVMqnYBbf6BwU6aav36iqtEGkzy7n896J0IVyIveT9d3J/ DhimhRUgPK+3KWZccbW7zWIeels1QKZ0rtdRBhIwTvT74Ek2PfAmTs6e/ZukICqi0BRFdbVh/h eUM= X-IronPort-AV: E=Sophos;i="5.68,290,1569292200"; d="scan'208,217";a="20780728" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-yw1-f72.google.com ([209.85.161.72]) by smtp06.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2019 22:16:41 -0430 Received: by mail-yw1-f72.google.com with SMTP id s128so7126709ywf.0 for ; Sun, 10 Nov 2019 17:46:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=W0wHFBN75ucX9h8BpITjwFTnYX4qif0RauNjBtDYyZU=; b=YUvS6VaFKqWXBf+PxA9YUGr3pAmh+F4KLfdTxabSz1DS5NC4I9dhVCFz+TUZ6CDRw2 ZnESQBtAizslDZs+OpVK/diy+Y/hYep0ZReH8EVkRQmFmhnp01N2cyzipHCTx5i3QbHy rAfvDsMG++CLcJ6/5TvTy3/QNmHe1/IFMQSXTBm40UT/IzZxI7QtPDM8sLg3gEEl46LD veaeaK2XicG6WcQxaGrL7sl/IeKSv9soQfns+xH/mFygIWKVOH0syw1CHqjQozC7njd7 9iD3fGJvCOgmfJZHTwG6YLRnJGTfnUJtSVs/al82ASidoc74IYgEFthEiWQMAwwvcyHl RN4Q== X-Gm-Message-State: APjAAAUrRh3eUXmjlxHtT3zqLJCisROoi/E2vOD/6RRJxexCY97joyyZ OLf/DnJeHqTi03o5P147bvvEjC8n4avEQEcvZz5PEd9NVOmcSn2nbIZonx6p+r0Yt2ssEmuNRUS kOk9UGn/IzTV6XfqomLOr3zLHA/lijxK3 X-Received: by 2002:a25:7a47:: with SMTP id v68mr18475350ybc.438.1573436798546; Sun, 10 Nov 2019 17:46:38 -0800 (PST) X-Google-Smtp-Source: APXvYqyejoWfXwMwYMWmOD9gxcpX89gsFNWKrtzWJhVwUDoyaA00QZm17V1zUw6UOXBi/BVBn0juiX1pysr9l7QlvpQ= X-Received: by 2002:a25:7a47:: with SMTP id v68mr18475340ybc.438.1573436797972; Sun, 10 Nov 2019 17:46:37 -0800 (PST) MIME-Version: 1.0 From: Jonathan Anderson Date: Sun, 10 Nov 2019 22:16:26 -0330 Message-ID: Subject: Broken ZFS boot on upgrade To: "freebsd-fs@FreeBSD.org" X-Rspamd-Queue-Id: 47BDJ24DfDz4cLY X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of p15jra@mun.ca has no SPF policy when checking 134.153.136.26) smtp.mailfrom=p15jra@mun.ca X-Spamd-Result: default: False [3.57 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; URI_COUNT_ODD(1.00)[3]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.79)[0.794,0]; NEURAL_SPAM_LONG(0.75)[0.748,0]; RCVD_IN_DNSWL_NONE(0.00)[26.136.153.134.list.dnswl.org : 127.0.10.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[jonathan.anderson@mun.ca,p15jra@mun.ca]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; FROM_NEQ_ENVFROM(0.00)[jonathan.anderson@mun.ca,p15jra@mun.ca]; IP_SCORE(-0.17)[ipnet: 134.153.0.0/16(-0.43), asn: 6579(-0.35), country: CA(-0.09)] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 01:46:43 -0000 Hello friendly FreeBSD people, I=E2=80=99ve gone and done it: I upgraded a key ZFS-on-root machine from 11= .2 to 12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on this list might be able to help me figure out what's wrong and what I can do about it? It looks like the ZFS code in the bootloader can't find anything in my root directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If I boot a rescue image from USB I can mount everything in the pool (`zpool import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs zroot/ROOT/default` from the loader prompt it gives me an empty result (so, e.g., no /boot). Booting fails with messages such as, "i/o error - all bloc= k copies unavailable". My pool consists of three mirrored vdevs, in which the first mirror uses GP= T partitioning (for the boot partitions) and the other two mirrors use whole disks. I recall reading somewhere that the bootloader ZFS code doesn't like non-partition-based vdevs... is that true? If so, perhaps the issue is that my upgrade caused /boot to live on one of the newer whole-disk-based mirrors, hiding it from the bootloader's view? I've backed up all of my data (a 28h process) and am resigned to re-installing if need be, but I would like to understand where I went wrong to avoid making the same mistakes again. What would you recommend for someone building a new array-of-mirrors pool? I see that the Handbook says that certain things *can* be done, but it=E2=80=99s not clear to me whether= they should or even must be done, e.g.: > partitions with bootcode and file systems needed for booting can be added. This allows booting from disks that are also members of a pool. There is no performance penalty on FreeBSD when using a partition rather than a whole disk The Handbook suggests that it's possible to break disks into multiple partitions that are added separately to a vdev, but is this a sensible thing to do? Is there any other ZFS lore that hasn't made it to the Handbook but that ought to be kept in mind from the outset? Thanks, Jon-- Assistant Professor Electrical and Computer Engineering Memorial University of Newfoundland https://www.engr.mun.ca/~anderson From owner-freebsd-fs@freebsd.org Mon Nov 11 06:10:38 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1B2BB1AC2E2 for ; Mon, 11 Nov 2019 06:10:38 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BL8X4Shwz3K1X for ; Mon, 11 Nov 2019 06:10:36 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id xAB6AKnf020621 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 11 Nov 2019 17:10:26 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id xAB6AFwS069202 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Nov 2019 17:10:15 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id xAB6AFvv069201; Mon, 11 Nov 2019 17:10:15 +1100 (AEDT) (envelope-from peter) Date: Mon, 11 Nov 2019 17:10:15 +1100 From: Peter Jeremy To: Jonathan Anderson Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111061015.GA50716@server.rulingia.com> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47BL8X4Shwz3K1X X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-7.71 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[rulingia.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-3.31)[ip: (-9.91), ipnet: 2001:19f0:5800::/38(-4.95), asn: 20473(-1.65), country: US(-0.05)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 06:10:38 -0000 --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2019-Nov-10 22:16:26 -0330, Jonathan Anderson = wrote: >I=E2=80=99ve gone and done it: I upgraded a key ZFS-on-root machine from 1= 1.2 to >12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on >this list might be able to help me figure out what's wrong and what I can >do about it? Based on your symptoms, it sounds like you might have a corrupt zpool.cache. /boot/zfs/zpool.cache should be rewritten on every boot but I had one system where that wasn't occurring and a FreeBSD upgrade (I don't currently recall the actual versions) resulted in more thorough validation checks, which failed. Can you share your actual layout ("gpart show", "zpool status", details of the non-partitioned disks, etc) - that might help us identify a problem. >It looks like the ZFS code in the bootloader can't find anything in my root >directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If >I boot a rescue image from USB I can mount everything in the pool (`zpool >import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs >zroot/ROOT/default` from the loader prompt it gives me an empty result (so, >e.g., no /boot). Booting fails with messages such as, "i/o error - all blo= ck >copies unavailable". If you boot from a rescue image and either delete (or rename) your existing zpool.cache or run # zpool set cachefile=3D/mnt/boot/zfs/zpool.cache zroot (where the path cachefile path maps to /boot/zfs/zpool.cache at boot), does that help? >My pool consists of three mirrored vdevs, in which the first mirror uses G= PT > partitioning (for the boot partitions) and the other two mirrors use >whole disks. Whole disks are not recommended for anything other than building partitions in. Ideally, you want all the disks to have GPT (or similar) partitions. If you don't need anything else on the disk, just create a single partition occupying the entire disk[1]. (I'd also recommend having a small root zpool that is a single, preferably mirrored, vdev, rather than a large root spread over multiple vdevs). >I recall reading somewhere that the bootloader ZFS code doesn't like >non-partition-based >vdevs... is that true? If so, perhaps the issue is that my upgrade caused >/boot to live on one of the newer whole-disk-based mirrors, hiding it from >the bootloader's view? That might be possible though it's not clear why it wouldn't have caused a problem in the past. Note that the bootloader is performing I/O via the BIOS so if the BIOS can't see the disks, the bootloader won't be able to read them. >> partitions with bootcode and file systems needed for booting can be >added. This allows booting from disks that are also members of a pool. >There is no performance penalty on FreeBSD when using a partition rather >than a whole disk Actually, Solaris might insist that you add a whole disk but, under the hood, ZFS actually partitions the disk with a single partition occupying the entire disk. >The Handbook suggests that it's possible to break disks into multiple >partitions that are added separately to a vdev, but is this a sensible >thing to do? Is there any other ZFS lore that hasn't made it to the >Handbook but that ought to be kept in mind from the outset? You can have multiple partitions on a disk and put different partitions into different vdevs but there's no point in having different partitions on the same disk in the same vdev - that will reduce both performance and resilience. [1] Though leaving a free GB or so can help if you need to replace a disk a= nd the replacement is slightly smaller. --=20 Peter Jeremy --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAl3I+0BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQ6yhAAj/hYwvytdS2IdtZfQtBcX29j2h5FRfIBMwBBkZJnB6hekCiOgGhPAuL0 D+FNAi1GY50NSe2qcgxvVOGqKlGcz5f8xoRUWlYg4u1y+cb1XwEXU87ujW4BoDWd kxnZaYozeB9IANa5FZwl9BdFcYTNIUzWA9Z2hc3ofCCEgG9Ckl+cnjlQNRXdbQAv cZEMof/mqNYfRtEXPQboAOcB5MVdZu3ytn/s5tlI2Nk9EiWyJ+zue6UZfBuTfraI 13Zt6AVWgXedyD1zDToAl5bfexlVuE8gjVZH3FOcGNfQgYvMUnnkzrK2/25vH3cJ 0rvSN53DU5zTIdyoI3xP80ck416gF/sVK/v5CyPmHdNkS1cqIHsCLfEOCfh+9PtP fNULo3LQf/cuY4SG8CkCh097NJ3Fq6X+3dHqlRFJbIkoShhCZNvx1XNXmy2KOI6M wlGi52M+PdDcU8hrUGYURlwbhn1dtMxSQTMIqLHq5tq15ivPaCi4lOLlDwBwZBE9 BfIV331gm8KcfRkW3H4qcbS8gIEsubuKTsIyJibLkxfNT0XUPNT8SrRaT9rKsdyN rsxTTg177E/PjE9awLSnOWVzhUjFy6e4Ml+YIZnSKWAudi6xVjyPO5MjUIdP+IYP +eJ+4ddRGjPIC0nfyQqx57InXyoQvDdOv8VdfSgC+mxVcWJB98k= =mbBZ -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh-- From owner-freebsd-fs@freebsd.org Mon Nov 11 08:18:05 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D585B1AECA2 for ; Mon, 11 Nov 2019 08:18:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BNzd0DZlz3Pdh for ; Mon, 11 Nov 2019 08:18:04 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f52.google.com with SMTP id y6so9137249lfj.2 for ; Mon, 11 Nov 2019 00:18:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=RddTq3bGQu8YjGo7XxiHlnJWQWM3Ax1bJmnecSWEwDY=; b=KkUY3dQdLSmFTdRyATbLnyFTJMBcrseVidJOyhywUjokjUiMMBxvzWS67RW+yFUKko qXHzKUB6Y4k4OL6D5uWDYgqnVHgWQA7ggzBcsRcsCS939OJYS5Pg1coQdq2rTu8PXsAZ F18j9JZyMmwnkG00laaS6XyRuAWExH0lqkJ2buWiD+/6xfEb8kAKQtkOkjCB/47tGPJA ocb2KKV4P/tE4LrZvu2xAerp0vbmJEBsDav+xy+MESN8i0YE3VFC1FjiUXHDJ2ze3Vy0 kR0qzCHcdwJ0b//IwodV9WhR+Nr1WkFNZ40nzpUwKKkL9NHy3IHpwgqvXz4zRWXRi3BY lDbg== X-Gm-Message-State: APjAAAX8AdfyqN2Ob0cJND79RFiJCOyHdwcEh1SRPu9Eik9qqNh4kH/B J4yvP/lq6h3KCEvcdmWjwpp+PNIHxsE= X-Google-Smtp-Source: APXvYqy177PRqp0f3dBIUCzHefb67yuOA4rONc+L/YahNwG0dbYkrdfXCRlNjgA6MfmCjB/RlnsB3g== X-Received: by 2002:ac2:5deb:: with SMTP id z11mr1749660lfq.35.1573460282882; Mon, 11 Nov 2019 00:18:02 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id a144sm3447867lfd.27.2019.11.11.00.18.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Nov 2019 00:18:02 -0800 (PST) Subject: Re: Broken ZFS boot on upgrade To: Jonathan Anderson , "freebsd-fs@FreeBSD.org" References: From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> Date: Mon, 11 Nov 2019 10:18:01 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47BNzd0DZlz3Pdh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.15 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[52.167.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.15)[ip: (-0.49), ipnet: 209.85.128.0/17(-3.19), asn: 15169(-2.00), country: US(-0.05)]; RWL_MAILSPIKE_POSSIBLE(0.00)[52.167.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 08:18:05 -0000 On 11/11/2019 03:46, Jonathan Anderson wrote: > but I would like to understand where I went wrong > to avoid making the same mistakes again Could it be that you have 2TB+ disk(s) and a relatively old BIOS ? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Nov 11 11:53:21 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A48D1B2E9F for ; Mon, 11 Nov 2019 11:53:21 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 47BTm00Mcvz44w9 for ; Mon, 11 Nov 2019 11:53:19 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id xABBrFsR067332; Mon, 11 Nov 2019 11:53:15 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id xABBrEEi021535; Mon, 11 Nov 2019 11:53:14 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id xABBrEQB021531; Mon, 11 Nov 2019 11:53:14 GMT Date: Mon, 11 Nov 2019 11:53:14 GMT Message-Id: <201911111153.xABBrEQB021531@higson.cam.lispworks.com> From: Martin Simmons To: Jonathan Anderson CC: freebsd-fs@freebsd.org In-reply-to: (message from Jonathan Anderson on Sun, 10 Nov 2019 22:16:26 -0330) Subject: Re: Broken ZFS boot on upgrade References: X-Rspamd-Queue-Id: 47BTm00Mcvz44w9 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.95 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.958,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.977,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; IP_SCORE(-0.02)[country: GB(-0.08)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 11:53:21 -0000 >>>>> On Sun, 10 Nov 2019 22:16:26 -0330, Jonathan Anderson said: > > but when I run `lszfs > zroot/ROOT/default` from the loader prompt it gives me an empty result (so, > e.g., no /boot). FWIW, I think that is normal unless you have created a child dataset of zroot/ROOT/default. You can try the ls command, e.g. ls ls /boot ls /boot/kernel ls /boot/kernel/kernel __Martin From owner-freebsd-fs@freebsd.org Mon Nov 11 12:51:41 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ED7CA1B4DDF for ; Mon, 11 Nov 2019 12:51:41 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) Received: from esa02.ucs.mun.ca (esa02.ucs.mun.ca [134.153.136.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BW3J630Cz48Gw for ; Mon, 11 Nov 2019 12:51:40 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) IronPort-SDR: qm9oHfbgUhjEpLT08UU2T8iP/iFaPAxgrIq5s5BbznlCrwROivQkLV2fYGgd0U6xLBAW9prT9f 6Cwj9tBkZWZygavy7ZnJUhXhVSDt1PR68eLoeI9YHalzV+ORXaa3ft3pxRDU0C/lQNQZkp0VF3 Rs05Kc3HshPvCBhA7/rvuPYXo/adeljJCtECX3zTwc8o1nu8rVQLZuI96twNMh83pISVWGYl94 CTmdb7JVsERUnMPSt2C17an9VwZoik/v9JQyo7WVZE2XTcLOjY4Hu2QiGEtRcIeWPRrqnsoXmZ 7hc= X-IronPort-AV: E=Sophos;i="5.68,293,1569292200"; d="scan'208";a="21054937" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-qt1-f199.google.com ([209.85.160.199]) by smtp02.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 09:21:40 -0430 Received: by mail-qt1-f199.google.com with SMTP id v92so17445317qtd.18 for ; Mon, 11 Nov 2019 04:51:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=JIfgPkKWoThaP8m1oVu8LIP0/N4HHJTyiR/bKuSi5I4=; b=YaCmh8V7y1/E9Wsdw8LUifog0BDuhr9TFIXu+fC4wv/dsakKWvNFk86TeTZ/oP89Ly cRwDvxsB+KVKdyvqtuDlZ2vRGJTkUm3lAYlOOH/v9iF5ywgZODXu9hFazaVlrDloc20M 6oVl3NebqU7SUMxHrjMLuc59ydF8YEFCUnR408DIybNuOL3O/mwxsRFsilOqVOBiTGVS 9j12On5uMw24ZsXosd6hmCLQyVeMvZL6NyozkSTzrs3mR/pSvmdzg4ymIAyMRT4/ZHOi 4AoL+sp33LJjmBYVHsq23nNLA4Kaa3/HzxWTL8twShd8VFvlUWOPUT4ZjQaTcrrjnvwM wGkg== X-Gm-Message-State: APjAAAUEhT5kvOjYsDttUZ6ODMxQjx/HUGq1C0iSeqnuOnQ58+thQONX fobSljczGVg/23J5kYA/6GMM44Sl8nX0jyptALC42P2bVqr98JLpM+WMTRvMTTMnxBRRhlSWY+N KHE+8PTFGVEXHaJJG X-Received: by 2002:a0c:ee41:: with SMTP id m1mr9230986qvs.201.1573476698459; Mon, 11 Nov 2019 04:51:38 -0800 (PST) X-Google-Smtp-Source: APXvYqzem9TawvkFtV6Uzm2qpj4uCbD6yv7Shse6vqnDyY3SqkjrMgZlcMhAb4Qa3V5hmWF4Q3sbyA== X-Received: by 2002:a0c:ee41:: with SMTP id m1mr9230957qvs.201.1573476697944; Mon, 11 Nov 2019 04:51:37 -0800 (PST) Received: from bagstock.jonandchrissy.ca (CPE00fc8db75533-CM00fc8db75530.cpe.net.cable.rogers.com. [174.117.179.42]) by smtp.gmail.com with ESMTPSA id u18sm11327822qth.20.2019.11.11.04.51.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 04:51:37 -0800 (PST) Date: Mon, 11 Nov 2019 09:21:35 -0330 From: Jonathan Anderson To: "Kevin P. Neal" Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111125135.GA70914@bagstock.jonandchrissy.ca> References: <20191111021326.GA61162@neutralgood.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191111021326.GA61162@neutralgood.org> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47BW3J630Cz48Gw X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jonathan.anderson@mun.ca has no SPF policy when checking 134.153.136.22) smtp.mailfrom=jonathan.anderson@mun.ca X-Spamd-Result: default: False [-1.26 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.992,0]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[22.136.153.134.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.17)[ipnet: 134.153.0.0/16(-0.43), asn: 6579(-0.34), country: CA(-0.09)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 12:51:42 -0000 On 11/10, Kevin P. Neal wrote: > Do you have snapshots? If you have a snapshot from before you upgraded > then you can roll back the upgrade. That's another way to get you back > to a bootable system. I do have snapshots, though mightn't the effectiveness of a rollback depend on the reason for the failure? For example, if the issue is the 2+ TB thing, mightn't new blocks created by the rollback also end up on the large, not-visible-to-BIOS disks? > If this is the case then can you remove disks one at a time and replace > them with GPT-partitioned disks? If you are using mirrors then this may > work if the smallest partition/disk currently in use per-vdev is no larger > than a disk that has been partitioned. Hmmmm... I think that, unfortunately, the mirrors are mostly exactly the same size (four 1 TiB and two 3 TiB), so there isn't any slack for resizing. > Sensible? Mmmmmm... I would say that multiple ZFS partitions on a single > disk is something to be avoided if possible. There may be cases where it > can't be avoided, but it's not the best. Multiple partitions on a disk in > the _same_ _pool_ sounds just plain bad and I wouldn't do it. Ok, thanks! Jon -- Jonathan Anderson jonathan@FreeBSD.org From owner-freebsd-fs@freebsd.org Mon Nov 11 13:05:29 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 721CE1B5206 for ; Mon, 11 Nov 2019 13:05:29 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) Received: from esa01.ucs.mun.ca (esa01.ucs.mun.ca [134.153.136.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BWMD3mjgz48tg for ; Mon, 11 Nov 2019 13:05:28 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) IronPort-SDR: ZEg9P/UKvzs4Db3OrIvcsyMo0YEbsGnKbN6mrdP/lSu6W0tUyViaeVZ+zNr4W2+H9DU0t7S4Yz qMtqpRp5XFxCQgABCAf0PiLgZTxg3XK9jO9KGTO4Mj5bJazBMfoCaP8n2CPAtZ0QXh+PjKB98a qPn/0retCQIR1Qsj+IbPsA74vUxMkPgqrp2YCCf20XT3ht7Ws39Of6lL4mMo6Qq5soIUER9Gl/ dPuclOZFr/+a8Ezi+Z6eYNjP3OVBndzzq+nvNVOTAs+fWdVfbKQ6mq/nZnKjgG73PQQAd67M71 SkU= X-IronPort-AV: E=Sophos;i="5.68,293,1569292200"; d="scan'208";a="23459206" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-qk1-f197.google.com ([209.85.222.197]) by smtp01.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 09:35:27 -0430 Received: by mail-qk1-f197.google.com with SMTP id p68so7705008qkf.9 for ; Mon, 11 Nov 2019 05:05:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=2nDNuIdeBySAZbSB62wRb3r6rk0oEfAyZuN7fhT/SZI=; b=ZdXQ3AhGcdv6VOLSCqW/bfS8Z3sa1nTYSRKN+Wl0B5CCuY0H6ehDxTIQxtsGTyzGan FqgAzmqmoSD7rSx3yDQAlSSxs+jPXGuRQV0fzMb+YomR3SaLQgmnpf4zFotAmLBZcrVO qODiLrRDg4SgXNKpL14FMtWZiaAkxzrlMCQxIYsfFWGLxrVfOKifo87iO4/dwI9tyje0 0YNhp9oKdyNBl8DaE0ttL9Kv4S08klfujHsvAZvTucg1QsCN+k4GyAAWLIfmEOVfSSL3 kUQ82valMIzX7F+LsI/8IEsyACkh1hFzA8MDk5m3ZJnHJ8e8IKjexOF5zDZmea+riLV9 6Dxg== X-Gm-Message-State: APjAAAUNCihxJShr4DmCJn8KUfdffmmMMorteeuhL/oRbj0JUfq+JO8o Oci+wPtAlvJIHFZKCum4lUp8fVYkGNji25WoSbNHgyzp+fWek7dTyaJThTyzpa++cU7+hwXVQxJ QDacloJ6aaLB6l1q1 X-Received: by 2002:a37:ba44:: with SMTP id k65mr10508262qkf.169.1573477525682; Mon, 11 Nov 2019 05:05:25 -0800 (PST) X-Google-Smtp-Source: APXvYqyp/XWJwPXDHjiuJ5chblQ8hpcgAc2bjayouqyQogfwKa2hMk0bjA61QnZ336XoZHglEvfk6A== X-Received: by 2002:a37:ba44:: with SMTP id k65mr10508209qkf.169.1573477524877; Mon, 11 Nov 2019 05:05:24 -0800 (PST) Received: from bagstock.jonandchrissy.ca (CPE00fc8db75533-CM00fc8db75530.cpe.net.cable.rogers.com. [174.117.179.42]) by smtp.gmail.com with ESMTPSA id m72sm7067617qke.5.2019.11.11.05.05.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 05:05:24 -0800 (PST) Date: Mon, 11 Nov 2019 09:35:21 -0330 From: Jonathan Anderson To: Peter Jeremy Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111130521.GB70914@bagstock.jonandchrissy.ca> References: <20191111061015.GA50716@server.rulingia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191111061015.GA50716@server.rulingia.com> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47BWMD3mjgz48tg X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jonathan.anderson@mun.ca has no SPF policy when checking 134.153.136.21) smtp.mailfrom=jonathan.anderson@mun.ca X-Spamd-Result: default: False [-1.26 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[21.136.153.134.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.17)[ipnet: 134.153.0.0/16(-0.42), asn: 6579(-0.33), country: CA(-0.09)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 13:05:29 -0000 Good morning, On 11/11, Peter Jeremy wrote: > Based on your symptoms, it sounds like you might have a corrupt zpool.cache. > /boot/zfs/zpool.cache should be rewritten on every boot but I had one system > where that wasn't occurring and a FreeBSD upgrade (I don't currently recall > the actual versions) resulted in more thorough validation checks, which > failed. The /boot/zfs/zpool.cache file is present, but it looks like the bootloader isn't able to see it (or anything else in /boot). > Can you share your actual layout ("gpart show", "zpool status", details of > the non-partitioned disks, etc) - that might help us identify a problem. Certainly. The output from `geom disk list` is: -- Geom name: ada0 Providers: 1. Name: ada0 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e3 descr: ST91000640NS lunid: 5000c5007a4e82cc ident: 9XG82F3D rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada1 Providers: 1. Name: ada1 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e3 descr: ST91000640NS lunid: 5000c5007a4edb75 ident: 9XG82GVR rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada2 Providers: 1. Name: ada2 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e2 descr: ST91000640NS lunid: 5000c50090aa52d1 ident: 9XG9SQVM rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada3 Providers: 1. Name: ada3 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e2 descr: ST91000640NS lunid: 5000c50090aa98d3 ident: 9XG9SQ7H rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada4 Providers: 1. Name: ada4 Mediasize: 3000592982016 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 descr: ST3000LM016-1N217V lunid: 5000c5009049c1c3 ident: W800RZ6A rotationrate: 5400 fwsectors: 63 fwheads: 16 Geom name: ada5 Providers: 1. Name: ada5 Mediasize: 3000592982016 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 descr: ST3000LM016-1N217V lunid: 5000c5008a91fc59 ident: W800SEP0 rotationrate: 5400 fwsectors: 63 fwheads: 16 -- The output from `geom part list` is: -- Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 1953525134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 524288 (512K) Sectorsize: 512 Stripesize: 0 Stripeoffset: 17408 Mode: r0w0e0 efimedia: HD(1,GPT,33b1f1e3-dd2d-11e4-9108-ecf4bbd78d94,0x22,0x400) rawuuid: 33b1f1e3-dd2d-11e4-9108-ecf4bbd78d94 rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f label: gptboot0 length: 524288 offset: 17408 type: freebsd-boot index: 1 end: 1057 start: 34 2. Name: ada0p2 Mediasize: 1000204327424 (932G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 541696 Mode: r1w1e2 efimedia: HD(2,GPT,33da20c1-dd2d-11e4-9108-ecf4bbd78d94,0x422,0x7470696d) rawuuid: 33da20c1-dd2d-11e4-9108-ecf4bbd78d94 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: zfs0 length: 1000204327424 offset: 541696 type: freebsd-zfs index: 2 end: 1953525134 start: 1058 Consumers: 1. Name: ada0 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e3 Geom name: diskid/DISK-9XG82GVR modified: false state: OK fwheads: 16 fwsectors: 63 last: 1953525134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: diskid/DISK-9XG82GVRp1 Mediasize: 524288 (512K) Sectorsize: 512 Stripesize: 0 Stripeoffset: 17408 Mode: r0w0e0 efimedia: HD(1,GPT,343639ad-dd2d-11e4-9108-ecf4bbd78d94,0x22,0x400) rawuuid: 343639ad-dd2d-11e4-9108-ecf4bbd78d94 rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f label: gptboot1 length: 524288 offset: 17408 type: freebsd-boot index: 1 end: 1057 start: 34 2. Name: diskid/DISK-9XG82GVRp2 Mediasize: 1000204327424 (932G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 541696 Mode: r1w1e1 efimedia: HD(2,GPT,345be840-dd2d-11e4-9108-ecf4bbd78d94,0x422,0x7470696d) rawuuid: 345be840-dd2d-11e4-9108-ecf4bbd78d94 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: zfs1 length: 1000204327424 offset: 541696 type: freebsd-zfs index: 2 end: 1953525134 start: 1058 Consumers: 1. Name: diskid/DISK-9XG82GVR Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e2 -- (omitting `da0`, which is my recovery USB stick) > If you boot from a rescue image and either delete (or rename) your > existing zpool.cache or run > # zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot > (where the path cachefile path maps to /boot/zfs/zpool.cache at boot), does > that help? Ok, I'll add that to the list of things to try when I regain physical access to the machine tomorrow. :) > Whole disks are not recommended for anything other than building partitions > in. Ideally, you want all the disks to have GPT (or similar) partitions. > If you don't need anything else on the disk, just create a single partition > occupying the entire disk[1]. OK, sounds good. Is there a specific reason for this recommendation? I understand that this advice is possible because FreeBSD handles cacheing differently from Solaris (which recommended using full disks), but is the GPT recommendation something about making ZFS partitions more visible to the bootloader? > (I'd also recommend having a small root zpool > that is a single, preferably mirrored, vdev, rather than a large root spread > over multiple vdevs). Indeed, if I'd kept my root pool separate from a larger data pool I wouldn't be experiencing this issue now. When I added the additional vdevs to the root pool I had been thinking that it would be convenient not to have to size the pools up front, but it seems that a little bit of such homework up front may have saved me a lot of grief this weekend! > That might be possible though it's not clear why it wouldn't have caused a > problem in the past. The last time I did a wholesale upgrade of /boot, there were fewer vdevs in the pool. So, if the issue is either a) the blocks living on a whole-disk vdev or b) the blocks living on a 3 TiB vdev, it could be this is just the first time that the contents of /boot have happened to land in a "bad" place. > Note that the bootloader is performing I/O via the > BIOS so if the BIOS can't see the disks, the bootloader won't be able to > read them. The BIOS can see the disks, but I see that a later email asked about the size of those disks... that could be the (an?) issue. > You can have multiple partitions on a disk and put different partitions > into different vdevs but there's no point in having different partitions > on the same disk in the same vdev - that will reduce both performance > and resilience. Great: I had kind of suspected that. Thank you, Jon -- Jonathan Anderson jonathan@FreeBSD.org From owner-freebsd-fs@freebsd.org Mon Nov 11 13:10:27 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F0D021B52D8 for ; Mon, 11 Nov 2019 13:10:27 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) Received: from esa05.ucs.mun.ca (esa05.ucs.mun.ca [134.153.136.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BWSy5xNkz48yj for ; Mon, 11 Nov 2019 13:10:26 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) IronPort-SDR: 6f1wh/pUol9DWKL749j/O9j6UydF8emT8a971OCKp+ca4YGcff/c4R3BC3GJBNizQDbwkWUEw9 sNqwKTpDc1z0gtriWXoRlFDC0fRqHEPsdCcs6ZSPdEbo8LvFtyrxlbSrsKlTRPWxt1rHxFRvOB 0V1vpxLjHu2xhNYjxL2QkjTkeUIfS2ZawYJafLZXOkz3n6cDZGLp7G0HXXrRqkp/p6MLFMBsw3 f6gzIUUMqXJEb7HuDNfd9egZzsg+Q+eyKDm4hsVrbPlxB5D2lZuXVtbTokTIGp52T6EphNrfOi pP8= X-IronPort-AV: E=Sophos;i="5.68,293,1569292200"; d="scan'208";a="20838968" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-qt1-f200.google.com ([209.85.160.200]) by smtp05.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 09:40:25 -0430 Received: by mail-qt1-f200.google.com with SMTP id h15so17530381qtn.6 for ; Mon, 11 Nov 2019 05:10:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=0Wp2NKhPhrtNRj3l0d4tzQ/VSgqNvETAjIa8c/B9JNA=; b=KBc1oniNAusupiT2SX4oPHhoTojXn5AED8NcTot5Jh90br7MfLyxkuYXA8Q2T9SHos 3rv4KRthcVbC8lJoG8oXE/N7uEo+cnpMUAn52kXK++oaeSP5lq9BQJO3l4ATCY5/CSoM 7p31K43DzQleFugPaNZcKthjnj/oAKwDQHxiNO6FSe6URVDrg5ssUJcVUSvEcjEmaqX/ AezBx9cTChw5x4dOU/C4koDRtsFLzWcKZWCKz2DJPF/WDSmHFZY7GDXlTEsNlnT+DCVi hc3kb07z+Sa/zO+9o46vw1VUZoDbFzKUdERDnCbwFXNZD5+Ms7OvTSaOY7hmALYjlBUf QGhQ== X-Gm-Message-State: APjAAAVKJ3B6ygdanwkgYlxN3erDCzIN1NT0L3P1wm36NlCdZ7wT2ZRj +TjmliYUuSqbkSwT6Ep/VSoJsLiYHmG6uloIGINC7S17U1Wi5pYKg1rFyLFNPyENWYvNjwqGNrk LT2eMsDiYyCjfRGbx X-Received: by 2002:a05:6214:484:: with SMTP id ay4mr23326859qvb.35.1573477824336; Mon, 11 Nov 2019 05:10:24 -0800 (PST) X-Google-Smtp-Source: APXvYqzya1XtFK35M92eIopEzOlOEVQpCD36v/qRPGaZnDpmYiEIcuduzRUFzptsxldzJgEHucBUAw== X-Received: by 2002:a05:6214:484:: with SMTP id ay4mr23326831qvb.35.1573477823917; Mon, 11 Nov 2019 05:10:23 -0800 (PST) Received: from bagstock.jonandchrissy.ca (CPE00fc8db75533-CM00fc8db75530.cpe.net.cable.rogers.com. [174.117.179.42]) by smtp.gmail.com with ESMTPSA id v20sm6685887qkg.92.2019.11.11.05.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 05:10:23 -0800 (PST) Date: Mon, 11 Nov 2019 09:40:21 -0330 From: Jonathan Anderson To: Andriy Gapon Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111131021.GC70914@bagstock.jonandchrissy.ca> References: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47BWSy5xNkz48yj X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jonathan.anderson@mun.ca has no SPF policy when checking 134.153.136.25) smtp.mailfrom=jonathan.anderson@mun.ca X-Spamd-Result: default: False [-1.26 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[25.136.153.134.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.17)[ipnet: 134.153.0.0/16(-0.41), asn: 6579(-0.33), country: CA(-0.09)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 13:10:28 -0000 On 11/11, Andriy Gapon wrote: > Could it be that you have 2TB+ disk(s) and a relatively old BIOS ? The first two vdevs use 1 TiB disks, but the third vdev has 3 TiB disks, so that sounds like a possible explanation... perhaps the contents of /boot previously resided on one of the 1 TiB vdevs but the new /boot lives on the new vdev post-upgrade. Is there a zfs admin command to ask which vdev(s) a file or directory's blocks reside on? My BIOS is from 2016, so not "old", but possibly old enough? Thank you, Jon -- Jonathan Anderson jonathan@FreeBSD.org From owner-freebsd-fs@freebsd.org Mon Nov 11 13:12:56 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 55E2B1B54EE for ; Mon, 11 Nov 2019 13:12:56 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) Received: from esa07.ucs.mun.ca (esa07.ucs.mun.ca [134.153.136.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BWWq0S2Lz49Hv for ; Mon, 11 Nov 2019 13:12:54 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) IronPort-SDR: zdcWPFJ/NdXFEyAmu6Yhky3QYwaPevn7BW0BConemzjap8w10AFYgQXxNqCKThi4CB6Fe+O8lJ TYIpmLmRd/VvglvupsARLGnhXPkeyMXZYHOa4oOTc53qerIhSp5XlbkCa9HjSigYKzZDqB8837 5+GjxoKQbEPSmVoAQBCdWT0ncfuC5qtzUE1Ium4GIYb++WfbIvFEOIBzpJFEzU1b0HIQw+78S0 T+YHDKpkGCrY3H4GhdbanB6b+B1IHWJlYg7fNBGZLUK9OS7AT/72P3iRaEmsRcOjyF5OnCkrkI qcc= X-IronPort-AV: E=Sophos;i="5.68,293,1569292200"; d="scan'208";a="19587172" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-qt1-f198.google.com ([209.85.160.198]) by smtp07.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 09:42:53 -0430 Received: by mail-qt1-f198.google.com with SMTP id q54so17542915qtk.15 for ; Mon, 11 Nov 2019 05:12:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=ZbZF7e4gRbnlsDf3lO1deBronw5r4VKc6qrvgjo3v08=; b=r8ZERMpje4S0Ct1lrsApwZ26o/ZU2Qq1MbfJtffDDjhsnUgMEY0LQoYDK3M2I+XB15 qd74p0qLoLZ6ycwOfiEjStS5v2Qo+i2LdAZATyNJ+9zuYn+PGz7dyrqj+ibrdlBO9EyO LdRYxE8MwRkFyaQsjLhq7jK6ugYmQ9VTSN2lz21QoASa8MHRbRJWejOPQlFKsOxv4a3N of6Hs7IJyCHg7iXHV3vKSr01UUG7itIUfMlIo5jBPoIDBRGftzN33mhNxYKpP7GOVyaz ccb8lokw8CEIS7xorr1Y9/xfIf2ge8g+7J9abxusJY5jOJ1IRqamcOm8+c2BsUd6oKbL jZkA== X-Gm-Message-State: APjAAAUl/mhZw2x6/18k6gwkQEtH1mMXkzeanJlEZZNAe+urikxqoXUO dEoXXWWWu6OXwRDGpKNftld/GFwgu0QlCdLAcx5+4gG39yV0lVdR7O1RzoiTrIpNzxCyiBJsa6l cG43vLsFHsIjV0+5S X-Received: by 2002:a0c:cd8c:: with SMTP id v12mr2337666qvm.47.1573477972391; Mon, 11 Nov 2019 05:12:52 -0800 (PST) X-Google-Smtp-Source: APXvYqxmWLV0ACu8eMTxi36t0GRuqltdKaSztZBQKyj23yP8uqpPCWgtd8qoqTCPoE1Q9ir0vByGtg== X-Received: by 2002:a0c:cd8c:: with SMTP id v12mr2337639qvm.47.1573477971979; Mon, 11 Nov 2019 05:12:51 -0800 (PST) Received: from bagstock.jonandchrissy.ca (CPE00fc8db75533-CM00fc8db75530.cpe.net.cable.rogers.com. [174.117.179.42]) by smtp.gmail.com with ESMTPSA id o28sm9133757qtk.4.2019.11.11.05.12.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 05:12:51 -0800 (PST) Date: Mon, 11 Nov 2019 09:42:48 -0330 From: Jonathan Anderson To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111131248.GD70914@bagstock.jonandchrissy.ca> References: <201911111153.xABBrEQB021531@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201911111153.xABBrEQB021531@higson.cam.lispworks.com> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47BWWq0S2Lz49Hv X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jonathan.anderson@mun.ca has no SPF policy when checking 134.153.136.27) smtp.mailfrom=jonathan.anderson@mun.ca X-Spamd-Result: default: False [-1.25 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[27.136.153.134.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.16)[ipnet: 134.153.0.0/16(-0.41), asn: 6579(-0.32), country: CA(-0.09)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 13:12:56 -0000 On 11/11, Martin Simmons wrote: > >>>>> On Sun, 10 Nov 2019 22:16:26 -0330, Jonathan Anderson said: > > > > but when I run `lszfs > > zroot/ROOT/default` from the loader prompt it gives me an empty result (so, > > e.g., no /boot). > > FWIW, I think that is normal unless you have created a child dataset of > zroot/ROOT/default. Oh I see, so `lszfs` lists datasets rather than contents? > You can try the ls command, e.g. > > ls > ls /boot > ls /boot/kernel > ls /boot/kernel/kernel These commands resulted in "i/o error - all block copies unavailable" errors. Thanks, Jon -- Jonathan Anderson jonathan@FreeBSD.org From owner-freebsd-fs@freebsd.org Mon Nov 11 13:18:29 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ECD681B55A3 for ; Mon, 11 Nov 2019 13:18:29 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BWfD4zNyz49NN for ; Mon, 11 Nov 2019 13:18:28 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f172.google.com with SMTP id y23so13723215ljh.10 for ; Mon, 11 Nov 2019 05:18:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=yPIXmT8VuEM8egqbcBbLmmJ2GeGDmOQ5MHQuyRfraHs=; b=cJ0gJD2O8QClXIC7HO1nnI4ftN4KbkracJExVDG9K0I7XaxP4cEg59GqMDp8zNnxW5 y9fMPh3JTUOnIUwNq2KD1BhnI0RyebrAzLvuTJA8NzrjkNI5w+xEfEiljlaCUW2GefFM aE0CDt7ikvtQ+UFuTmtCLGW0djNv+/FeLnClyiR+WZrEJfUjqGZ1gzpqJxCcIpFlT01s XoCbppKDfg0BB9HiEP6jfDL2gYOLtqoS1sQGJCsvEVcSYrBhoGLKTvpQbCuFERxWDOTD eQPzhFqhTMxJQ2CcUnxLiUxvO1Pat05g7tfweO3ircJmixKyPF8R9nY3JIvzP+fpjEmM DKCQ== X-Gm-Message-State: APjAAAXwqDTaMR6z2MfQxmuQqMM8YtwwdOCdfFScAVHPOQBOwGi74jeT 576NQK2LN1wUnslwYmfIq/NKPtiC8jk= X-Google-Smtp-Source: APXvYqxhd6gZ5/qeOC2sdqM3/Gd2+H779XYYU2qHPXxYJH2s4/JCAFVAt3rnxHuRz7cUEMCXE+cslA== X-Received: by 2002:a2e:98c6:: with SMTP id s6mr14822721ljj.235.1573478306463; Mon, 11 Nov 2019 05:18:26 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 12sm7590856lju.55.2019.11.11.05.18.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Nov 2019 05:18:25 -0800 (PST) Subject: Re: Broken ZFS boot on upgrade To: Jonathan Anderson Cc: "freebsd-fs@FreeBSD.org" References: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> <20191111131021.GC70914@bagstock.jonandchrissy.ca> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Mon, 11 Nov 2019 15:18:24 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <20191111131021.GC70914@bagstock.jonandchrissy.ca> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47BWfD4zNyz49NN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.12 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[172.208.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.12)[ip: (-0.35), ipnet: 209.85.128.0/17(-3.19), asn: 15169(-2.00), country: US(-0.05)]; RWL_MAILSPIKE_POSSIBLE(0.00)[172.208.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 13:18:30 -0000 On 11/11/2019 15:10, Jonathan Anderson wrote: > On 11/11, Andriy Gapon wrote: >> Could it be that you have 2TB+ disk(s) and a relatively old BIOS ? > > The first two vdevs use 1 TiB disks, but the third vdev has 3 TiB disks, so that > sounds like a possible explanation... perhaps the contents of /boot previously > resided on one of the 1 TiB vdevs but the new /boot lives on the new vdev > post-upgrade. Is there a zfs admin command to ask which vdev(s) a file or > directory's blocks reside on? You need to find out the inode number first (ls -i or stat -s). Then you can use zdb -ddddd For example: $ zdb -ddddd rpool/ROOT/20171102 4557 ... 0 L1 0:16e1304000:2000 20000L/2000P F=102 B=26531456/26531456 0 L0 0:2d155d000:20000 20000L/20000P F=1 B=26531456/26531456 20000 L0 0:2d1c91000:20000 20000L/20000P F=1 B=26531456/26531456 40000 L0 0:2d1cb1000:20000 20000L/20000P F=1 B=26531456/26531456 60000 L0 0:129eeef000:20000 20000L/20000P F=1 B=26531456/26531456 80000 L0 0:129f138000:20000 20000L/20000P F=1 B=26531456/26531456 a0000 L0 0:129ef0f000:20000 20000L/20000P F=1 B=26531456/26531456 c0000 L0 0:129f1c3000:20000 20000L/20000P F=1 B=26531456/26531456 e0000 L0 0:129f2c6000:20000 20000L/20000P F=1 B=26531456/26531456 [... lots of ouput ...] 0:129f2c6000:20000 is a block pointer. 0 is a vdev number, typically it's n-th device zpool status output. 129f2c6000 is a hexadecimal offset within a vdev, in bytes. 20000 is a hexadecimal block size. > My BIOS is from 2016, so not "old", but possibly old enough? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Nov 11 14:52:33 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B95B11B7592 for ; Mon, 11 Nov 2019 14:52:33 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 47BYkm558Dz4GVP for ; Mon, 11 Nov 2019 14:52:32 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id xABEqS6Z075664; Mon, 11 Nov 2019 14:52:28 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id xABEqRUY023105; Mon, 11 Nov 2019 14:52:27 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id xABEqRXl023101; Mon, 11 Nov 2019 14:52:27 GMT Date: Mon, 11 Nov 2019 14:52:27 GMT Message-Id: <201911111452.xABEqRXl023101@higson.cam.lispworks.com> From: Martin Simmons To: Jonathan Anderson CC: freebsd-fs@freebsd.org In-reply-to: <20191111131248.GD70914@bagstock.jonandchrissy.ca> (message from Jonathan Anderson on Mon, 11 Nov 2019 09:42:48 -0330) Subject: Re: Broken ZFS boot on upgrade References: <201911111153.xABBrEQB021531@higson.cam.lispworks.com> <20191111131248.GD70914@bagstock.jonandchrissy.ca> X-Rspamd-Queue-Id: 47BYkm558Dz4GVP X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.95 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.956,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.973,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; IP_SCORE(-0.02)[country: GB(-0.08)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 14:52:33 -0000 >>>>> On Mon, 11 Nov 2019 09:42:48 -0330, Jonathan Anderson said: > IronPort-SDR: ceFSuzFcg1v+wJx9uExExUpnIS9KKyHX2ru5QfYI5wuUSU/1dtZ5bBhqndGGrT9oQpi3iWnC5y > > On 11/11, Martin Simmons wrote: > > >>>>> On Sun, 10 Nov 2019 22:16:26 -0330, Jonathan Anderson said: > > > > > > but when I run `lszfs > > > zroot/ROOT/default` from the loader prompt it gives me an empty result (so, > > > e.g., no /boot). > > > > FWIW, I think that is normal unless you have created a child dataset of > > zroot/ROOT/default. > > Oh I see, so `lszfs` lists datasets rather than contents? Yes. > > You can try the ls command, e.g. > > > > ls > > ls /boot > > ls /boot/kernel > > ls /boot/kernel/kernel > > These commands resulted in "i/o error - all block copies unavailable" errors. That suggests it is having problem accessing the root directory, not just the boot files. __Martin From owner-freebsd-fs@freebsd.org Mon Nov 11 20:14:02 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 28BE81BD832 for ; Mon, 11 Nov 2019 20:14:02 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) Received: from esa07.ucs.mun.ca (esa07.ucs.mun.ca [134.153.136.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Bhsh2MhVz4cM1 for ; Mon, 11 Nov 2019 20:14:00 +0000 (UTC) (envelope-from jonathan.anderson@mun.ca) IronPort-SDR: Jn8I7aimyPFCiTiKnktTh4UVR7WUo/CmMb+p6Iaa4/2nuHot7TCQ3uXEArGNc6zxIHqsv+rxf/ xFBj5eAOZDDXE0H9pESkcFfoC16oHY6Oq1OexYWgxAy+LV5n0f5HtoLAyEXTJuB7YYsXOJhtFf +T+En1XylhU1QnT4HYg+9zYl1rmNYiwdXY0sroOszJHYWl3eFO7WqWAN4qPFx5qTTuxaoIo4gK Lmur9QcfUWD7pJAVJWkvJ8M4HA5ayK/CYNLVszUiSsBXzrPShxhaootrkODoIBsJg/KRt3yzPe mhs= X-IronPort-AV: E=Sophos;i="5.68,293,1569292200"; d="scan'208";a="19610839" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-qk1-f198.google.com ([209.85.222.198]) by smtp07.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2019 16:43:50 -0430 Received: by mail-qk1-f198.google.com with SMTP id a186so8556587qkb.18 for ; Mon, 11 Nov 2019 12:13:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=amC9uFo3vRqZuj9QBGBMtACBJWbBy+kmYH+tHB4N5mI=; b=m2B13qCjuK8aSD+3+5jdQw/egByUXQCX4RScVcpmOI1tuH1mm+VA7N0Yu9R7BSCoq8 jrTjx8XX+vkRl3Fqx3FkDZERsP6yypNlqRAL7GiWGc3JLRvX+AFMFhNfa1hdBF0CziBR zMaLtToj1s/eRU4D0iog+idqiTfjk9FOF8DHDLsc/aWwxXia1ZpdGsszlu3Oc07DOPik NOO2hslkkM/MGW9t16PGcpDfZeJ3GBZ+v8oyx7oQDhGBRp2xx3j2LetoqUWdaOu1Zzdt DQROBpO1mkHgrySRELbKw05uzMm4IlHB+ukGEAn18g2dMUgCQ2i6nxq4S20PEZ1vvG2W MhAw== X-Gm-Message-State: APjAAAWJKnxgKrQR9EhDrmTH+tcdDBYGJxmVQFNo8MY76SBRR3qiBEa+ Uwt+wpGPcfjIQ9qVDrrkiefQeuWwh65P4N7FTbxPUdDzKhbcohH1TkLysKQy76fiktTswKpdUzD VCLTaUO35DdNFsj2j X-Received: by 2002:ac8:342b:: with SMTP id u40mr27383799qtb.87.1573503228534; Mon, 11 Nov 2019 12:13:48 -0800 (PST) X-Google-Smtp-Source: APXvYqy8WDOto0aHYpD0KiRTmyqNQfKhIkUrzcITCKlfTkSmGLCi0gnGQdsPYGQ+s0t5H+y8SPpT5w== X-Received: by 2002:ac8:342b:: with SMTP id u40mr27383770qtb.87.1573503228171; Mon, 11 Nov 2019 12:13:48 -0800 (PST) Received: from bagstock.jonandchrissy.ca (CPE00fc8db75533-CM00fc8db75530.cpe.net.cable.rogers.com. [174.117.179.42]) by smtp.gmail.com with ESMTPSA id r36sm10095298qta.27.2019.11.11.12.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 12:13:47 -0800 (PST) Date: Mon, 11 Nov 2019 16:43:45 -0330 From: Jonathan Anderson To: Andriy Gapon Cc: "freebsd-fs@FreeBSD.org" Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111201345.GA79423@bagstock.jonandchrissy.ca> References: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> <20191111131021.GC70914@bagstock.jonandchrissy.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47Bhsh2MhVz4cM1 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jonathan.anderson@mun.ca has no SPF policy when checking 134.153.136.27) smtp.mailfrom=jonathan.anderson@mun.ca X-Spamd-Result: default: False [-1.25 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.991,0]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[27.136.153.134.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.16)[ipnet: 134.153.0.0/16(-0.40), asn: 6579(-0.32), country: CA(-0.09)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 20:14:02 -0000 On 11/11, Andriy Gapon wrote: > You need to find out the inode number first (ls -i or stat -s). > Then you can use zdb -ddddd That's very helpful, thanks! I've attempted to do this, but zdb is giving surprising errors. My root dataset is mounted at /mnt/zroot: zroot/ROOT/default on /mnt/zroot (zfs, local, noatime, nfsv4acls) and ls -i /mnt/zroot gives me inode numbers for things in / (including /boot), but when I execute zdb with that dataset and inode number I get an error: # zdb -ddddd zroot/ROOT/default 4495 failed to own dataset 'zroot/ROOT/default': No such file or directory zdb: can't open 'zroot/ROOT/default': No such file or directory This despite zpool status saying that everything is fine, me being able to run `find /boot` with no errors, etc. I get the same error whether the dataset is mounted or unmounted. At this point I'm probably best off flattening this system and restoring my most recent backup snapshot, but this behaviour is truly curious. Thanks to everyone for your help, Jon -- Jonathan Anderson jonathan@FreeBSD.org From owner-freebsd-fs@freebsd.org Tue Nov 12 18:45:17 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3B6FA1B577B for ; Tue, 12 Nov 2019 18:45:17 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 47CGrr0ZRkz3QBg; Tue, 12 Nov 2019 18:45:15 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id xACIjBwT009156; Tue, 12 Nov 2019 18:45:11 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id xACIjB1h008451; Tue, 12 Nov 2019 18:45:11 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id xACIjAVo008447; Tue, 12 Nov 2019 18:45:10 GMT Date: Tue, 12 Nov 2019 18:45:10 GMT Message-Id: <201911121845.xACIjAVo008447@higson.cam.lispworks.com> From: Martin Simmons To: Jonathan Anderson CC: avg@FreeBSD.org, freebsd-fs@FreeBSD.org In-reply-to: <20191111201345.GA79423@bagstock.jonandchrissy.ca> (message from Jonathan Anderson on Mon, 11 Nov 2019 16:43:45 -0330) Subject: Re: Broken ZFS boot on upgrade References: <1cb4895b-c84d-6204-18fa-53eac7195ad6@FreeBSD.org> <20191111131021.GC70914@bagstock.jonandchrissy.ca> <20191111201345.GA79423@bagstock.jonandchrissy.ca> X-Rspamd-Queue-Id: 47CGrr0ZRkz3QBg X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.948,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.97)[-0.968,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; IP_SCORE(-0.02)[country: GB(-0.08)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2019 18:45:17 -0000 >>>>> On Mon, 11 Nov 2019 16:43:45 -0330, Jonathan Anderson said: > > On 11/11, Andriy Gapon wrote: > > You need to find out the inode number first (ls -i or stat -s). > > Then you can use zdb -ddddd > > That's very helpful, thanks! I've attempted to do this, but zdb is giving > surprising errors. My root dataset is mounted at /mnt/zroot: > > zroot/ROOT/default on /mnt/zroot (zfs, local, noatime, nfsv4acls) > > and ls -i /mnt/zroot gives me inode numbers for things in / (including /boot), > but when I execute zdb with that dataset and inode number I get an error: > > # zdb -ddddd zroot/ROOT/default 4495 > failed to own dataset 'zroot/ROOT/default': No such file or directory > zdb: can't open 'zroot/ROOT/default': No such file or directory > > This despite zpool status saying that everything is fine, me being able to run > `find /boot` with no errors, etc. I get the same error whether the dataset is > mounted or unmounted. Try adding the -e option as well. It might also be useful to check the zdb output for inum 1. __Martin