From owner-freebsd-fs@freebsd.org Mon Nov 11 01:46:43 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7477E1BF9A1 for ; Mon, 11 Nov 2019 01:46:43 +0000 (UTC) (envelope-from p15jra@mun.ca) Received: from esa06.ucs.mun.ca (esa06.ucs.mun.ca [134.153.136.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.mun.ca", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47BDJ24DfDz4cLY for ; Mon, 11 Nov 2019 01:46:42 +0000 (UTC) (envelope-from p15jra@mun.ca) IronPort-SDR: CV17VZTCHizXiLJ6joelu8VA0eQtUujITNPNTdz6ES2BPfJwtsforsaV+MIKIaWbO+/xbm3J78 EGTEl30YTD+/zz8QDZVNyBSoXaxvpQ45pJFJ9Ld0okGexeTw60hux2t+gPcxFSEHKoALVFh/a0 P+cuJ5K3XOtHu4J0pRhJUBWTs94xsVMqnYBbf6BwU6aav36iqtEGkzy7n896J0IVyIveT9d3J/ DhimhRUgPK+3KWZccbW7zWIeels1QKZ0rtdRBhIwTvT74Ek2PfAmTs6e/ZukICqi0BRFdbVh/h eUM= X-IronPort-AV: E=Sophos;i="5.68,290,1569292200"; d="scan'208,217";a="20780728" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from mail-yw1-f72.google.com ([209.85.161.72]) by smtp06.ucs.mun.ca with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Nov 2019 22:16:41 -0430 Received: by mail-yw1-f72.google.com with SMTP id s128so7126709ywf.0 for ; Sun, 10 Nov 2019 17:46:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=W0wHFBN75ucX9h8BpITjwFTnYX4qif0RauNjBtDYyZU=; b=YUvS6VaFKqWXBf+PxA9YUGr3pAmh+F4KLfdTxabSz1DS5NC4I9dhVCFz+TUZ6CDRw2 ZnESQBtAizslDZs+OpVK/diy+Y/hYep0ZReH8EVkRQmFmhnp01N2cyzipHCTx5i3QbHy rAfvDsMG++CLcJ6/5TvTy3/QNmHe1/IFMQSXTBm40UT/IzZxI7QtPDM8sLg3gEEl46LD veaeaK2XicG6WcQxaGrL7sl/IeKSv9soQfns+xH/mFygIWKVOH0syw1CHqjQozC7njd7 9iD3fGJvCOgmfJZHTwG6YLRnJGTfnUJtSVs/al82ASidoc74IYgEFthEiWQMAwwvcyHl RN4Q== X-Gm-Message-State: APjAAAUrRh3eUXmjlxHtT3zqLJCisROoi/E2vOD/6RRJxexCY97joyyZ OLf/DnJeHqTi03o5P147bvvEjC8n4avEQEcvZz5PEd9NVOmcSn2nbIZonx6p+r0Yt2ssEmuNRUS kOk9UGn/IzTV6XfqomLOr3zLHA/lijxK3 X-Received: by 2002:a25:7a47:: with SMTP id v68mr18475350ybc.438.1573436798546; Sun, 10 Nov 2019 17:46:38 -0800 (PST) X-Google-Smtp-Source: APXvYqyejoWfXwMwYMWmOD9gxcpX89gsFNWKrtzWJhVwUDoyaA00QZm17V1zUw6UOXBi/BVBn0juiX1pysr9l7QlvpQ= X-Received: by 2002:a25:7a47:: with SMTP id v68mr18475340ybc.438.1573436797972; Sun, 10 Nov 2019 17:46:37 -0800 (PST) MIME-Version: 1.0 From: Jonathan Anderson Date: Sun, 10 Nov 2019 22:16:26 -0330 Message-ID: Subject: Broken ZFS boot on upgrade To: "freebsd-fs@FreeBSD.org" X-Rspamd-Queue-Id: 47BDJ24DfDz4cLY X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of p15jra@mun.ca has no SPF policy when checking 134.153.136.26) smtp.mailfrom=p15jra@mun.ca X-Spamd-Result: default: False [3.57 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[mun.ca]; AUTH_NA(1.00)[]; URI_COUNT_ODD(1.00)[3]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.79)[0.794,0]; NEURAL_SPAM_LONG(0.75)[0.748,0]; RCVD_IN_DNSWL_NONE(0.00)[26.136.153.134.list.dnswl.org : 127.0.10.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[jonathan.anderson@mun.ca,p15jra@mun.ca]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:6579, ipnet:134.153.0.0/16, country:CA]; FROM_NEQ_ENVFROM(0.00)[jonathan.anderson@mun.ca,p15jra@mun.ca]; IP_SCORE(-0.17)[ipnet: 134.153.0.0/16(-0.43), asn: 6579(-0.35), country: CA(-0.09)] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2019 01:46:43 -0000 Hello friendly FreeBSD people, I=E2=80=99ve gone and done it: I upgraded a key ZFS-on-root machine from 11= .2 to 12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on this list might be able to help me figure out what's wrong and what I can do about it? It looks like the ZFS code in the bootloader can't find anything in my root directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If I boot a rescue image from USB I can mount everything in the pool (`zpool import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs zroot/ROOT/default` from the loader prompt it gives me an empty result (so, e.g., no /boot). Booting fails with messages such as, "i/o error - all bloc= k copies unavailable". My pool consists of three mirrored vdevs, in which the first mirror uses GP= T partitioning (for the boot partitions) and the other two mirrors use whole disks. I recall reading somewhere that the bootloader ZFS code doesn't like non-partition-based vdevs... is that true? If so, perhaps the issue is that my upgrade caused /boot to live on one of the newer whole-disk-based mirrors, hiding it from the bootloader's view? I've backed up all of my data (a 28h process) and am resigned to re-installing if need be, but I would like to understand where I went wrong to avoid making the same mistakes again. What would you recommend for someone building a new array-of-mirrors pool? I see that the Handbook says that certain things *can* be done, but it=E2=80=99s not clear to me whether= they should or even must be done, e.g.: > partitions with bootcode and file systems needed for booting can be added. This allows booting from disks that are also members of a pool. There is no performance penalty on FreeBSD when using a partition rather than a whole disk The Handbook suggests that it's possible to break disks into multiple partitions that are added separately to a vdev, but is this a sensible thing to do? Is there any other ZFS lore that hasn't made it to the Handbook but that ought to be kept in mind from the outset? Thanks, Jon-- Assistant Professor Electrical and Computer Engineering Memorial University of Newfoundland https://www.engr.mun.ca/~anderson