From owner-freebsd-questions@freebsd.org Fri Dec 13 20:57:39 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1BC21D6B00 for ; Fri, 13 Dec 2019 20:57:39 +0000 (UTC) (envelope-from matt@conundrum.com) Received: from mail-lj1-x230.google.com (mail-lj1-x230.google.com [IPv6:2a00:1450:4864:20::230]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47ZNKG2sGvz4Vbv for ; Fri, 13 Dec 2019 20:57:38 +0000 (UTC) (envelope-from matt@conundrum.com) Received: by mail-lj1-x230.google.com with SMTP id u17so143516lja.4 for ; Fri, 13 Dec 2019 12:57:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=conundrum-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=8UKgfrTFlngPV6jvCmWt0cXe9afLoSSZh5km7OAhAfs=; b=Q7tjWebQN43SO4bJUeOWsQXAgSir5A0EZV9LVvszY34CpUOaaSf5Bd3oH7Xh7oK80C 8CpZ7MaFUgYVM/rLbDPLAi1qQENQpoj/48fxZIOfs794je3xlFtSp1ikJ7d80yOk0WLd LrehMZb1UeBuyt/UXmBQZ02u+1BBTggROM7k492IDDwI4hMU3ZPfTjK2LIKJ3L9ZnaGg HMj8/psomwAsx53+tcwG7KCksrxVa93IRmkhHRKlt7ixIbdtjEA6nBs6PT8Sh5VNspv3 VMexI+9kjsi9KToOTLeuQljnOnVsL3NCOqktkGKWLV0PoiiQCBJJUFIefPqRFO7d9Xqt 9ZLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=8UKgfrTFlngPV6jvCmWt0cXe9afLoSSZh5km7OAhAfs=; b=huh6p3eDONwFQCTOtfFxLGUkBoW2DpEpDrb7zmbdACjKTQP+eg22+XVgDFHbREzqYi YUiRIReC4aKpvxS5mxhOPjvZaiPuFkD/suzS1AtIlUb1AZXNOpV3MWx5tpZV9GOL7Aqv QMZ/XsH27m+dXfbhnf0G+mq+ZvpFqsZMBn1j7OTW9mQTTd3pTe2+mPBQcK2bWTT/Uw6a 7o22vutVU/FKUVY/j4YrSUnGjaB6zi2lGv6F4qw+23rIDmH+w2BbV2NaJ6d7Z3GeAZ4m qRk0UMIiq9DyU16vlmVsGYijs4H+VCTZ7O1k17IQcDmBVk3fjCm48Qnhz6zs9r5iqAEZ Q+wA== X-Gm-Message-State: APjAAAXSKAYxq2gv1tZz9YVUE5hDjadWY20N6OlDXvtLZCiEElm1htkQ MTDhPn6g2jpTJ6lzBWRdtYd/to/r3TyVTF1fka6X9j+lR0ikFw== X-Google-Smtp-Source: APXvYqwgxxLpP8REorgclrIw0Smk57MfLNISDc46ZiHv6bDrg8cdr1dFi+loOK50dp6nU97o8zlIWmzXS0JzMunUp04= X-Received: by 2002:a2e:884d:: with SMTP id z13mr11202980ljj.116.1576270654992; Fri, 13 Dec 2019 12:57:34 -0800 (PST) MIME-Version: 1.0 From: Matthew Pounsett Date: Fri, 13 Dec 2019 15:57:24 -0500 Message-ID: Subject: Root volume renumbered unexpectedly, no longer boots To: freebsd-questions@freebsd.org X-Rspamd-Queue-Id: 47ZNKG2sGvz4Vbv X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=conundrum-com.20150623.gappssmtp.com header.s=20150623 header.b=Q7tjWebQ; dmarc=none; spf=none (mx1.freebsd.org: domain of matt@conundrum.com has no SPF policy when checking 2a00:1450:4864:20::230) smtp.mailfrom=matt@conundrum.com X-Spamd-Result: default: False [-5.05 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[conundrum-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[conundrum.com]; DKIM_TRACE(0.00)[conundrum-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[0.3.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.75)[ip: (-9.14), ipnet: 2a00:1450::/32(-2.67), asn: 15169(-1.91), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Dec 2019 20:57:39 -0000 We have a large-ish FreeBSD 11.2-p7 file server with two 24-disk ZFS pools (20 live 4 spares each) and a single SSD boot volume. Yesterday we pulled some dead drives from the ZFS pools and replaced them with new drives intended to become spares. After powering the system back up, it looks like the boot volume has been renumbered from da0 to da4. I thought renumbering like this wasn't supposed to happen for at least the last decade, since ATA_STATIC_ID was introduced to the kernel, but there's little doubt that's what's happened. Automatic boot now fails and drops to the third stage loader prompt when the kernel tries to mount the root volume from ufs:/dev/da0p2. I can manually try to mount the root volume as ufs:/dev/da4p2, and the system begins to load the root volume, but then hangs. The only two lines printed after loading da4 are related to loading up the ZFS pools. I can't reproduce the messages again now (explained below), so quoting them verbatim isn't possible, but they're related to the ZFS version being behind and suggesting I upgrade the pools. The messages themselves are not unusual,and I'm used to seeing similar messages in the 'zpool status' output for a while now. What is unusual is that the system seems to hang at this point. I'm concerned that the re-ordering of drives might be causing problems for the system trying to put the ZFS pools back together. I don't really know, though. Does anyone have any insight into what's going on here? There is a new wrinkle... since booting from a USB stick so that I could get into the box and double-check some things, and confirm the location of the root volume, the BIOS no longer seems to see da4 as a potential boot volume. I'm hoping that goes back to the way it was once the USB stick is removed. At the moment I have no way to even get the box to try/fail to boot from its normal boot volume. The machine is many thousands of miles remote, so I haven't tried to do this yet... I can invoke some remote help once that's necessary. BIOS issue aside, I'm hoping there's a way I can pin this drive back to da0. I don't know how that could be done, but if anyone has any suggestions I'd happily try them. Failing that, I suppose I can just insert a vfs.root.mountfrom option in loader.conf. Can anyone clue me into what's happening here, or suggest some further troubleshooting that will help me gain some insight? Thanks!