From nobody Fri Sep 6 18:54:32 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0llX2p0pz5VVJY for ; Fri, 06 Sep 2024 18:54:48 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0llX1123z4jsT for ; Fri, 6 Sep 2024 18:54:47 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; none dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=6ahwzrwrw5gza7THBUUTSaiuPSVxdU6Qlwyv3witcGA=; b=yZc3KZ5k63CrloPlqt9nlOTEPIlZn+9NtrqWjh98NL+5sReSyltmPdJPuCOJCL9l0AiHH5ljfdoGB2Wh9Pir511aAE9BXpoOm8h3V+ilD236dji81LtO/sJaZ77PejqZE19GuQzf3oGw7CGmeVGfA/y252ZSrom57LnngL8FzbobOaLdfjSAu38mjEoGQwaY/V7vi2D+oQGPiQSynhtqa774lJMwipR7CjdfP707gHtP3KFo5vIYbK6bGU 2JHuSsYG2NCaZgxTKkJ2OTY3z7T6e/yzUqKCwk1Kr5TgrqyG5r3wMyYnS6lsjAsB/qijR+xw23X2anQ03oArJrqmH/OA== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 18:54:45 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 6cfd1947 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 14:54:43 -0400 (EDT) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 From: Chris Ross In-Reply-To: <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> Date: Fri, 6 Sep 2024 14:54:32 -0400 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> To: Wes Morgan X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US] X-Rspamd-Queue-Id: 4X0llX1123z4jsT > On Sep 6, 2024, at 14:39, Wes Morgan wrote: >=20 >=20 > You should make the changes to your /boot/loader.conf as suggested = earlier by Freddie Cash and reboot. This will eliminate all the = confusion with diskid. Then run "zpool clear", which, if da3 is still = online and not completely dead, the pool should come out of the faulted = state. Check zpool status to look for this alleged replacement in = progress. If it is truly trying to replace a device, it should show up = in zpool status with the actual device, or the guid if it can't find the = device. I saw and appreiciated that response, but didn=E2=80=99t respond on that = thread because I don=E2=80=99t _want_ to turn all of those things off. = At least, I don=E2=80=99t want to refer to everything by the = auto-numbered da# that I think that will cause. And, Freddie, your = comment about GPT partition labels I think doesn=E2=80=99t apply because = I don=E2=80=99t have GPT on my disks. Just all one big ZFS device. = This is why I=E2=80=99m looking at glabel=E2=80=99s generic labeling = now. The former da3 is off-line, out of the chassis. I replaced a disk in a = full chassis, having them both online at the same time is not possible. = That drive in ZFS=E2=80=99s mind is only faulted because I tried = =E2=80=9Czpool offline -f=E2=80=9D on it to see if that helped. > If you have initiated a replace, and the replacing disk has now been = "lost" or unlabeled, you are in a bind. I ran into this problem many = years ago, and I thought it was fixed, but the bug was called something = like "can't replace a replacing vdev". I ultimately solved my problem by = manually editing a fake vdev to have the same guid as the missing = device, restarting the replace and then canceling it before zfs realized = it was fake. But, I am almost certain that zpool cancel can do this now, = with the guid. I didn=E2=80=99t initiate a replace until after the disks were = physically changed. Although in this conversation realize that things = likely got confused by the replacement in the kernel=E2=80=99s mind of = da3 with what used to be da4. :-/ > If da10 has a label that says it is in the pool, it is probably the = "replacing" vdev and should be picked up=E2=80=A6 Da10, now also /dev/label/drive03, seems to think it=E2=80=99s in the = pool somewhere, according to zdb -l. But I=E2=80=99m not sure if this helps. And, following your other = message saying I shouldn=E2=80=99t put labels on disks that are to be used in their entirety as ZFS devices, I=E2=80=99v= e deleted that label and zlabelclear=E2=80=99d this device now. (since the zfs label still had = the /dev/label/ path in it) - Chris