From nobody Fri Sep 6 16:50:55 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0j0w2NMpz5VVKL for ; Fri, 06 Sep 2024 16:51:12 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0j0v60Vgz430d; Fri, 6 Sep 2024 16:51:11 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; none dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=+vQZwpJzUJyzjDzELQkVWNjqgz2QMiLkmHm0ryW22Js=; b=uN/PdT8mMssH/Ny+0JZuCPUK0v6U3S5MlI0Ks1NuG3O2i8VG2MMBE+kZOBSrmDAVaj2TlW6JWWifBcGwyzMY3kv5TWc/tYDgrufoBDBjSM9J/QGoIGKkwM9Im1Yr2xeW9kPYaujKkjqEBNm441RTiZLsAChEIjiiaUfjG1xS0PHb7wRcPN5XCGwP63SiQR/b2in6k4sBOf7mCyxamdNMg/1YbL6CMy8habBPZnGg+I3WanjZ9Zlzt6GBx/ imRUG0eXRlt31+hC3gP+7NxDaGD5d+a2Ay7Eh4/WmxlEVbePtPFspJO4DbV6rFVmcMz7jS5KrXbZe3VIkUH+/Zt2dpug== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 16:51:09 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 8b2aee94 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 12:51:08 -0400 (EDT) Content-Type: text/plain; charset=us-ascii List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 From: Chris Ross In-Reply-To: Date: Fri, 6 Sep 2024 12:50:55 -0400 Cc: FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> To: Alan Somers , mike tancsa X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US] X-Rspamd-Queue-Id: 4X0j0v60Vgz430d > On Sep 6, 2024, at 11:32, Alan Somers wrote: >=20 > "zpool replace" is indeed the correct command. There's no need to run > "zpool offline" first, and "zpool remove" is wrong. Since "zpool > replace" is still failing, are you sure that da10 is still the correct > device name after all disks got renumbered? If you're sure, then you > might run "zdb -l /dev/da10" to see what ZFS thinks is on that disk. >=20 I can confirm that da10 is still the new disk I put into place of prior = da3. > On Sep 6, 2024, at 11:43, mike tancsa wrote: > I would triple check to see what the devices are that are part of the = pool. I wish there was a way to tell zfs to only display one or the = other. So list out what diskid/DISK-K1GMBN9D, diskid/DISK-K1GMEDMD... = to diskid/DISK-3WJ7ZMMJ are in terms of /dev/da* actually are. I have = some controllers that will re-order the disks on every reboot. glabel = status and camcontrol devlist should help verify camcontrol devlist lets me know that the three HGST drives making up zraid1-1 are da3,da4,da5 and the three WD drives making up zraid1-2 are da6,da7,da8. So, like before, just moved down a number because the prior da3 went away and a new disk in that physical slot became da10. (da9 is a loose JBOD single with ufs on it, previously da10, in slot 12 of 12) da10 is in fact still the disk in slot3 of the chassis, zdb -l shows the below. I did add and remove it as a spare while trying things, that may be why it shows up this way. - Chris % sudo zdb -l /dev/da10 ------------------------------------ LABEL 0 ------------------------------------ version: 5000 name: 'tank' state: 0 txg: 0 pool_guid: 3456317866677065800 errata: 0 hostid: 2747523522 hostname: 'frizzen02.devit.ciscolabs.com' top_guid: 2495145666029787532 guid: 2495145666029787532 vdev_children: 3 vdev_tree: type: 'disk' id: 0 guid: 2495145666029787532 path: '/dev/da10' phys_path: = 'id1,enc@n584b2612f2c321bd/type@0/slot@3/elmdesc@ArrayDevice03' whole_disk: 1 metaslab_array: 0 metaslab_shift: 0 ashift: 12 asize: 22000965255168 is_log: 0 create_txg: 18008413 features_for_read: com.delphix:hole_birth com.delphix:embedded_data create_txg: 18008413 labels =3D 0 1 2 3=