From nobody Fri Sep 6 14:30:31 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0dtt2JqMz5VBpb for ; Fri, 06 Sep 2024 14:30:46 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0dts3l8Tz4WYB for ; Fri, 6 Sep 2024 14:30:45 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=relay.distal.com header.s=mail header.b=YAPkIJCX; dmarc=none; spf=pass (mx1.freebsd.org: domain of cross+freebsd@relay.distal.com designates 168.235.95.80 as permitted sender) smtp.mailfrom=cross+freebsd@relay.distal.com dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=lpoCb6LtSfaD5nMc47AkIQCI+71toQspk6kbP1Elwu4=; b=YAPkIJCX2vG24IHo0WYie+CC1dyiVsruDhIH7p3gY4DjuPRtHyEwBui/0UekYksPcwexYxjFLIfD2K5yKQWQoluLZd6+ycmJDGK/tkDmu0C4sKXeP7mIoxxVPj+4jrRLgotEJuN5nlrAThZIk8c7JLT28GiOdoCibs2W6322cDXjSEeKI6d9liHPHVnFOADmT32zM5MsLJwk8twVN1eBTBUK7AxMa5eRgIrrqa5ysS1/4+1gVyUHgYBAPq t8ccB9rXuSxB7QrypKd9UXzo1YIaHZlqH43jQlHIuu5K6Ni8+SNOwhAMcfdevnqzEBR2VH3ynQS1mVqiPRMRw29GvmCQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 14:30:43 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 984b57cb (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Fri, 6 Sep 2024 10:30:42 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 Date: Fri, 6 Sep 2024 10:30:31 -0400 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> To: FreeBSD Filesystems In-Reply-To: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> Message-Id: X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.19 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; FORGED_SENDER(0.30)[cross@distal.com,cross@relay.distal.com]; R_SPF_ALLOW(-0.20)[+a:relay.dynu.com]; R_DKIM_ALLOW(-0.20)[relay.distal.com:s=mail]; MIME_GOOD(-0.10)[text/plain]; TAGGED_FROM(0.00)[freebsd]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[distal.com]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[relay.distal.com:+]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,cross@relay.distal.com]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4X0dts3l8Tz4WYB Oh, sorry. Failed to indicate versions. I was running 13.2 on amd64, = and while facing this problem have updated to 14.1. The current state is 14.1, and I=E2=80=99m still seeing the same, but I started in 13.2 and = saw this. > On Sep 6, 2024, at 10:24, Chris Ross wrote: >=20 > Hello. I have searched the interwebs a bit and seen tell of this and = others > like it, but I haven=E2=80=99t found a solution. >=20 > I have a pool with three 3-disk raidz1 vols. I want to replace the = disks > in the first vdev with larger disks. I=E2=80=99ve done this before, = but may=E2=80=99ve > done something wrong here. >=20 > I belive I used =E2=80=9Czpool remove tank da3=E2=80=9D, but command = history doesn=E2=80=99t > have that. I=E2=80=99ve used many commands since I started. I = might=E2=80=99ve > =E2=80=9Czpool offline=E2=80=9Dd the device. I=E2=80=99m sorry I = don=E2=80=99t remember the original > command. >=20 > Then replaced the disk and rebooted. This of course renumbered the > disks. :-(. But, finding the new/replacement disk (da10), I try to > =E2=80=9Czpool replace tank da3 da10=E2=80=9D. > This always produces: >=20 > cannot replace da3 with da10: already in replacing/spare config; wait = for completion or use 'zpool detach=E2=80=99 >=20 > Now, I can=E2=80=99t use =E2=80=9Czpool detach=E2=80=9D I=E2=80=99ve = learned because that doesn=E2=80=99t work > on zraid. And I can=E2=80=99t tell what it _thinks_ is happening. I = even > Did a scrub of the pool and let that finish, but am still seeing the > same. >=20 > I have now: >=20 > =E2=80=948<=E2=80=948<=E2=80=948<--- >>>>>>> zpool status -v tank > pool: tank > state: DEGRADED > status: One or more devices are faulted in response to persistent = errors. > Sufficient replicas exist for the pool to continue functioning = in a > degraded state. > action: Replace the faulted device, or use 'zpool clear' to mark the = device > repaired. > scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 = 09:08:34 2024 > config: >=20 > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external = device fault > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 >=20 > errors: No known data errors > =E2=80=948<=E2=80=948<=E2=80=948<=E2=80=94 >=20 > I=E2=80=99ll note that before the switcharoo, the second and third = vdev=E2=80=99s listed > =E2=80=9Cda4 da5 da6=E2=80=9D and =E2=80=9Cda7 da8 da9=E2=80=9D. The = moving around of names caused the > above, which again I=E2=80=99ve seen before, and am fine with. >=20 > (Oh, you can see I ran a =E2=80=9Czpool offline -f=E2=80=9D on it most = recently. But > that was in the list of things I=E2=80=99ve tried that haven=E2=80=99t = helped.) >=20 > Please let me know if someone knows how I=E2=80=99ve gotten to this = state, and > what I need to do to correct it. What is the "in replacing/spare = config=E2=80=9D > meaning? >=20 > - Chris >=20