From nobody Mon Jun 20 13:42:13 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 93FDD85BED3 for ; Mon, 20 Jun 2022 13:42:17 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from mail.jld3.net (mail.jld3.net [45.55.236.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LRW6K0j4zz4WVm for ; Mon, 20 Jun 2022 13:42:17 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from localhost (localhost [127.0.0.1]) by mail.jld3.net (Postfix) with ESMTP id E05D34160D for ; Mon, 20 Jun 2022 07:42:16 -0600 (MDT) X-Virus-Scanned: amavisd-new at jld3.net Received: from mail.jld3.net ([127.0.0.1]) by localhost (mail.jld3.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id AnTR-zfmCHcm for ; Mon, 20 Jun 2022 07:42:16 -0600 (MDT) Received: from [172.21.35.205] (c-24-9-144-115.hsd1.co.comcast.net [24.9.144.115]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: jld) by mail.jld3.net (Postfix) with ESMTPSA id 4483D4087C for ; Mon, 20 Jun 2022 07:42:16 -0600 (MDT) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.jld3.net 4483D4087C DKIM-Filter: OpenDKIM Filter v2.11.0 mail.jld3.net 4483D4087C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jld3.net; s=8d052f02dde2; t=1655732536; bh=NMyX2gQO1bvrmgOA/0GQBxhL2MELzHaPBA+yr0OKIyk=; h=From:To:Subject:Date:Message-ID:MIME-Version:From; b=WAWLIRiehzn+nlGFrl7U0wQDLB9K6r6nzZLlpyAhWvwLutO+7uuTpl4YldAUWTtS7 db8+CLBriFn0lMa8ocTAdq4A7qGjm3vYBjiyWZnTE2cuSK+qS0QuUAc1m/WV5VlHuK 1h+ZFn6IqBUL9ryZJVoeU0emNE1G1jy6XsBynoNg= From: "John Doherty" To: freebsd-fs@FreeBSD.org Subject: "spare-X" device remains after resilvering Date: Mon, 20 Jun 2022 07:42:13 -0600 X-Mailer: MailMate (1.13.2r5673) Message-ID: <34A91D31-1883-40AE-82F3-57B783532ED7@jld3.net> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Rspamd-Queue-Id: 4LRW6K0j4zz4WVm X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jld3.net header.s=8d052f02dde2 header.b=WAWLIRie; dmarc=pass (policy=quarantine) header.from=jld3.net; spf=pass (mx1.freebsd.org: domain of bsdlists@jld3.net designates 45.55.236.93 as permitted sender) smtp.mailfrom=bsdlists@jld3.net X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[jld3.net:s=8d052f02dde2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:45.55.236.93:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[jld3.net:+]; DMARC_POLICY_ALLOW(-0.50)[jld3.net,quarantine]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:14061, ipnet:45.55.192.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[24.9.144.115:received] X-ThisMailContainsUnwantedMimeParts: N Hi, I have a zpool that currently looks like this (some lines elided for brevity; all omitted devices are online and apparently fine): pool: zp1 state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: resilvered 1.76T in 1 days 00:38:14 with 0 errors on Sun Jun 19 22:31:46 2022 config: NAME STATE READ WRITE CKSUM zp1 DEGRADED 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 ... gpt/disk9 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 gpt/disk10 ONLINE 0 0 0 ... gpt/disk19 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 gpt/disk20 ONLINE 0 0 0 ... gpt/disk29 ONLINE 0 0 0 raidz2-3 DEGRADED 0 0 0 gpt/disk30 ONLINE 0 0 0 3343132967577870793 OFFLINE 0 0 0 was /dev/gpt/disk31 ... spare-9 DEGRADED 0 0 0 6960108738988598438 OFFLINE 0 0 0 was /dev/gpt/disk39 gpt/disk41 ONLINE 0 0 0 spares 16713572025248921080 INUSE was /dev/gpt/disk41 gpt/disk42 AVAIL gpt/disk43 AVAIL gpt/disk44 AVAIL My question is why the "spare-9" device still exists after the resilvering completed. Based on past experience, my expectation was that it would exist for the duration of the resilvering and after that, only the "gpt/disk41" device would appear in the output of "zpool status." I also expected that when the resilvering completed, the "was /dev/gpt/disk41" device would be removed from the list of spares. I took the "was /dev/gpt/disk31" device offline deliberately because it was causing a lot of "CAM status: SCSI Status Error" errors. Next step for this pool is to replace that with one of the available spares but I'd like to get things looking a little cleaner before doing that. I don't have much in the way of ideas here. One thought was to export the pool and then do "zpool import zp1 -d /dev/gpt" and see if that cleaned things up. This system is running 12.2-RELEASE-p4, which I know is a little out of date. I'm going to update it 13.1-RELEASE soon but the more immediate need is to get this zpool in good shape. Any insights or advice much appreciated. Happy to provide any further info that might be helpful. Thanks.