From eugen@grosbein.net Tue May 3 12:43:00 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 223941AA8554 for ; Tue, 3 May 2022 12:43:10 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kt04F1ZfLz3FpQ for ; Tue, 3 May 2022 12:43:09 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.16.1/8.16.1) with ESMTPS id 243Ch5sR053296 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 3 May 2022 12:43:06 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: eugene@zhegan.in Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.16.1/8.16.1) with ESMTPS id 243Ch526050325 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 3 May 2022 19:43:05 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: cannot remove/detach missing disk from zmirror: no valid replicas To: "Eugene M. Zheganin" , freebsd-stable@freebsd.org References: <5f9fcb73-b824-b9be-8a40-41401a774a04@zhegan.in> From: Eugene Grosbein Message-ID: Date: Tue, 3 May 2022 19:43:00 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: <5f9fcb73-b824-b9be-8a40-41401a774a04@zhegan.in> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.2 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4Kt04F1ZfLz3FpQ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=fail (mx1.freebsd.org: domain of eugen@grosbein.net does not designate 2a01:4f8:c2c:26d8::2 as permitted sender) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-1.28 / 15.00]; ARC_NA(0.00)[]; R_SPF_FAIL(1.00)[-all]; FREEFALL_USER(0.00)[eugen]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.18)[-0.178]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-stable]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N 02.05.2022 12:23, Eugene M. Zheganin wrote: > Hello, > > > Any chance I can sole this without replacing ? (Yeah, I accidentally issued "attach" instead of "replace" being fully confident that I will be able to "detach" later. I couldn't be more wrong.) > > > [root@replica:~]# zpool status > pool: zfsroot > state: DEGRADED > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Thu Jan 1 03:00:06 1970 > 188G scanned at 122B/s, 21.7G issued at 14B/s, 2.03T total > 21.8G resilvered, 1.04% done, no estimated completion time > config: > > NAME STATE READ WRITE CKSUM > zfsroot DEGRADED 0 0 0 > mirror-0 DEGRADED 0 0 0 > gpt/zfsroot1 ONLINE 0 0 0 (resilvering) > gpt/zfsroot0 UNAVAIL 0 0 0 cannot open > diskid/DISK-31P58VAASp3 ONLINE 0 0 0 (resilvering) > > errors: 6 data errors, use '-v' for a list > [root@replica:~]# zpool detach zfsroot gpt/zfsroot0 > cannot detach gpt/zfsroot0: no valid replicas > [root@replica:~]# uname -a > FreeBSD replica.scorista.ru 13.1-RC5 FreeBSD 13.1-RC5 releng/13.1-n250141-2e9ad6042be GENERIC amd64 > [root@replica:~]# zpool remove zfsroot gpt/zfsroot0 > cannot remove gpt/zfsroot0: operation not supported on this type of pool > [root@replica:~]# zpool split -R /newroot zfsroot newroot diskid/DISK-31P58VAASp3 > Unable to split zfsroot: pool is busy > > [root@replica:~]# Reproducing... # truncate -s 5G file1.img file2.img file3.img # zpool create ztest mirror $(realpath file1.img) $(realpath file2.img) # dd if=/dev/urandom bs=1m count=$((4*1024)) of=/ztest/file 4096+0 records in 4096+0 records out 4294967296 bytes transferred in 26.212189 secs (163853817 bytes/sec) # zpool list ztest NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ztest 4.50G 4.00G 509M - - 64% 88% 1.00x ONLINE - # zpool export ztest # rm file2.img # zpool import -d . ztest # zpool status -v ztest pool: ztest state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q config: NAME STATE READ WRITE CKSUM ztest DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 /home/eugen/file1.img ONLINE 0 0 0 1055108590663069279 UNAVAIL 0 0 0 was /home/eugen/file2.img # zpool attach ztest $(realpath file1.img) $(realpath file3.img) && zpool status -v pool: ztest state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue May 3 12:38:54 2022 4.00G scanned at 4.00G/s, 16.3M issued at 16.3M/s, 4.00G total 12.3M resilvered, 0.40% done, 00:04:10 to go config: NAME STATE READ WRITE CKSUM ztest DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 /home/eugen/file1.img ONLINE 0 0 0 1055108590663069279 UNAVAIL 0 0 0 was /home/eugen/file2.img /home/eugen/file3.img ONLINE 0 0 0 (resilvering) errors: No known data errors So, how did you achieve your situation when you have 6 data errors and two parts of mirror in both ONLINE and (resilvering) state? I suspect "6 data errors" are your problem.