From nobody Tue May 3 18:03:30 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 23B0B1AB0E99 for ; Tue, 3 May 2022 18:03:35 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from elf.hq.norma.perm.ru (mail.norman-retail.ru [128.127.146.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.norma.perm.ru", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kt79y2knpz3LtM for ; Tue, 3 May 2022 18:03:34 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from [192.168.243.7] ([192.168.243.7]) by elf.hq.norma.perm.ru (8.16.1/8.15.2) with ESMTP id 243I3Vf7047152 for ; Tue, 3 May 2022 23:03:31 +0500 (+05) (envelope-from eugene@zhegan.in) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=vivat-retail.ru; s=key; t=1651601012; bh=5wYH1yK8edxcbkqxqPAChuStxSYAOZlAdOD+mQMQa3s=; h=Date:Subject:To:References:From:In-Reply-To; b=QoGdKEHgzYeWt95qJjuC8thf1QZ1ocT2mFhHfuTc2PrscX2kTwyrDPYft4xIVpEyr 2bPC+n/jhtR/Vu1vXVTmFcwpb5AYrQFcUpNOZJdTqJlzPfkbczvkVtZPsyIU5E8Cm5 VuTyfj0UQ+qSXh5Ky+SCxpcO0gfsYUkesaRfoSIs= Message-ID: <29cb2c28-c34b-995f-9771-34d9d11ba9df@zhegan.in> Date: Tue, 3 May 2022 23:03:30 +0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1 Subject: Re: cannot remove/detach missing disk from zmirror: no valid replicas Content-Language: ru To: freebsd-stable@freebsd.org References: <5f9fcb73-b824-b9be-8a40-41401a774a04@zhegan.in> From: "Eugene M. Zheganin" In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Kt79y2knpz3LtM X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=fail ("headers rsa verify failed") header.d=vivat-retail.ru header.s=key header.b=QoGdKEHg; dmarc=none; spf=pass (mx1.freebsd.org: domain of eugene@zhegan.in designates 128.127.146.8 as permitted sender) smtp.mailfrom=eugene@zhegan.in X-Spamd-Result: default: False [-2.29 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; R_DKIM_REJECT(1.00)[vivat-retail.ru:s=key]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_NA(0.00)[zhegan.in]; R_SPF_ALLOW(-0.20)[+a]; DKIM_TRACE(0.00)[vivat-retail.ru:-]; NEURAL_HAM_SHORT(-0.99)[-0.994]; MLMMJ_DEST(0.00)[freebsd-stable]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:212494, ipnet:128.127.146.0/24, country:RU]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello, 03.05.2022 17:43, Eugene Grosbein пишет: > Reproducing... > > So, how did you achieve your situation when you have 6 data errors and two parts of mirror > in both ONLINE and (resilvering) state? > > I suspect "6 data errors" are your problem. Well, everything was simple: 1) 12.1, disk died, I issued "attach" instead of "replace" 2) it attached, but started resilvering in cycles, finding more and more errors (which is understandable, since both disk started to die aproximately at the same time - one was eletrically dead, another was SMART-dying) 3) at some time on 12.2 I've even had this (guess with different guids, didn't figure out to look this up), but I swear, gpt/zfsroot0 was twice in the list: both as ONLINE and UNAVAIL: config:         NAME                         STATE     READ WRITE CKSUM         zfsroot                      DEGRADED     0     0     0           mirror-0                   DEGRADED     0     0     0             gpt/zfsroot1             ONLINE       0     0     0             gpt/zfsroot0             UNAVAIL      0     0     0 cannot open             gpt/zfsroot0   ONLINE       0     0     0 (resilvering) 4) then I upgraded to 13.x to see if it wilkl resolve the situation. It actually did, kind of - gpt/zfsroot changed to DISK-31P58VAASp3. So, I guess you cannot detach/remove anything if these things are present: - something is resilvering - pool isn't "cleared" with zpool clear. Eugene.