From nobody Mon Jan 8 17:27:03 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T81G42bwHz56BP1; Mon, 8 Jan 2024 17:27:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T81G427xgz4mDN; Mon, 8 Jan 2024 17:27:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704734828; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kXL8y2YDr559KJJb+uYLCbBer3qc5RPGihp5jvNGx54=; b=mO+8D1KxBrqf8iRX6S3oixqfjugTDbaf2I+uDjczRWsMBxkTyFtA6I/lh0B0YUyDWAW414 d8yZyghw4Uq82jU7cDBRHw3x3qTxaU2Z9Lg9WhaYTcxUhOpPJmDaDosMbAnugY5V+zBORK vpG8ta0AJh/SNDBQm4P+/TMP21mj1GPiFzKbVX6s0loHkad540Q3ouTVVwB+RoMNEJgVZ9 pFkmEuA5Y7uLX6ElvPPdDCtY87EFGu4mMnWyxLM+vZwpiNCFRRrq/wK5PGCz5IIwohfCia i+tXTmHZKM8X2Dbwd8UBm+1Zm+RVyrvwqhjzCBUC7ddWFfbDlPOa0Vb+fDx0GQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704734828; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kXL8y2YDr559KJJb+uYLCbBer3qc5RPGihp5jvNGx54=; b=cPJg3l+oARnmrPJT7uYgtEbA7hAvFrLf7pgDvpz8zLrJN0626/D56bq1IMYQ//w0ffRsDn u5p7S7JO9IV0ORV9gYkBd66+zQKO38cXHy0J8DHfUj7TspkeFmaYN8guX5Jph+Cljk+Mv0 Uikg+ajD/r9hK9X1c3snZt4N8aHm32fjA7c8/4HUIkLtzRXZbgiLRhSSs2hBhS0PFWWWmO DNnK1uAFSE4vC3W9PHXvVGVxQuImGHCYeX9TglW8ULIhPDOkshOL8UexBzYf3J/lBaRaJx 8bKD40c86NWG73g3bqJpDcMqYmWpN+yVpn9tA6bNysC2RFAPe3q8lc2vExtjLg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1704734828; a=rsa-sha256; cv=none; b=a9JQL7gU80AiEZbvkvOz6hXFN4Tjcv6inoGOF7739VomC3mKFcOgycYnACS0fdpx9ATUN/ R/9VREI9rtm5V2RVEW1mnw9Va5JBEcZu7LPsdhiuIqj6YudJSezmfqg4wfPlmgXlf91Zr/ JIQN+xVggaeRc76FAab8sWPZvYeROX9/126Ugpnqtgn3ip85+zADeKRTyffkViQ5kkEWJf ynYpxxIFbq+unM1Lnf/9pTHKGw5z6AqnsWEbgPtOKxcYpPc/VGiQwEktCiLPjKUpjFU6dw L5socr6paLq3RuapUbbn9r8zar/CW/Qg/jUygmIgqL14qf3f5e2COGCpvBSjYg== Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T81G40SbNz1DRc; Mon, 8 Jan 2024 17:27:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [IPV6:2001:470:923f:1:8967:d924:426d:40bd] (unknown [IPv6:2001:470:923f:1:8967:d924:426d:40bd]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 3D919362; Mon, 8 Jan 2024 20:27:05 +0300 (MSK) Message-ID: Date: Mon, 8 Jan 2024 18:27:03 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: lev@FreeBSD.org Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. Content-Language: en-US To: Larry Rosenman Cc: freebsd-fs , freebsd-stable References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> <962b242d-546f-46ce-9eb2-9bd2a10f4608@FreeBSD.org> <30315c170f7146a5e1a05e4a2eff3d1b@lerctr.org> <262428cf89227e192953e4540875bc41@lerctr.org> From: Lev Serebryakov Organization: FreeBSD In-Reply-To: <262428cf89227e192953e4540875bc41@lerctr.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 08.01.2024 16:44, Larry Rosenman wrote: >>>>> So, if this is a mirror, then ada0 blank and ada1 with good data, in theory >>>>> you should be fine. However, perhaps ZFS is finding that there's an error from >>>>> ada1 for real. Does all of ada1 read with a simple dd? >>>>   Yep, it is read with dd, I've checked it >>>> >>>>> Not sure about the losing devices you described later on. >>>>> >>>>>      > ZFS: i/o error - all block copies unavailable >>>>>      > ZFS: can't read MOS of pool zroot >>>>>      > >>>>>      > >>>>>      >   To be honest, I thinks there is something else. Because sequence of events were (sorry, too long, but I think, tht every detail matters here): >>>>> >>>>> >>>>> Yea. There's something that's failing, which zio_read is woefully under reporting for our diagnostic efforts. And/or something is >>>>> getting confused by the blank disk and/or the partially resilvered disk. >>>> >>>>   My theory, that something is confused when one disk is 512/4096 and other is 512/512. >>>> >>>>   I want to check it on VM, but can not find VM that both (1) allows CMS boot and (2) allows to configure logical and physical sector of virtual HDD. >>>> >>>>   bhyve could configure sector sizes, but doesn't support BIOS, and VBox and qemu-system can not emulate sector sizes (or I can not google proper configuration). >>> >>> When I first saw this, I wonder what ashift is set to on the pool? >>   old pool was with ashift=9, but new one is with ashift=12. > I wonder if the ashif=9 caused the issue when you added the 4Kn disk? I'm looking for ways to check this hypothesis without additional hardware. -- // Lev Serebryakov