From nobody Mon Jan 8 15:44:12 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7yzL6yMrz567Nm; Mon, 8 Jan 2024 15:44:14 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7yzL59kSz4RbD; Mon, 8 Jan 2024 15:44:14 +0000 (UTC) (envelope-from ler@lerctr.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Content-Transfer-Encoding:Content-Type:Message-ID:References: In-Reply-To:Subject:Cc:To:From:Date:MIME-Version:Sender:Reply-To:Content-ID: Content-Description; bh=42LSM3esW806EpSEuqGI3fc26vgwVHNjRaPF4pY8wBI=; b=RFuu8 YglIFqFGhnDPYtAdEkzH7+LI+yur55mevDwgJ9RTrJEjz1eqr58yL/im6W7nhn+jnL2TC8txBHyoF 0BpKceEGjdbL9M8hv/MHm9zAlwp9gcbIcVzdCpZCtCrdP3VXc2LqodqVnZYKxQPBCQArEgFoSLYI9 OcFr4cKF1B8+7Gn169M92zZ+hPSzI3F741jPRzG+a3vaBj0ON+rfP07HI/Kc7d4LSInFRvvbxKmsL 4aqAhVGYP4FKKT+E4q8fg1hPvwRufc0jrPxoL87Qt4APkRU6yxlK/OpIK21/bMPJKLc469KQVUzts YC0iiMJ2GuxZ8VAT7aWl9sXwwr2Ug==; Received-SPF: pass (thebighonker.lerctr.org: domain of lerctr.org designates 192.147.25.65 as permitted sender) client-ip=192.147.25.65; envelope-from=ler@lerctr.org; helo=webmail.lerctr.org; Received: from thebighonker.lerctr.org ([192.147.25.65]:23548 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.97.1 (FreeBSD)) (envelope-from ) id 1rMrnQ-00000000MYb-1IUd; Mon, 08 Jan 2024 09:44:12 -0600 Received: from 99-190-128-217.lightspeed.austtx.sbcglobal.net ([99.190.128.217]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Mon, 08 Jan 2024 09:44:12 -0600 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Date: Mon, 08 Jan 2024 09:44:12 -0600 From: Larry Rosenman To: lev@freebsd.org Cc: freebsd-fs , freebsd-stable Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. In-Reply-To: References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> <962b242d-546f-46ce-9eb2-9bd2a10f4608@FreeBSD.org> <30315c170f7146a5e1a05e4a2eff3d1b@lerctr.org> Message-ID: <262428cf89227e192953e4540875bc41@lerctr.org> X-Sender: ler@lerctr.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4T7yzL59kSz4RbD X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:18474, ipnet:192.147.25.0/24, country:US] On 01/08/2024 5:15 am, Lev Serebryakov wrote: > On 08.01.2024 1:15, Larry Rosenman wrote: > >>>> So, if this is a mirror, then ada0 blank and ada1 with good data, in >>>> theory >>>> you should be fine. However, perhaps ZFS is finding that there's an >>>> error from >>>> ada1 for real. Does all of ada1 read with a simple dd? >>>   Yep, it is read with dd, I've checked it >>> >>>> Not sure about the losing devices you described later on. >>>> >>>>      > ZFS: i/o error - all block copies unavailable >>>>      > ZFS: can't read MOS of pool zroot >>>>      > >>>>      > >>>>      >   To be honest, I thinks there is something else. Because >>>> sequence of events were (sorry, too long, but I think, tht every >>>> detail matters here): >>>> >>>> >>>> Yea. There's something that's failing, which zio_read is woefully >>>> under reporting for our diagnostic efforts. And/or something is >>>> getting confused by the blank disk and/or the partially resilvered >>>> disk. >>> >>>   My theory, that something is confused when one disk is 512/4096 and >>> other is 512/512. >>> >>>   I want to check it on VM, but can not find VM that both (1) allows >>> CMS boot and (2) allows to configure logical and physical sector of >>> virtual HDD. >>> >>>   bhyve could configure sector sizes, but doesn't support BIOS, and >>> VBox and qemu-system can not emulate sector sizes (or I can not >>> google proper configuration). >> >> When I first saw this, I wonder what ashift is set to on the pool? > old pool was with ashift=9, but new one is with ashift=12. I wonder if the ashif=9 caused the issue when you added the 4Kn disk? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106