From nobody Sun Jan 7 20:49:24 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7Tp31ngLz55wBc; Sun, 7 Jan 2024 20:49:31 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7Tp31JTcz4qyd; Sun, 7 Jan 2024 20:49:31 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704660571; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moJLxTRm4w6sk8CN21OufYz+qGwGUupMr1SNWmDFC1Q=; b=W8P3/dEzRcF4tth0VLGB3GbZeETlEEU8uMOdUKg6Eez4HvpljD3gTHN6VfRVoW5E9YZGzg GpF3QFz7Z35hEBzTiewXyAnOTAGoUAbuVzbFQhgbYsa1X7bBB3Ze5sv+wQsKgGfB151MlU 5PKH6m5FcIVnIO/qkstjGOkiWqAQA+YTp6jVIme+Ql9zFvk+YldrMP2uVHWDI8JrMmhHhu sgYkeIoqSSpoKhMFdIQDn76edAR5EMelbMlRFXTbCF2Sobj58MfNZPH63+TrHMazYOAehP lfUS2Bd8Iu2HJuErvPEJohrGfh/LMjJdMox7+dkktf+2bxeUFu3pY8xciFoATw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704660571; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moJLxTRm4w6sk8CN21OufYz+qGwGUupMr1SNWmDFC1Q=; b=L60jcZwsNHL1kjr1H1ARnsAm3L8JcFnIUhitK7FWGAjNkt4HpKXFO5m2N0IJT44ALgzFPL /vCIdLPZEeBrzKL4s+iT/QEovIOwBQ9vAxt1h68tDObj9I+MwEeuG/6Gh/M/uDw5neBoI7 QhHib7KO0Ph27Ce/EiiBMcYVqCGcDoggaj3DQO4hU4rDa7ICx1xo7lZPguY7z4/PGWY5d1 P+rcKAXj6VyM6TbLgeBaz6hhvjTPS0sEgh4vRaGQLBXZY+J0jFTvtml8+q8Q9ozOlbY97U xyz3Q+u7skCr9TyXqxwabBBzWnK606EAmc8YU3xD5q7LVffaMqYhiMy5s8Lrwg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1704660571; a=rsa-sha256; cv=none; b=ddezyY86Sf3CjFNXmpd8gaaoPQJ1/MB8YBtEi8/o375BElPhFrD36+LBMpYHmjP/uvyxWd V0kPr2gZhhOADfTSlQxBLH168KjBmgehaDIdz68zONA2vo5B59pyMnzy1yTVPXb8EZHKJc pSvEJtQedG7vBwmg+SGuP7B4LLcgQjn/j7HPrs8+nRvzwW8s5cmrXz81IO69RgH9kQpPhh aAx0LgejwW9N/gGKTxBZ/OeeMzAgZaPYWXZWPxNwPslNyeRifZbK5Rgf7JbOC0fP0vLh7g SyexvL6x706W8SRljXas/RSMt+AlXhPh2ERb0cFXTSVLFYHOrgnRUNYgua26oQ== Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T7Tp26ksKzZ7X; Sun, 7 Jan 2024 20:49:30 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.136.24] (83-84-181-95.cable.dynamic.v4.ziggo.nl [83.84.181.95]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 419AC1A2; Sun, 7 Jan 2024 23:49:25 +0300 (MSK) Message-ID: Date: Sun, 7 Jan 2024 21:49:24 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: lev@FreeBSD.org Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. To: Warner Losh Cc: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs , freebsd-stable References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> Content-Language: en-US From: Lev Serebryakov Organization: FreeBSD In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 07.01.2024 19:34, Warner Losh wrote: > I must have missed it. What were the diagnostics? zio_read error: 5 zio_read error: 5 zio_read error: 5 ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zroot To be honest, I thinks there is something else. Because sequence of events were (sorry, too long, but I think, tht every detail matters here): (1) Update to 13.2 from 12.4. With installation of new gptzfsboot with gpart on both disks. It could place new /boot far away, but see (2) (2) Reboot, which completed, but showed that ada0 has problems (3) Replacement of ada0 by DC technicians, new disk is 512/4096, old disk is 512/512, pool has ashift=9 (4) Server refuses to boot from ada1 (ada0 is empty) with diagnostics (see above) (5) Linux rescue system, passing 2 devices to qemu with FreeBSD (because Linux shows that ZFS is on whole disk, not on partition!). (6) Re-creation of GPT on ada0, start of resilver (with sub-optimal ashift!). (7) Interruption of resilver with reboot, because it is painfully slow under qemu. (8) Wipe of ada0 (at this point resilver status of pool becomes crazy) to put live FreeBSD image to boot somehow. (9) Many tries to cancel resilver and boot from single-disk "historical" pool on ada1, no success. I've attributed it to the strange state of pool: one component, no mirrior, but "resilvering". (10) Boot from small UFS partition (which replaces swap partition). (11) Pool on ada1 (old, live, 512/512 disk) is still "Reslivering" without any additional components (with zero speed, of course). (12) Prepare partitions on ada0 again, creating new pool with ashift=12, send|receive. (13) Removing partition on ada1 (old one, ashift=9, still resilvering after many-many reboots with only one device in it). (14) Boot from fresh ada0 pool - same errors from gptzfsboot, fail, and gptzfsboot says about OLD pool (which should not be available as GPT on ada1 was wiped out!!!!) (15) Boot from UFS again. (16) Adding parition of ada1 as second component of new pool, resilvering successful. (17) Boot with gptzfsboot still fails! With brand-new ashift=12 pool! Now bootloader reports new pool name, but still fails to boot. You see, buildworld update could place /boot too far away. But there was one last successful boot between (1) and (3)! And state of pool on live disk ada1 was very strange: I can not cancel resilver no matter what I've tried till I zap GPT and start over. > If people want to continue to support BIOS booting (or rather, booting using the CSM interfaces), then somebody is going to need to step up to the plate and implement a similar option in bsdinstall, bectl, freebsd-update, etc. I can use UEFI boot without problems, but now I'm not sure, will it work for me now. -- // Lev Serebryakov