From nobody Sun Jan 7 20:56:59 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7Tyl0Wvlz55x4S; Sun, 7 Jan 2024 20:57:03 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7Tyk6hgXz4sYn; Sun, 7 Jan 2024 20:57:02 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704661022; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GovAzm586aX1tFxsxjyg7XMYUy6unepRRsd0F5nEXa0=; b=BAWcgtWO/FFCC4OsdS4rXuaVJl2gvLa6wjC+CDRNpN2JvJnf4cQdG6pMHb0E8pLgP5gt0x 36c/qo0YJa8FYJFeF495g6t/HZpWuSAmVZpvORmzFpnDlQ605fEvLayPugN7aFqNBNMT9q GgMhRmulzOo8NJ7rdsmTUu7iuvnKcRvfSDKxs0KImBznAzeGW/ZDsdjfkgoYLTEbtgTcgJ nkKnsJMclCtJkIj7lLh35zKg0YEF8NPPuNUa3r1xMqxQMrqtbTxy1V/nOWQMm4o4paqWgc Pbkiz+k6oQiqk56scLk/WY5an6J6rUPRJz06Gxli7432CThYkqUk1i/L/bGBcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704661022; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GovAzm586aX1tFxsxjyg7XMYUy6unepRRsd0F5nEXa0=; b=Y3Q/05BzYGfeZoI2etAA1fqCTmrzBQMKdmYIXvwL21nW/Y8RZNLaS2+zkP5EmNP2Hqu4RI JNpYc7YM4NPxv7aN4979DUfO1qaGkjt+CWoxXJOg2mxUkIHkX8qs1t7gqQQiFfkFeIi6+W 0iRlK2nEJ3K8YZzvwSQH0La5WWw2JNmMNzzx7L5GDV8gxXHkXbUZfNfNdlpsHg/lxkLYed ZYq3dPLJbclqDd3zfNV5ohaunmaeC4RVG7QDQONDlnJDJ3avOvO0vwKTRAhU4JJpM7G3QX +0LKJaxbJa7MZfstUpF1J/53S0O3OCUw7BoIH5FbwZroxnhqwmh12elujrb7Iw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1704661022; a=rsa-sha256; cv=none; b=uj1taT38/WGPMkkSdf8E6afJdc/pRv+A8/wXntZYhSFop0jW0C97F702LvA3Yw6kRQsJO0 6rSlQCnmo+StxhOXwmTufg+RY7VqDPXFoBw9fUoY3iNBKX/aBeBIJCD3vFIZIJ6qSULHVK UKe+uOD/8ooxPRG7XrY+O3kB/ZjC9p0AxmZZT4UN5a5vbzlHnTGvDfYKUeYX/5L4yOxTX1 xxedGQ+vQpldQOawgXupoQSmWf6+1vBkm53RocHzivcE59jJ2hkNtAw7Mkgwmium/qeQKf DejQ+YA05nqA929v/iGs1LMZ+++psF32k3GULD95IZBTiF4EcWwt06AmLNfyRg== Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T7Tyk526NzZVZ; Sun, 7 Jan 2024 20:57:02 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.136.24] (83-84-181-95.cable.dynamic.v4.ziggo.nl [83.84.181.95]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 95589332; Sun, 7 Jan 2024 23:56:59 +0300 (MSK) Message-ID: <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> Date: Sun, 7 Jan 2024 21:56:59 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. Content-Language: en-US From: Lev Serebryakov To: Warner Losh Cc: freebsd-fs , freebsd-stable Reply-To: lev@FreeBSD.org, lev@FreeBSD.org References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> Organization: FreeBSD In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 07.01.2024 21:49, Lev Serebryakov wrote: > On 07.01.2024 19:34, Warner Losh wrote: > >> I must have missed it. What were the diagnostics? Oh, and two "nvlist inconsistency" before that vvvv > zio_read error: 5 > zio_read error: 5 > zio_read error: 5 > ZFS: i/o error - all block copies unavailable > ZFS: can't read MOS of pool zroot > > >  To be honest, I thinks there is something else. Because sequence of events were (sorry, too long, but I think, tht every detail matters here): > > (1) Update to 13.2 from 12.4. With installation of new gptzfsboot with gpart on both disks. It could place new /boot far away, but see (2) > (2) Reboot, which completed, but showed that ada0 has problems > (3) Replacement of ada0 by DC technicians, new disk is 512/4096, old disk is 512/512, pool has ashift=9 > (4) Server refuses to boot from ada1 (ada0 is empty) with diagnostics (see above) > (5) Linux rescue system, passing 2 devices to qemu with FreeBSD (because Linux shows that ZFS is on whole disk, not on partition!). > (6) Re-creation of GPT on ada0, start of resilver (with sub-optimal ashift!). > (7) Interruption of resilver with reboot, because it is painfully slow under qemu. > (8) Wipe of ada0 (at this point resilver status of pool becomes crazy) to put live FreeBSD image to boot somehow. > (9) Many tries to cancel resilver and boot from single-disk "historical" pool on ada1, no success. I've attributed it to the strange state of pool: one component, no mirrior, but "resilvering". > (10) Boot from small UFS partition (which replaces swap partition). > (11) Pool on ada1 (old, live, 512/512 disk) is still "Reslivering" without any additional components (with zero speed, of course). > (12) Prepare partitions on ada0 again, creating new pool with ashift=12, send|receive. > (13) Removing partition table on ada1 (with old pool, ashift=9, still resilvering after many-many reboots with only one device in it). And pleas note: this pool on ada1 (old, live disk) was NOT upgraded after 12-STABLE. It was old, 12-STABLE "level" pool with all new features disabled. -- // Lev Serebryakov