From nobody Sat Nov 25 03:39:13 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ScczL4W9Wz52Q1G for ; Sat, 25 Nov 2023 03:39:26 +0000 (UTC) (envelope-from freebsd-stable1@sentry.org) Received: from shadow.sentry.org (shadow.sentry.org [210.8.237.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "shadow.sentry.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ScczK3BWrz3WtS for ; Sat, 25 Nov 2023 03:39:25 +0000 (UTC) (envelope-from freebsd-stable1@sentry.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of freebsd-stable1@sentry.org designates 210.8.237.106 as permitted sender) smtp.mailfrom=freebsd-stable1@sentry.org; dmarc=none Received: from shadow.sentry.org (localhost [127.0.0.1]) by shadow.sentry.org (8.17.1/8.16.1) with ESMTP id 3AP3dDeY027203 for ; Sat, 25 Nov 2023 14:39:13 +1100 (AEDT) (envelope-from freebsd-stable1@sentry.org) To: FreeBSD-STABLE Mailing List From: Trev Subject: ZFS high CPU use after backup and panic on shutdown Message-ID: <6ccc544a-7919-57ab-5572-db67fa09ae76@sentry.org> Date: Sat, 25 Nov 2023 14:39:13 +1100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Message whitelisted by Sendmail access database, not delayed by milter-greylist-4.6.4 (shadow.sentry.org [0.0.0.0]); Sat, 25 Nov 2023 14:39:13 +1100 (AEDT) X-Spamd-Result: default: False [-0.59 / 15.00]; FORGED_MUA_MOZILLA_MAIL_MSGID_UNKNOWN(2.50)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-0.90)[-0.895]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2764, ipnet:210.8.0.0/15, country:AU]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[sentry.org]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4ScczK3BWrz3WtS X-Spamd-Bar: / I recently updated from source FreeBSD 12-STABLE to FreeBSD 13-STABLE (stable/13-221a60a42: Tue Nov 14 15:36:40 AEDT 2023). Ever since, after my ZFS backup to an external USB drive, the system continues to consume 100% of one core of the 2011 Mac mini (i7, 16G). Example backup command from my shell script: zfs send data/www@${snapshot} | bzip2 > /mnt/zfs-data-www-${snapshot}.bz2 Neither top, vmstat nor iostat give any clue as to what system process is using 100% of one core. To stop this phenomenon I have to shutdown and reboot the system which I do with "shutdown -r now". This always results in a kernel panic: Waiting (max 60 seconds) for system process `vnlru' to stop... done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining... 4 6 4 2 1 2 1 0 0 0 done All buffers synced. Uptime: 1d23h34m0s Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0x440 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80584d3c stack pointer = 0x28:0xfffffe00c7352d80 frame pointer = 0x28:0xfffffe00c7352df0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (arc_prune) trap number = 12 panic: page fault cpuid = 3 time = 1700828648 KDB: stack backtrace: #0 0xffffffff805bf675 at kdb_backtrace+0x65 #1 0xffffffff8057b0d2 at vpanic+0x152 #2 0xffffffff8057af73 at panic+0x43 #3 0xffffffff80845519 at trap_fatal+0x389 #4 0xffffffff8084556f at trap_pfault+0x4f #5 0xffffffff8081f30e at calltrap+0x8 #6 0xffffffff81a7274a at arc_prune_task+0x7a #7 0xffffffff81a2865f at taskq_run+0x1f #8 0xffffffff805d2e52 at taskqueue_run_locked+0x162 #9 0xffffffff805d3d72 at taskqueue_thread_loop+0xb2 #10 0xffffffff8053ed81 at fork_exit+0x71 #11 0xffffffff8082038e at fork_trampoline+0xe Uptime: 1d23h34m0s Dumping 1306 out of 16346 MB:..2%..12%..21%..31%..41%..51%..61%..72%..81%..91% The current process is always "arc_prune". Where to go from here to resolve this? From nobody Mon Nov 27 01:10:47 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SdnbH4l7Qz52Svj for ; Mon, 27 Nov 2023 01:11:07 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: from clamta10.bpe.bigpond.com (clamta10.bpe.bigpond.com [203.42.22.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SdnbD64S4z4Hm5 for ; Mon, 27 Nov 2023 01:11:04 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bigpond.net.au header.s=202303 header.b="lY+37Aj "; spf=pass (mx1.freebsd.org: domain of areilly@bigpond.net.au designates 203.42.22.26 as permitted sender) smtp.mailfrom=areilly@bigpond.net.au; dmarc=pass (policy=reject) header.from=bigpond.net.au DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bigpond.net.au; s=202303; h=To:Date:Message-Id:Subject:Mime-Version: Content-Type:From; bh=u3cGnVLKY8Zpzszur8LkugUO1rNpFr6GtutemYLYTYU=; b=lY+37Aj hVP9tdWoZRrXmUeUNqxyiXp359ymkLvqz9F32OQ3Iqt3k2x+W0r5DhvRkmBwZp8buKxEWPhnCv5MZ YW0Z4WEZHd5ZdoxludeGWBZxIbB4oE4JXZzooVJKwLRrg0lBhP6HfoeL4Jlrg21E5kBObG1V9f3ph 0GamfCUjysorohUnsgy6HoIkw2iDUxFYNP12XBTeNtF1f94z7ev/EijZeqKN7rK5EuQz8Lm+nbf9D nn1REt0U9jx7lbzPSaHIYJRLg0Eoic/Jdz2RtVRuqJeQ/6e7hk2WYyHutVA15ZDvB44EhVyMFIjg5 q+3pDFmARhXSKo2HaXlFNeTeQzg==; Received: from claprdcmr03 by claprdomr10 with esmtp (envelope-from ) id 1r7Q9L-000EaF-1j for freebsd-stable@freebsd.org; Mon, 27 Nov 2023 12:10:59 +1100 Received: from [121.223.155.16] (helo=smtpclient.apple) by claprdcmr03 with esmtpa (envelope-from ) id 1r7Q9L-0004FR-0m for freebsd-stable@freebsd.org; Mon, 27 Nov 2023 12:10:59 +1100 From: Andrew Reilly Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: An amusing comedy of errors: my FreeBSD-14 upgrade Message-Id: <8C86654F-C686-4878-8764-A5C4E90B4264@bigpond.net.au> Date: Mon, 27 Nov 2023 12:10:47 +1100 To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.3774.200.91.1.1) X-tce-id: areilly@bigpond.net.au X-tce-ares-id: e{0afa20f9-23d3-4b05-bdf2-49f3e3631a68}1 X-tce-spam-action: no action X-tce-spam-score: 0.0 X-Cm-Analysis: v=2.4 cv=IO/ESCjG c=1 sm=1 tr=0 ts=6563eca3 a=g7JjhvAvvPZ9ycLFhywHJA==:117 a=g7JjhvAvvPZ9ycLFhywHJA==:17 a=kj9zAlcOel0A:10 a=BNY50KLci1gA:10 a=pvAka7dGAAAA:8 a=YvgnshHwSUfJoFZQJ_AA:9 a=CjuIK1q_8ugA:10 a=2JfvvIpjdqwvDtMeNY0X:22 X-Cm-Envelope: MS4xfARo6o7onwm+WjHBVQ94l9eDuiPkblepbJDcNSumHqzpSiGiAFWEmsywzKftq+qlvU/S7rRgwtsejdDlXQw/tgNYeAz7hsh8T6kB4Q3LLmJ2Sx9Dp/aM kiQayWsbUsnFxsFzqusJ/7YMMst0nMzUZLb0ZLuGPJr4ahqb48DgifUWsmv2wK4XGAczwB8XuN1KEA== X-Spamd-Result: default: False [-3.22 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.95)[-0.946]; NEURAL_HAM_MEDIUM(-0.68)[-0.677]; DMARC_POLICY_ALLOW(-0.50)[bigpond.net.au,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[bigpond.net.au:s=202303]; R_SPF_ALLOW(-0.20)[+ip4:203.42.22.0/25]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[203.42.22.26:from]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1221, ipnet:203.40.0.0/13, country:AU]; FREEMAIL_ENVFROM(0.00)[bigpond.net.au]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[bigpond.net.au:+]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[bigpond.net.au]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4SdnbD64S4z4Hm5 X-Spamd-Bar: --- [TL/DR: -14 removed pam_opie.so , so leaving those = lines in pam.d/system etc prevented su/sudo;=20 zpool upgrade root drive without updating gptzfsboot boot loader = prevented rebooting.] Thought that I'd write this down, in case it helps anyone. The upgrade = from stable-13 to stable-14 that I made to my file server over the = weekend was one of the bumpier upgrades in my thirty year history with = FreeBSD (yes, I was there at the transition from the patchkit). I'm = happy to relate that all of the pain related here was self-inflicted, = and FreeBSD itself shone through with its delightful robustness and = straightforward nature. All ends well. My file server, for now, is a miniITX AMD Zen 1700 system with four 8T = spinning rust drives in a single RaidZ Zpool, about 250G of NVME M.2 = flash holding root, usr, var and swap (boot to ZFS) and a USB connected = backup system also running ZFS. I track -stable and update ports (with = portmaster), user-space and kernel weekly. The start was very simple, now that FreeBSD is on git: I just git = switch'ed to the stable/14 branch, which went without a hitch. Then I ran my usual weekly rebuild script, which got to the end without = fuss, but with the usual complaint from etcupdate that there were some = unresolved issues. I should have been paying more attention to that: I = was not using etcupdate correctly, and had not been since switching over = from mergemaster a year or so ago. Needless to say, misconfiguration = was the start of the troubles, and they kicked in immediately: I = couldn't reboot after the upgrade. I couldn't reboot or sudo or su = because my /etc/pam.d configuration still referred to pam_opie.so, = because I had not noticed that being removed. I _could_ still ssh into = the system because my ssh config had disabled pam. Didn't help though, = because I was still stuck as me, and couldn't edit the config files, = because of sudo (I've since rebuilt sudo to not use pam either!) Easy enough to fix, right? Power down and reboot into single-user mode = and go from there. Unfortunately I had ripped the graphics card out of = the system some long time ago as an attempt to keep a bit of heat out of = the box and had apparently lost it in a couple of intervening house = moves. Perhaps I'd donated it to the electronics recycling mob along = with a box of old cables and power supplies. Too late to go and get one = Saturday afternoon, I found a store some distance away that would sell = me one on Sunday morning. With new graphics card in hand, I powered = down, took the lid off the server, carefully lifted out the hard-drive = cage and installed the GPU. Plugged in monitor and keyboard and powered = up. Single user mode did the job: edited the pam.d files and was just about = to reboot when I checked zpool status to see why the boot messages had = said something about my main array operating in "degraded" mode. One = drive was apparently not found/attached. On closer inspection I = discovered that I'd dislocated the power supply plug when I took the = cage out. I'd fix that when I did the next power-down. But in the mean = time, zpool status had also taunted me with new features that I could = enable. So I did, on the root drive. And power-cycled. And stared = dumbly at the boot screen telling me that it couldn't find any bootable = drives, because the one that was there had an incompatible zpool = version. Aargh! The boot loader had not been updated! Couldn't even = get to single user mode to fix it. I downloaded the 14-release bootonly image from the FreeBSD web site and = found a suitable thumb drive to put it on. Power cycled the box again = and told the boot menu to boot from the thumb drive. There followed a = great deal of gpart footling while I tried to remember just how I had = the drives arranged, but in the end I found the magic incantation (gpart = bootcode -p /boot/gptzfsboot -i 1 nda0) to install the new version of = the boot loader. Rebooted again, this time to the main system, rather = than the thumb drive, and that worked. ZFS resilvered the previously = missing drive quicker than I could notice, and subsequent scrubs found = nothing in need of fixing. Everything is now hunky-dory. Thanks to the always-wonderful FreeBSD team for continuing to produce a = system that can be understood at sufficient detail to fairly easily dig = oneself out of what might otherwise be catastrophic misadventures! From nobody Mon Nov 27 13:09:19 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sf5XD09nRz52WQx for ; Mon, 27 Nov 2023 13:09:32 +0000 (UTC) (envelope-from freebsd-stable1@sentry.org) Received: from shadow.sentry.org (shadow.sentry.org [210.8.237.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "shadow.sentry.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sf5XB6snNz3Bvn for ; Mon, 27 Nov 2023 13:09:30 +0000 (UTC) (envelope-from freebsd-stable1@sentry.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of freebsd-stable1@sentry.org designates 210.8.237.106 as permitted sender) smtp.mailfrom=freebsd-stable1@sentry.org; dmarc=none Received: from shadow.sentry.org (localhost [127.0.0.1]) by shadow.sentry.org (8.17.1/8.16.1) with ESMTP id 3ARD9JUF074671 for ; Tue, 28 Nov 2023 00:09:19 +1100 (AEDT) (envelope-from freebsd-stable1@sentry.org) Subject: Re: ZFS high CPU use after backup and panic on shutdown From: Trev To: FreeBSD-STABLE Mailing List References: <6ccc544a-7919-57ab-5572-db67fa09ae76@sentry.org> Message-ID: <67606352-8546-7016-5e65-0f5c4dec8a19@sentry.org> Date: Tue, 28 Nov 2023 00:09:19 +1100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: <6ccc544a-7919-57ab-5572-db67fa09ae76@sentry.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Message whitelisted by Sendmail access database, not delayed by milter-greylist-4.6.4 (shadow.sentry.org [0.0.0.0]); Tue, 28 Nov 2023 00:09:20 +1100 (AEDT) X-Spamd-Result: default: False [-0.70 / 15.00]; FORGED_MUA_MOZILLA_MAIL_MSGID_UNKNOWN(2.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:2764, ipnet:210.8.0.0/15, country:AU]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[sentry.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1] X-Rspamd-Queue-Id: 4Sf5XB6snNz3Bvn X-Spamd-Bar: / Trev wrote on 25/11/23 2:39 pm: > I recently updated from source FreeBSD 12-STABLE to FreeBSD 13-STABLE > (stable/13-221a60a42: Tue Nov 14 15:36:40 AEDT 2023). > > Ever since, after my ZFS backup to an external USB drive, the system > continues to consume 100% of one core of the 2011 Mac mini (i7, 16G). > > Example backup command from my shell script: > > zfs send data/www@${snapshot} | bzip2 > /mnt/zfs-data-www-${snapshot}.bz2 > > Neither top, vmstat nor iostat give any clue as to what system process > is using 100% of one core. To stop this phenomenon I have to shutdown > and reboot the system which I do with "shutdown -r now". > > This always results in a kernel panic: [CHOMP] Resolved by updating the source to stable/13-ecb4d2c6e: Sat Nov 25 11:31:12 AEDT 2023 and rebuilding. [Also, thanks to Mike Karels for off-list help with top.] From nobody Mon Nov 27 15:50:39 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sf96P3bk9z5349J for ; Mon, 27 Nov 2023 15:50:53 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sf96N65Mjz4VfS for ; Mon, 27 Nov 2023 15:50:52 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com; dmarc=none Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-50aab20e828so6256940e87.2 for ; Mon, 27 Nov 2023 07:50:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701100251; x=1701705051; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=pqSDyDuOHhNmmhW521XSOLAKDAHSrUmpg0dnU86RS0g=; b=XArQ/rwcx1rXiuHOzOdZW9dNnrFILZzisGLSLX0LCDRVlnHVdHjSom58jpPImS+tyZ EQsTLRmxWgAqiW/OmVqFnxjbjQD7Mw9bCLj/3GZ4S286+CewPincAQhICbHZaZYniPe5 QaMmBCx9K8eYgvZSm8Fr+x4pajT3F9RBoA2PkypBFglJ/1llMFY0NPex6h/1bhu2VqYE 8+YuiS8fYOKMaF5kWuLSsY0IOFOLdzNyLHeSrFWT+VjSNd/hjtUB8rDe8+FtWdP+8gC4 ncc0grqEGLfZOKyZrIQA/TtFal0dwKQTYPGt0+6s9mrKMhikhBpB/kzzrbG1IP3A13rq X15w== X-Gm-Message-State: AOJu0Yz8wfDRJMzMn43ceX+tBQpymPOUiInPl8m/z75GdVmrNZLXLmIL +MhIU+9T3sAcIQplcPURsFCIky1OGDM28K8Z71Tzzt0NC0s= X-Google-Smtp-Source: AGHT+IFn6ZhrvZ8YOVjH2IUfqaO1z99FzoeS3L9xGAJe5CqBk+Um+vqbA0mgsu/QR0Y75i85fyIF56O1/gNkjEAgZrw= X-Received: by 2002:a05:6512:31c4:b0:50b:acab:5452 with SMTP id j4-20020a05651231c400b0050bacab5452mr4657978lfe.41.1701100250661; Mon, 27 Nov 2023 07:50:50 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 From: Ed Maste Date: Mon, 27 Nov 2023 10:50:39 -0500 Message-ID: Subject: Potential ZFS data corruption issue To: freebsd-stable stable Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-2.62 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.995]; NEURAL_HAM_MEDIUM(-0.63)[-0.626]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.42:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.42:from]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[carpeddiem]; ARC_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Rspamd-Queue-Id: 4Sf96N65Mjz4VfS X-Spamd-Bar: -- Dear FreeBSD community, We want to bring your attention to a potential data corruption issue affecting multiple versions of OpenZFS. It was initially reported against OpenZFS 2.2.0 but also affects earlier and later versions. This issue can be reproduced with targeted effort but it has not been observed frequently in real-world scenarios. This issue is not related to block cloning, although it is possible that enabling block cloning increases the probability of encountering the issue. It is unclear if the issue is reproducible on the version of ZFS in FreeBSD 12.4. A short term workaround is available for FreeBSD 14.0 and 13.2 by setting the vfs.zfs.dmu_offset_next_sync sysctl to 0: # echo vfs.zfs.dmu_offset_next_sync=0 >> /etc/sysctl.conf # sysctl vfs.zfs.dmu_offset_next_sync=0 The workaround may result in inaccurate reporting of file holes in sparse files, reintroducing OpenZFS issue 6958[1]. See the ZFS(4) man page for more information on this setting. The workaround does not deterministically prevent the issue but does drastically reduce the likelihood of encountering it. For more information, see OpenZFS issue 15526[2] and OpenZFS pull request 15571[3]. FreeBSD bug report PR 275308[4] is open to track the FreeBSD erratum update which will bring in the fix, after it is committed to OpenZFS. Please check the FreeBSD PR regularly if you are looking for details on the timeline of the FreeBSD erratum update. We want to assure the FreeBSD community that we are actively monitoring the situation. Thank you for your understanding and continued support. [1] https://github.com/openzfs/zfs/issues/6958 [2] https://github.com/openzfs/zfs/issues/15526 [3] https://github.com/openzfs/zfs/pull/15571 [4] https://bugs.freebsd.org/275308 From nobody Tue Nov 28 19:51:09 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SftPM3pjDz52SXs for ; Tue, 28 Nov 2023 19:51:19 +0000 (UTC) (envelope-from SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SftPL2tLGz4Xj5 for ; Tue, 28 Nov 2023 19:51:18 +0000 (UTC) (envelope-from SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of "SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz"; dmarc=none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id AA0AAD788C for ; Tue, 28 Nov 2023 20:51:10 +0100 (CET) Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id AA40FD788B for ; Tue, 28 Nov 2023 20:51:09 +0100 (CET) Message-ID: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> Date: Tue, 28 Nov 2023 20:51:09 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: FreeBSD-STABLE Mailing List Content-Language: cs-Cestina From: Miroslav Lachman <000.fbsd@quip.cz> Subject: Missing disk partition devices and GPT lables in /dev/ Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [1.54 / 15.00]; AUTH_NA(1.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz]; NEURAL_SPAM_SHORT(0.28)[0.284]; NEURAL_SPAM_MEDIUM(0.21)[0.213]; NEURAL_HAM_LONG(-0.17)[-0.170]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; R_DKIM_NA(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_NA(0.00)[quip.cz]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SftPL2tLGz4Xj5 X-Spamd-Bar: + I have strange problem on one of my machines - the disk devices in /dev/ are only shown as ada0 and ada1 but no partition devices, no devices in /dev/gpt/ or /dev/gptid/, these directories are completely missing even though there are GPT labels. This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 Seagate NAS 4TB, manually partitioned with gpart utility, zpool created and then all the date migrated by zfs send and zfs receive from old 2TB disks. Everything works, system is running but device nodes in /dev/ are missing. I noticed this only when I want to create gmirrored swap from ada0p4 and ada1p4 but got "No such file or directory." There are only diskid style devices in /dev/diskid/ gpart show -l shows correct GPT labels for partitions # gpart show -l => 40 7814037088 diskid/DISK-Z3010G8V GPT (3.6T) 40 1024 1 bootG8V (512K) 1064 40960 2 efiG8V (20M) 42024 83886080 3 diskG8Vsys (40G) 83928104 20971520 4 swapG8V (10G) 104899624 7707033600 5 diskG8Vtank0 (3.6T) 7811933224 2103904 - free - (1.0G) => 40 7814037088 diskid/DISK-Z302YC90 GPT (3.6T) 40 1024 1 bootC90 (512K) 1064 40960 2 efiC90 (20M) 42024 83886080 3 diskC90sys (40G) 83928104 20971520 4 swapC90 (10G) 104899624 7707033600 5 diskC90tank0 (3.6T) 7811933224 2103904 - free - (1.0G) But gpart show -p does not show ada0p1 ada0p2 ..ada0pN, it shows diskid style entries # gpart show -p => 40 7814037088 diskid/DISK-Z3010G8V GPT (3.6T) 40 1024 diskid/DISK-Z3010G8Vp1 freebsd-boot (512K) 1064 40960 diskid/DISK-Z3010G8Vp2 efi (20M) 42024 83886080 diskid/DISK-Z3010G8Vp3 freebsd-zfs (40G) 83928104 20971520 diskid/DISK-Z3010G8Vp4 freebsd-swap (10G) 104899624 7707033600 diskid/DISK-Z3010G8Vp5 freebsd-zfs (3.6T) 7811933224 2103904 - free - (1.0G) => 40 7814037088 diskid/DISK-Z302YC90 GPT (3.6T) 40 1024 diskid/DISK-Z302YC90p1 freebsd-boot (512K) 1064 40960 diskid/DISK-Z302YC90p2 efi (20M) 42024 83886080 diskid/DISK-Z302YC90p3 freebsd-zfs (40G) 83928104 20971520 diskid/DISK-Z302YC90p4 freebsd-swap (10G) 104899624 7707033600 diskid/DISK-Z302YC90p5 freebsd-zfs (3.6T) 7811933224 2103904 - free - (1.0G) # gpart list | grep label label: bootG8V label: efiG8V label: diskG8Vsys label: swapG8V label: diskG8Vtank0 label: bootC90 label: efiC90 label: diskC90sys label: swapC90 label: diskC90tank0 # ll /dev/gpt/ ls: /dev/gpt/: No such file or directory # ll /dev/ada* crw-r----- 1 root operator 0x73 Nov 27 00:00 /dev/ada0 crw-r----- 1 root operator 0x81 Nov 27 00:00 /dev/ada1 Sysctl seems normal to me. # sysctl kern.geom.label kern.geom.label.disk_ident.enable: 1 kern.geom.label.gptid.enable: 1 kern.geom.label.gpt.enable: 1 kern.geom.label.ufs.enable: 1 kern.geom.label.ufsid.enable: 1 kern.geom.label.reiserfs.enable: 1 kern.geom.label.ntfs.enable: 1 kern.geom.label.msdosfs.enable: 1 kern.geom.label.iso9660.enable: 1 kern.geom.label.flashmap.enable: 1 kern.geom.label.ext2fs.enable: 1 kern.geom.label.debug: 0 This is the only way I can access disk partitions # ll /dev/diskid/ total 0 crw-r----- 1 root operator 0x7f Nov 27 00:00 DISK-Z3010G8V crw-r----- 1 root operator 0x99 Nov 27 00:00 DISK-Z3010G8Vp1 crw-r----- 1 root operator 0x9b Nov 27 00:00 DISK-Z3010G8Vp2 crw-r----- 1 root operator 0x9d Nov 27 00:00 DISK-Z3010G8Vp3 crw-r----- 1 root operator 0x9f Nov 27 00:00 DISK-Z3010G8Vp4 crw-r----- 1 root operator 0xa1 Nov 27 00:00 DISK-Z3010G8Vp5 crw-r----- 1 root operator 0xad Nov 27 00:00 DISK-Z302YC90 crw-r----- 1 root operator 0xc5 Nov 27 00:00 DISK-Z302YC90p1 crw-r----- 1 root operator 0xc7 Nov 27 00:00 DISK-Z302YC90p2 crw-r----- 1 root operator 0xc9 Nov 27 00:00 DISK-Z302YC90p3 crw-r----- 1 root operator 0xcb Nov 27 00:00 DISK-Z302YC90p4 crw-r----- 1 root operator 0xcd Nov 27 00:00 DISK-Z302YC90p5 I created two pools (sys and tank1) with adaN devices when there were 4 disks and the system was booted from USB flash drive with FreeBSD 13.2; current ada0 was ada2 and current ada1 was ada3. These commands were used # zpool create -m none sys mirror ada2p3 ada3p3 # zpool create -m none tank1 mirror ada2p5 ada3p5 But now, when the system is booted of these disks, they are reported only by diskids. # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT sys 39.5G 15.9G 23.6G - - 12% 40% 1.00x ONLINE - mirror-0 39.5G 15.9G 23.6G - - 12% 40.2% - ONLINE diskid/DISK-Z3010G8Vp3 40G - - - - - - - ONLINE diskid/DISK-Z302YC90p3 40G - - - - - - - ONLINE tank1 3.58T 1.22T 2.36T - - 0% 34% 1.00x ONLINE - mirror-0 3.58T 1.22T 2.36T - - 0% 34.0% - ONLINE diskid/DISK-Z3010G8Vp5 3.59T - - - - - - - ONLINE diskid/DISK-Z302YC90p5 3.59T - - - - - - - ONLINE I did similar setup and migration many times and never seen this problem. What can be wrong with this system? Kind regards Miroslav Lachman From nobody Tue Nov 28 19:55:30 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SftVV2Fk5z52SjJ for ; Tue, 28 Nov 2023 19:55:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SftVV0YLXz4YxD for ; Tue, 28 Nov 2023 19:55:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a03a900956dso27858466b.1 for ; Tue, 28 Nov 2023 11:55:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701201342; x=1701806142; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/BtFeTNvmDJ/yRUrrr6GSWHxzprurqsmsCHS8r/JK14=; b=C+SdPd6ceJiO8lSulQjmrZv1hTUaXLb8kW/NTKO8QEqOyEoO+AU6ReH+Br8LWt+b93 RzC4Cs042UiANM41xkbtwosfsZqs94qxKeXwZ89MozI0gfJOzDqVxABq1f9j4XpF8ER+ vxbj+LxJq0LJ+H/0OA6I+4QeiKEgbtegmOJtDF4cVQBp/7gbtQqZ7AmWjKzacNRS9yo5 7yRjAxc3k4WtolZa0K8696c6t+yUFPBYOUgMKIV3PyYUqIhUTUiDvDHk7HfOX/OSG9An /aXBrd4RjrxH93wbpHrZG3kW3dPJqpJ44jMAWKNDYLATjDckoqoTB09UDcmmFXrXefzF FaVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701201342; x=1701806142; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/BtFeTNvmDJ/yRUrrr6GSWHxzprurqsmsCHS8r/JK14=; b=WFDLIqMgDC94fYOnd04zgRcESy0fR48Jf+tXoywKvwo2gvVz+bpl1Yb/CRQqEAPzAv w1MgB9SWVxgST4CrwubTJGMV9s6lqEhAXVfuqbh9s/GEIv6gK8E0rJvt/BJvV/X0LF3d n9U2W8uvngPtN8RQj5PPO6wNcHFispR5gP/xDX/59wJIiMTVBb+yhRicuHQGFu6Uez0d Bf/Cw53tIdAjXoh0jhv/3PNXv+igmtNBGbVvDphyFHiNBtIxUvgUCw5dXtEvaZUFdRsx TYub/J4dOkE09SkNSfgET/Go89yssyqlHUu05Bge1XJA74WLjAFOcAJIgnp4ek5oKGo4 /R3A== X-Gm-Message-State: AOJu0Yy+MiNhEYJT4Wko33U4B94qrj3/1HGWERXGtqiS5NXy+ad1KvtH sXwLaRBNPiGQua52Q5UXi9HkuBh5P5bTBWdCAntakQ== X-Google-Smtp-Source: AGHT+IFm4yNCHEFMOkSKE2p6iGXWvl2mdIc9uKtubs6EpqCADp888Q/eTmbM2yTkPjfACY1+sngnBaR/ramJeYmWCp8= X-Received: by 2002:a17:907:765b:b0:9e3:b158:6fb8 with SMTP id kj27-20020a170907765b00b009e3b1586fb8mr15214082ejc.32.1701201341704; Tue, 28 Nov 2023 11:55:41 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> In-Reply-To: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> From: Warner Losh Date: Tue, 28 Nov 2023 12:55:30 -0700 Message-ID: Subject: Re: Missing disk partition devices and GPT lables in /dev/ To: Miroslav Lachman <000.fbsd@quip.cz> Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000a9fd16060b3bcd43" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SftVV0YLXz4YxD --000000000000a9fd16060b3bcd43 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 28, 2023 at 12:51=E2=80=AFPM Miroslav Lachman <000.fbsd@quip.cz= > wrote: > I have strange problem on one of my machines - the disk devices in /dev/ > are only shown as ada0 and ada1 but no partition devices, no devices in > /dev/gpt/ or /dev/gptid/, these directories are completely missing even > though there are GPT labels. > This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 > Seagate NAS 4TB, manually partitioned with gpart utility, zpool created > and then all the date migrated by zfs send and zfs receive from old 2TB > disks. Everything works, system is running but device nodes in /dev/ are > missing. I noticed this only when I want to create gmirrored swap from > ada0p4 and ada1p4 but got "No such file or directory." > > There are only diskid style devices in /dev/diskid/ > > gpart show -l shows correct GPT labels for partitions > > # gpart show -l > =3D> 40 7814037088 diskid/DISK-Z3010G8V GPT (3.6T) > 40 1024 1 bootG8V (512K) > 1064 40960 2 efiG8V (20M) > 42024 83886080 3 diskG8Vsys (40G) > 83928104 20971520 4 swapG8V (10G) > 104899624 7707033600 5 diskG8Vtank0 (3.6T) > 7811933224 2103904 - free - (1.0G) > > =3D> 40 7814037088 diskid/DISK-Z302YC90 GPT (3.6T) > 40 1024 1 bootC90 (512K) > 1064 40960 2 efiC90 (20M) > 42024 83886080 3 diskC90sys (40G) > 83928104 20971520 4 swapC90 (10G) > 104899624 7707033600 5 diskC90tank0 (3.6T) > 7811933224 2103904 - free - (1.0G) > > > But gpart show -p does not show ada0p1 ada0p2 ..ada0pN, it shows diskid > style entries > > # gpart show -p > =3D> 40 7814037088 diskid/DISK-Z3010G8V GPT (3.6T) > 40 1024 diskid/DISK-Z3010G8Vp1 freebsd-boot (512K) > 1064 40960 diskid/DISK-Z3010G8Vp2 efi (20M) > 42024 83886080 diskid/DISK-Z3010G8Vp3 freebsd-zfs (40G) > 83928104 20971520 diskid/DISK-Z3010G8Vp4 freebsd-swap (10G) > 104899624 7707033600 diskid/DISK-Z3010G8Vp5 freebsd-zfs (3.6T) > 7811933224 2103904 - free - (1.0G) > > =3D> 40 7814037088 diskid/DISK-Z302YC90 GPT (3.6T) > 40 1024 diskid/DISK-Z302YC90p1 freebsd-boot (512K) > 1064 40960 diskid/DISK-Z302YC90p2 efi (20M) > 42024 83886080 diskid/DISK-Z302YC90p3 freebsd-zfs (40G) > 83928104 20971520 diskid/DISK-Z302YC90p4 freebsd-swap (10G) > 104899624 7707033600 diskid/DISK-Z302YC90p5 freebsd-zfs (3.6T) > 7811933224 2103904 - free - (1.0G) > > > # gpart list | grep label > label: bootG8V > label: efiG8V > label: diskG8Vsys > label: swapG8V > label: diskG8Vtank0 > label: bootC90 > label: efiC90 > label: diskC90sys > label: swapC90 > label: diskC90tank0 > > > # ll /dev/gpt/ > ls: /dev/gpt/: No such file or directory > > > # ll /dev/ada* > crw-r----- 1 root operator 0x73 Nov 27 00:00 /dev/ada0 > crw-r----- 1 root operator 0x81 Nov 27 00:00 /dev/ada1 > > > Sysctl seems normal to me. > > # sysctl kern.geom.label > kern.geom.label.disk_ident.enable: 1 > kern.geom.label.gptid.enable: 1 > kern.geom.label.gpt.enable: 1 > kern.geom.label.ufs.enable: 1 > kern.geom.label.ufsid.enable: 1 > kern.geom.label.reiserfs.enable: 1 > kern.geom.label.ntfs.enable: 1 > kern.geom.label.msdosfs.enable: 1 > kern.geom.label.iso9660.enable: 1 > kern.geom.label.flashmap.enable: 1 > kern.geom.label.ext2fs.enable: 1 > kern.geom.label.debug: 0 > > > This is the only way I can access disk partitions > > # ll /dev/diskid/ > total 0 > crw-r----- 1 root operator 0x7f Nov 27 00:00 DISK-Z3010G8V > crw-r----- 1 root operator 0x99 Nov 27 00:00 DISK-Z3010G8Vp1 > crw-r----- 1 root operator 0x9b Nov 27 00:00 DISK-Z3010G8Vp2 > crw-r----- 1 root operator 0x9d Nov 27 00:00 DISK-Z3010G8Vp3 > crw-r----- 1 root operator 0x9f Nov 27 00:00 DISK-Z3010G8Vp4 > crw-r----- 1 root operator 0xa1 Nov 27 00:00 DISK-Z3010G8Vp5 > crw-r----- 1 root operator 0xad Nov 27 00:00 DISK-Z302YC90 > crw-r----- 1 root operator 0xc5 Nov 27 00:00 DISK-Z302YC90p1 > crw-r----- 1 root operator 0xc7 Nov 27 00:00 DISK-Z302YC90p2 > crw-r----- 1 root operator 0xc9 Nov 27 00:00 DISK-Z302YC90p3 > crw-r----- 1 root operator 0xcb Nov 27 00:00 DISK-Z302YC90p4 > crw-r----- 1 root operator 0xcd Nov 27 00:00 DISK-Z302YC90p5 > > > I created two pools (sys and tank1) with adaN devices when there were 4 > disks and the system was booted from USB flash drive with FreeBSD 13.2; > current ada0 was ada2 and current ada1 was ada3. > > These commands were used > > # zpool create -m none sys mirror ada2p3 ada3p3 > > # zpool create -m none tank1 mirror ada2p5 ada3p5 > > But now, when the system is booted of these disks, they are reported > only by diskids. > > # zpool list -v > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ > FRAG CAP DEDUP HEALTH ALTROOT > sys 39.5G 15.9G 23.6G - - > 12% 40% 1.00x ONLINE - > mirror-0 39.5G 15.9G 23.6G - - > 12% 40.2% - ONLINE > diskid/DISK-Z3010G8Vp3 40G - - - - > - - - ONLINE > diskid/DISK-Z302YC90p3 40G - - - - > - - - ONLINE > tank1 3.58T 1.22T 2.36T - - > 0% 34% 1.00x ONLINE - > mirror-0 3.58T 1.22T 2.36T - - > 0% 34.0% - ONLINE > diskid/DISK-Z3010G8Vp5 3.59T - - - - > - - - ONLINE > diskid/DISK-Z302YC90p5 3.59T - - - - > - - - ONLINE > > > I did similar setup and migration many times and never seen this problem. > > What can be wrong with this system? > The problem is a geom design point. When you have multiple aliases that you can open, the others are removed from the filesystem view when one is opened. It looks like from the above that the mirror-0 was created with the diskid, so that's the device that zfs opens, leaving the ada2 devices removed. Warner --000000000000a9fd16060b3bcd43 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Nov 28, 2023 at 12:51=E2=80= =AFPM Miroslav Lachman <000.fbsd@qui= p.cz> wrote:
I have strange problem on one of my machines - the disk devices in /dev= /
are only shown as ada0 and ada1 but no partition devices, no devices in /dev/gpt/ or /dev/gptid/, these directories are completely missing even though there are GPT labels.
This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 Seagate NAS 4TB, manually partitioned with gpart utility, zpool created and then all the date migrated by zfs send and zfs receive from old 2TB disks. Everything works, system is running but device nodes in /dev/ are missing. I noticed this only when I want to create gmirrored swap from
ada0p4 and ada1p4 but got "No such file or directory."

There are only diskid style devices in /dev/diskid/

gpart show -l shows correct GPT labels for partitions

# gpart show -l
=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 diskid/DISK-Z3= 010G8V=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A01=C2=A0 bootG8V=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02= =C2=A0 efiG8V=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 diskG8Vsys= =C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04=C2=A0 swapG8V=C2=A0 (= 10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05=C2=A0 diskG8Vtank0=C2=A0 (3.6T)<= br> =C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -=C2=A0 = (1.0G)

=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 diskid/DISK-Z3= 02YC90=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A01=C2=A0 bootC90=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02= =C2=A0 efiC90=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 diskC90sys= =C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04=C2=A0 swapC90=C2=A0 (= 10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05=C2=A0 diskC90tank0=C2=A0 (3.6T)<= br> =C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -=C2=A0 = (1.0G)


But gpart show -p does not show ada0p1 ada0p2 ..ada0pN, it shows diskid style entries

# gpart show -p
=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 =C2=A0 diskid/= DISK-Z3010G8V=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 diskid/DISK-Z3010G8Vp1=C2=A0 freebsd-boot=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 diskid/DISK-Z3010G8Vp2=C2=A0 efi=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 diskid/DISK-Z= 3010G8Vp3=C2=A0 freebsd-zfs=C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 diskid/DISK-Z3010G= 8Vp4=C2=A0 freebsd-swap=C2=A0 (10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 diskid/DISK-Z3010G8Vp5=C2=A0= freebsd-zfs=C2=A0 (3.6T)
=C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -= =C2=A0 (1.0G)

=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 =C2=A0 diskid/= DISK-Z302YC90=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 diskid/DISK-Z302YC90p1=C2=A0 freebsd-boot=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 diskid/DISK-Z302YC90p2=C2=A0 efi=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 diskid/DISK-Z= 302YC90p3=C2=A0 freebsd-zfs=C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 diskid/DISK-Z302YC= 90p4=C2=A0 freebsd-swap=C2=A0 (10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 diskid/DISK-Z302YC90p5=C2=A0= freebsd-zfs=C2=A0 (3.6T)
=C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -= =C2=A0 (1.0G)


# gpart list | grep label
=C2=A0 =C2=A0 label: bootG8V
=C2=A0 =C2=A0 label: efiG8V
=C2=A0 =C2=A0 label: diskG8Vsys
=C2=A0 =C2=A0 label: swapG8V
=C2=A0 =C2=A0 label: diskG8Vtank0
=C2=A0 =C2=A0 label: bootC90
=C2=A0 =C2=A0 label: efiC90
=C2=A0 =C2=A0 label: diskC90sys
=C2=A0 =C2=A0 label: swapC90
=C2=A0 =C2=A0 label: diskC90tank0


# ll /dev/gpt/
ls: /dev/gpt/: No such file or directory


# ll /dev/ada*
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x73 Nov 27 00:00 /dev/a= da0
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x81 Nov 27 00:00 /dev/a= da1


Sysctl seems normal to me.

# sysctl kern.geom.label
kern.geom.label.disk_ident.enable: 1
kern.geom.label.gptid.enable: 1
kern.geom.label.gpt.enable: 1
kern.geom.label.ufs.enable: 1
kern.geom.label.ufsid.enable: 1
kern.geom.label.reiserfs.enable: 1
kern.geom.label.ntfs.enable: 1
kern.geom.label.msdosfs.enable: 1
kern.geom.label.iso9660.enable: 1
kern.geom.label.flashmap.enable: 1
kern.geom.label.ext2fs.enable: 1
kern.geom.label.debug: 0


This is the only way I can access disk partitions

# ll /dev/diskid/
total 0
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x7f Nov 27 00:00 DISK-Z= 3010G8V
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x99 Nov 27 00:00 DISK-Z= 3010G8Vp1
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x9b Nov 27 00:00 DISK-Z= 3010G8Vp2
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x9d Nov 27 00:00 DISK-Z= 3010G8Vp3
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x9f Nov 27 00:00 DISK-Z= 3010G8Vp4
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xa1 Nov 27 00:00 DISK-Z= 3010G8Vp5
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xad Nov 27 00:00 DISK-Z= 302YC90
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xc5 Nov 27 00:00 DISK-Z= 302YC90p1
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xc7 Nov 27 00:00 DISK-Z= 302YC90p2
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xc9 Nov 27 00:00 DISK-Z= 302YC90p3
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xcb Nov 27 00:00 DISK-Z= 302YC90p4
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xcd Nov 27 00:00 DISK-Z= 302YC90p5


I created two pools (sys and tank1) with adaN devices when there were 4 disks and the system was booted from USB flash drive with FreeBSD 13.2; current ada0 was ada2 and current ada1 was ada3.

These commands were used

# zpool create -m none sys mirror ada2p3 ada3p3

# zpool create -m none tank1 mirror ada2p5 ada3p5

But now, when the system is booted of these disks, they are reported
only by diskids.

# zpool list -v
NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2=A0 CKPOINT=C2=A0 E= XPANDSZ
FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP=C2=A0 =C2=A0 HEALTH=C2=A0 ALTROOT
sys=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A039.5G=C2=A0 15.9G=C2=A0 23.6G=C2=A0 =C2=A0 =C2=A0 =C2= =A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
12%=C2=A0 =C2=A0 40%=C2=A0 1.00x=C2=A0 =C2=A0 ONLINE=C2=A0 -
=C2=A0 =C2=A0mirror-0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 39.5G=C2=A0 15.9G=C2=A0 23.6G=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0-
12%=C2=A0 40.2%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z3010G8Vp3=C2=A0 =C2=A0 40G=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z302YC90p3=C2=A0 =C2=A0 40G=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
tank1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A03.58T=C2=A0 1.22T=C2=A0 2.36T=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
0%=C2=A0 =C2=A0 34%=C2=A0 1.00x=C2=A0 =C2=A0 ONLINE=C2=A0 -
=C2=A0 =C2=A0mirror-0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 3.58T=C2=A0 1.22T=C2=A0 2.36T=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0-
0%=C2=A0 34.0%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z3010G8Vp5=C2=A0 3.59T=C2=A0 =C2=A0 =C2=A0 = -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z302YC90p5=C2=A0 3.59T=C2=A0 =C2=A0 =C2=A0 = -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE


I did similar setup and migration many times and never seen this problem.
What can be wrong with this system?

The= problem is a geom design point. When you have multiple aliases that you ca= n open, the others are removed from the filesystem view when one is opened.= It looks like from the above that the mirror-0 was created with the diskid= , so that's the device that zfs opens, leaving the ada2 devices removed= .

Warner
--000000000000a9fd16060b3bcd43-- From nobody Tue Nov 28 20:03:02 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sftg72Kgyz52TbB for ; Tue, 28 Nov 2023 20:03:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oi1-x22a.google.com (mail-oi1-x22a.google.com [IPv6:2607:f8b0:4864:20::22a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sftg70jSWz4bRR for ; Tue, 28 Nov 2023 20:03:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x22a.google.com with SMTP id 5614622812f47-3b85c88710eso2218924b6e.3 for ; Tue, 28 Nov 2023 12:03:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701201793; x=1701806593; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/DBp1+PlGI0oqKKWS91NusrWU4NcdGMIueNbC+7N8F0=; b=cIJ1jHcSyrEhv5W6y49PbYaC3yaMm4ZR+dGHvk245GGyYHz+ogn1sIVBya1OC95dgo FeqFsx1gFreV6HL/qu0V1fpoCs8zisatGh4o3YGPMMntOs4vTAEyAbMZPUDxHhJ/H7P5 VX9G8o2T8QYCOBOu8pCSb0ynDiU7MDZncJPgGI78PYQqcdZlSLXxUeilEdNPVIjDwThW 0u0j5cgRkcN+8Hw/rLERiiFJw8/CS/MTaE+IfISVN/LGChtit+n7Kb8XYxOlckxkDc0+ RvwIG3dVtW7NjS1OU6s7J6WyTA3n72sttT9nDPV0+rQ3L7nSnJD1Xm3XY7sdp9CXCmiP n0Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701201793; x=1701806593; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/DBp1+PlGI0oqKKWS91NusrWU4NcdGMIueNbC+7N8F0=; b=XVuWgUVFRIA4lNxz5QFmx2VfZjiHxIx/RfhJemMUCRhDXDlIdMMsKzovJtjQOf1GdG LZgjN+B18tqfpn9T/ClHDAMInfMlfwFdKFr3l5Bp7jNmtD1rUVRQMpH7FfKKY1HYkUJJ 2977Yh4n7zvx5V9iQNU07TpGahuA/vB9HIk7zcfd7b0SiH1KqodAmQWJGtrnw9kVjryv 3Y2KAplO0mEjbNLUZme1Po80pTE4mn97hQKoQWzQaJGDXgOWjdaRVlNxupOP8Z54nr2R Z9vJsNG4vZ1PwMqT3Oe0tbzdp4FtPImYMzQwvPsSZMgvpdm5ejYwEK/rJx0oEBu896W5 wXwA== X-Gm-Message-State: AOJu0YzhGejUh+qFQboukvIWWNwPKZDz7B8AhAOIkBm0UhPfHfwz+mPo JEYRGkx1x1Cu5CMJUkBtxJF7AzSm8G2qs1Y1AtA= X-Google-Smtp-Source: AGHT+IGEVNnaGe5DWEmdbtuKzsR6lwVL0zVzEhK8t3MNTf7U8okf2IDeacrRCjncas11sZS5Ijye7TK+4vGBu5uPGNk= X-Received: by 2002:a05:6808:18a9:b0:3b8:4d1e:1da with SMTP id bi41-20020a05680818a900b003b84d1e01damr18686040oib.37.1701201793375; Tue, 28 Nov 2023 12:03:13 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> In-Reply-To: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> From: Freddie Cash Date: Tue, 28 Nov 2023 12:03:02 -0800 Message-ID: Subject: Re: Missing disk partition devices and GPT lables in /dev/ To: Miroslav Lachman <000.fbsd@quip.cz> Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="00000000000095e29b060b3be826" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Sftg70jSWz4bRR --00000000000095e29b060b3be826 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 28, 2023 at 11:51=E2=80=AFAM Miroslav Lachman <000.fbsd@quip.cz= > wrote: > I have strange problem on one of my machines - the disk devices in /dev/ > are only shown as ada0 and ada1 but no partition devices, no devices in > /dev/gpt/ or /dev/gptid/, these directories are completely missing even > though there are GPT labels. > This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 > Seagate NAS 4TB, manually partitioned with gpart utility, zpool created > and then all the date migrated by zfs send and zfs receive from old 2TB > disks. Everything works, system is running but device nodes in /dev/ are > missing. I noticed this only when I want to create gmirrored swap from > ada0p4 and ada1p4 but got "No such file or directory." > > There are only diskid style devices in /dev/diskid/ > Sysctl seems normal to me. > > # sysctl kern.geom.label > kern.geom.label.disk_ident.enable: 1 > kern.geom.label.gptid.enable: 1 > Turn these two (above) off (set to 0). Then reboot. > What can be wrong with this system? > Welcome to the wonderful world of GEOM tasting, "first match wins", and Highlander's "There can be only 1" style of aliasing/labelling. :) The first alias path to a disk that is found (/dev/diskid, /dev/gptid, /dev/gpt, /dev/ada*) and opened causes all other aliases to be hidden. The disk/GPT IDs seem to take precedence over other aliases/labels. If you want to access your disks via GPT labels, then disable the other naming schemes via loader.conf. --=20 Freddie Cash fjwcash@gmail.com --00000000000095e29b060b3be826 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Nov 28, 2023 at 11:51=E2=80=AFAM = Miroslav Lachman <000.fbsd@quip.cz> wrote:
I have strange problem on one of my machines - the= disk devices in /dev/
are only shown as ada0 and ada1 but no partition devices, no devices in /dev/gpt/ or /dev/gptid/, these directories are completely missing even though there are GPT labels.
This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 Seagate NAS 4TB, manually partitioned with gpart utility, zpool created and then all the date migrated by zfs send and zfs receive from old 2TB disks. Everything works, system is running but device nodes in /dev/ are missing. I noticed this only when I want to create gmirrored swap from
ada0p4 and ada1p4 but got "No such file or directory."

There are only diskid style devices in /dev/diskid/

Sysctl seems normal to me.

# sysctl kern.geom.label
kern.geom.label.disk_ident.enable: 1
kern.geom.label.gptid.enable: 1

What can be wrong with this system?


The first alias path to a disk that is found (/dev/diskid, /dev/gptid, /= dev/gpt, /dev/ada*) and opened causes all other aliases to be hidden.=C2=A0= The disk/GPT IDs seem to take precedence over other aliases/labels.=C2=A0 = If you want to access your disks via GPT labels, then disable the other nam= ing schemes via loader.conf.

--
Freddie Cash
fjw= cash@gmail.com
--00000000000095e29b060b3be826-- From nobody Tue Nov 28 22:26:28 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SfxrV6cDBz52frr for ; Tue, 28 Nov 2023 22:26:34 +0000 (UTC) (envelope-from SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SfxrV2yMdz3Pf8 for ; Tue, 28 Nov 2023 22:26:34 +0000 (UTC) (envelope-from SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 2BA63D789A; Tue, 28 Nov 2023 23:26:32 +0100 (CET) Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 74777D788C; Tue, 28 Nov 2023 23:26:28 +0100 (CET) Message-ID: <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> Date: Tue, 28 Nov 2023 23:26:28 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Missing disk partition devices and GPT lables in /dev/ Content-Language: cs-Cestina, en-US To: stable@freebsd.org References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> Cc: Warner Losh In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Queue-Id: 4SfxrV2yMdz3Pf8 On 28/11/2023 20:55, Warner Losh wrote: > > The problem is a geom design point. When you have multiple aliases that > you can open, the others are removed from the filesystem view when one > is opened. It looks like from the above that the mirror-0 was created > with the diskid, so that's the device that zfs opens, leaving the ada2 > devices removed. No I really didn't use diskids for pool creation. Those diskids are too long to type. :) That's why I explicitly post about commands I used: > # zpool create -m none sys mirror ada2p3 ada3p3 > > # zpool create -m none tank1 mirror ada2p5 ada3p5 I understand that "some" aliases are hidden when other style of aliases were used but I never see ada0pX missing even if gpt labels are used. Also partition p4 prepared for swap was not used by any name style / alias so I expect to see it there but it is not there. ANd the behavior is the exact opposite - when I now tried to create swap with diskids, /dev/gpt/ directory was instantly created a populated with gpt label aliases: # gmirror label -F -h gmGswap0 /dev/diskid/DISK-Z3010G8Vp4 /dev/diskid/DISK-Z302YC90p4 # ll /dev/gpt/ total 0 crw-r----- 1 root operator 0xf9 Nov 28 23:22 swapC90 crw-r----- 1 root operator 0x115 Nov 28 23:22 swapG8V This is normal example from another machine with similar setup: # ls -1 /dev/ada* /dev/ada0 /dev/ada0p1 /dev/ada0p2 /dev/ada0p3 /dev/ada1 /dev/ada1p1 /dev/ada1p2 /dev/ada1p3 /dev/ada2 /dev/ada2p2 /dev/ada3 /dev/ada3p2 # ls -1 /dev/gpt/ boot0 boot1 disk0tank0 disk1tank0 ssd0tank1 ssd1tank1 swap0 swap1 # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ssdtank1 199G 51.4G 148G - - 75% 25% 1.00x ONLINE - mirror-0 199G 51.4G 148G - - 75% 25.8% - ONLINE gpt/ssd0tank1 200G - - - - - - - ONLINE gpt/ssd1tank1 200G - - - - - - - ONLINE tank0 1.80T 714G 1.10T - - 52% 38% 1.00x ONLINE - mirror-0 1.80T 714G 1.10T - - 52% 38.8% - ONLINE gpt/disk0tank0 1.81T - - - - - - - ONLINE gpt/disk1tank0 1.81T - - - - - - - ONLINE As you can see there are aliases in /dev/gpt/, there are partitions for /dev/adaX devices and the pools are made of gpt labels. This is what I expect from the system in my first post. I will try to disable disk_ident as suggested by Freddie Cash. Kind regards Miroslav Lachman From nobody Tue Nov 28 23:13:48 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sfyv36pXwz52jKP for ; Tue, 28 Nov 2023 23:13:51 +0000 (UTC) (envelope-from SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sfyv26b0Fz3TMP for ; Tue, 28 Nov 2023 23:13:50 +0000 (UTC) (envelope-from SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of "SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz"; dmarc=none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 6C222D798A for ; Wed, 29 Nov 2023 00:13:49 +0100 (CET) Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id BFEA9D788C for ; Wed, 29 Nov 2023 00:13:48 +0100 (CET) Message-ID: <35a60698-30f4-4ca5-9b86-f274a4c2beb6@quip.cz> Date: Wed, 29 Nov 2023 00:13:48 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Missing disk partition devices and GPT lables in /dev/ Content-Language: cs-Cestina, en-US To: stable@freebsd.org References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [2.46 / 15.00]; AUTH_NA(1.00)[]; NEURAL_SPAM_SHORT(0.90)[0.899]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz]; NEURAL_SPAM_MEDIUM(0.25)[0.251]; NEURAL_SPAM_LONG(0.10)[0.101]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; DMARC_NA(0.00)[quip.cz]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=hzR4=HJ=quip.cz=000.fbsd@elsa.codelab.cz]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Sfyv26b0Fz3TMP X-Spamd-Bar: ++ On 28/11/2023 23:26, Miroslav Lachman wrote: > On 28/11/2023 20:55, Warner Losh wrote: > As you can see there are aliases in /dev/gpt/, there are partitions for > /dev/adaX devices and the pools are made of gpt labels. > This is what I expect from the system in my first post. > > I will try to disable disk_ident as suggested by Freddie Cash. I rebooted the machine with disabled disk_ident and gptid but nothing changed: # ls -1 /dev/ada* /dev/ada0 /dev/ada1 # ls -1 /dev/gpt ls: /dev/gpt: No such file or directory # zpool list -v | cut -c1-28 NAME sys mirror-0 diskid/DISK-Z3010G8Vp3 diskid/DISK-Z302YC90p3 tank1 mirror-0 diskid/DISK-Z3010G8Vp5 diskid/DISK-Z302YC90p5 # sysctl kern.geom.label kern.geom.label.disk_ident.enable: 0 kern.geom.label.gptid.enable: 0 kern.geom.label.gpt.enable: 1 kern.geom.label.ufs.enable: 1 kern.geom.label.ufsid.enable: 1 kern.geom.label.reiserfs.enable: 1 kern.geom.label.ntfs.enable: 1 kern.geom.label.msdosfs.enable: 1 kern.geom.label.iso9660.enable: 1 kern.geom.label.flashmap.enable: 1 kern.geom.label.ext2fs.enable: 1 kern.geom.label.debug: 0 # ls -1 /dev/diskid/ DISK-Z3010G8V DISK-Z3010G8Vp1 DISK-Z3010G8Vp2 DISK-Z3010G8Vp3 DISK-Z3010G8Vp4 DISK-Z3010G8Vp5 DISK-Z302YC90 DISK-Z302YC90p1 DISK-Z302YC90p2 DISK-Z302YC90p3 DISK-Z302YC90p4 DISK-Z302YC90p5 Why are diskid aliases created if they should be disabled by sysctl kern.geom.label.disk_ident.enable? Kind regards Miroslav Lachman From nobody Wed Nov 29 18:23:23 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SgSPh6dDzz5348F for ; Wed, 29 Nov 2023 18:23:36 +0000 (UTC) (envelope-from mike@jellydonut.org) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SgSPh4R63z4JgM for ; Wed, 29 Nov 2023 18:23:36 +0000 (UTC) (envelope-from mike@jellydonut.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-50bc22c836bso183866e87.0 for ; Wed, 29 Nov 2023 10:23:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jellydonut.org; s=google; t=1701282215; x=1701887015; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=C42M5th/nrpQJBp3qpDz/wvRLfBpemcdoAyCFlZCZhQ=; b=ieUWOMzWwcnxSy0o9UZVXKyi2rIWtk2cCRBSnWxZUktJKnGzGVjKbKgr5uiCUlXHxh kcxrYxQDBdOvGB6UyF/8m/lSwj6iIq3Y4LKaCX6YSp6Y7qGD7yZb4QSUGD2x4RZGeEP+ W4jk4ItRTtjYndB8um61qeg7zheVvk0i3Rj8Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701282215; x=1701887015; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=C42M5th/nrpQJBp3qpDz/wvRLfBpemcdoAyCFlZCZhQ=; b=m+yzrBv3J3N6fjfCo/5injdIjuFBXs04binmC7tz8iGAIuca4fl5v4AY15N7sJ8B62 PLJ+iy2q8pUHBXm3T57105U4cle0kC3nYHzPTRTDXxB2b7RUGhzhY2oyWyzAnGkxapoI tOXnVpkZAZjKNAOvMcHRdfom4b2nPh6uXX31fufg8neaJ0N/THgEJE15v/bfCAMfjruE zSzRcpKJ1DRMPLdEYu4ZI4P0CDdW0fxRUx3mX1Wj/jaZ4zrC85wMgIN7wjKRJNwSzt8B nywwJT1o5PcAYFAPspZKMPJxqfxx8J+jKOUfovIDBshiCC9lWtkgfVO+h1uxuRgLC1If z5gg== X-Gm-Message-State: AOJu0YwYbz/9a/wX7FHeA0qULC0MYAW6DPhVP8mPzhvmjN+AJrMn2o/M 8ShbLyPase0MRcKZz8Ir8R/4n7oiQswrnfqw1A3VeESc6oM81F7d X-Google-Smtp-Source: AGHT+IGELGK3glQipyy5AFWWn+73l3pWQCiozfT4oSswD0fcVB1ZSg3u7ivbO6PBpGlsfJw0jWppm2TS4ig4bV35kM8= X-Received: by 2002:a19:5e12:0:b0:50b:c2cf:eee1 with SMTP id s18-20020a195e12000000b0050bc2cfeee1mr2143331lfb.28.1701282214536; Wed, 29 Nov 2023 10:23:34 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> In-Reply-To: <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> From: Michael Proto Date: Wed, 29 Nov 2023 13:23:23 -0500 Message-ID: Subject: Re: Missing disk partition devices and GPT lables in /dev/ To: Miroslav Lachman <000.fbsd@quip.cz> Cc: stable@freebsd.org, Warner Losh Content-Type: multipart/alternative; boundary="0000000000000f8144060b4ea2d4" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SgSPh4R63z4JgM --0000000000000f8144060b4ea2d4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 28, 2023 at 5:26=E2=80=AFPM Miroslav Lachman <000.fbsd@quip.cz>= wrote: > > I understand that "some" aliases are hidden when other style of aliases > were used but I never see ada0pX missing even if gpt labels are used. > > > This has always been the case on my systems going back to at least 11.0 (when I first started using ZFS) using diskid over gpt. When I import a vdev via /dev/disk/SERIALp1 its associated /dev/ada1p1 disappears. -Michael Proto --0000000000000f8144060b4ea2d4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Nov 28, 2023 at 5:26=E2=80=AF= PM Miroslav Lachman <000.fbsd@quip.c= z> wrote:

I understand that "some" aliases are hidden when other style of a= liases
were used but I never see ada0pX missing even if gpt labels are used.



This has always been the case on m= y systems going back to at least 11.0 (when I first started using ZFS) usin= g diskid over gpt. When I import a vdev via /dev/disk/SERIALp1 its associat= ed /dev/ada1p1 disappears.


-Michael= Proto
--0000000000000f8144060b4ea2d4-- From nobody Wed Nov 29 20:58:49 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SgWs11ljSz52FW3 for ; Wed, 29 Nov 2023 20:59:01 +0000 (UTC) (envelope-from freebsd-stable1@sentry.org) Received: from shadow.sentry.org (shadow.sentry.org [210.8.237.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "shadow.sentry.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SgWrz4zcrz4d6t for ; Wed, 29 Nov 2023 20:58:59 +0000 (UTC) (envelope-from freebsd-stable1@sentry.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of freebsd-stable1@sentry.org designates 210.8.237.106 as permitted sender) smtp.mailfrom=freebsd-stable1@sentry.org; dmarc=none Received: from shadow.sentry.org (localhost [127.0.0.1]) by shadow.sentry.org (8.17.1/8.16.1) with ESMTP id 3ATKwnKm062701 for ; Thu, 30 Nov 2023 07:58:49 +1100 (AEDT) (envelope-from freebsd-stable1@sentry.org) Subject: Re: ZFS high CPU use after backup and panic on shutdown From: Trev To: FreeBSD-STABLE Mailing List References: <6ccc544a-7919-57ab-5572-db67fa09ae76@sentry.org> <67606352-8546-7016-5e65-0f5c4dec8a19@sentry.org> Message-ID: Date: Thu, 30 Nov 2023 07:58:49 +1100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: <67606352-8546-7016-5e65-0f5c4dec8a19@sentry.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Message whitelisted by Sendmail access database, not delayed by milter-greylist-4.6.4 (shadow.sentry.org [0.0.0.0]); Thu, 30 Nov 2023 07:58:49 +1100 (AEDT) X-Spamd-Result: default: False [1.55 / 15.00]; FORGED_MUA_MOZILLA_MAIL_MSGID_UNKNOWN(2.50)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_HAM_MEDIUM(-0.75)[-0.747]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2764, ipnet:210.8.0.0/15, country:AU]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[sentry.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4SgWrz4zcrz4d6t X-Spamd-Bar: + Trev wrote on 28/11/23 12:09 am: > Trev wrote on 25/11/23 2:39 pm: >> I recently updated from source FreeBSD 12-STABLE to FreeBSD 13-STABLE >> (stable/13-221a60a42: Tue Nov 14 15:36:40 AEDT 2023). >> >> Ever since, after my ZFS backup to an external USB drive, the system >> continues to consume 100% of one core of the 2011 Mac mini (i7, 16G). >> >> Example backup command from my shell script: >> >> zfs send data/www@${snapshot} | bzip2 > /mnt/zfs-data-www-${snapshot}.bz2 >> >> Neither top, vmstat nor iostat give any clue as to what system process >> is using 100% of one core. To stop this phenomenon I have to shutdown >> and reboot the system which I do with "shutdown -r now". >> >> This always results in a kernel panic: > > [CHOMP] > > Resolved by updating the source to stable/13-ecb4d2c6e: Sat Nov 25 > 11:31:12 AEDT 2023 and rebuilding. > > [Also, thanks to Mike Karels for off-list help with top.] ARGH. After a few days the issue has again returned. top -S -H shows that the process which continues to consume 100% of one CPU core is arc_prune, the same process that causes a kernel panic on shutdown. Any ideas on how to resolve this? From nobody Wed Nov 29 21:10:29 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SgX6S2vQXz52GGF for ; Wed, 29 Nov 2023 21:10:40 +0000 (UTC) (envelope-from pete@twisted.org.uk) Received: from toybox.twisted.org.uk (toybox.twisted.org.uk [178.250.76.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SgX6R11dJz4jH4 for ; Wed, 29 Nov 2023 21:10:39 +0000 (UTC) (envelope-from pete@twisted.org.uk) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=twisted.org.uk header.s=tbx header.b=aqBEJMOQ; spf=pass (mx1.freebsd.org: domain of pete@twisted.org.uk designates 178.250.76.50 as permitted sender) smtp.mailfrom=pete@twisted.org.uk; dmarc=pass (policy=none) header.from=twisted.org.uk DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=twisted.org.uk; s=tbx; h=Content-Transfer-Encoding:Content-Type:In-Reply-To :From:References:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To:Cc: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=A46/NqhY3nsGewM04TkVdUvb1n+411XLHchW99dg+s8=; t=1701292239; x=1702156239; b=aqBEJMOQz/ZEzlL/blEDMZeqWVvF/Ry15uluIDxzPzscHZ8YRQ+tApiO8HSE5JBwRvnZPWN8+ZP 2+ridOp5pHdr3MShcnGm0yKbOagU8e657GTKuUCEeB3Nk3syhrMkfm9PPZFvp7LLjx8DV8e21Minp caNminSbWWJohI6y9H0QLeETiYo+FdW+A9g+aULvoh3NzcaVmBNtsAei5+JLvOETJyM46o4mAwhSA VAnqMuJ92W9QWsIlURkbL4F1S1hYfaawwHspu4dnP5Q4HcV5t3rSdsf7jIx0+RMBRzwWyn6Y2HXZz pIiUrwC4PCykxGzbeUkhUoG1f1AVnyOxtV3A==; Received: from mailnull by toybox.twisted.org.uk with spamc-scanned (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r8RpG-0001gV-0S for stable@freebsd.org; Wed, 29 Nov 2023 21:10:30 +0000 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on toybox.twisted.org.uk X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=4.0.0 X-Spam-Score: -1.0 () Received: from balta.twisted.org.uk ([2001:470:6cc4:1::57]) by toybox.twisted.org.uk with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r8RpG-0001gS-0E for stable@freebsd.org; Wed, 29 Nov 2023 21:10:30 +0000 Message-ID: <3b1f5ed3-57ff-46f4-b8b5-1d57ef1b22df@twisted.org.uk> Date: Wed, 29 Nov 2023 21:10:29 +0000 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: ZFS high CPU use after backup and panic on shutdown To: stable@freebsd.org References: <6ccc544a-7919-57ab-5572-db67fa09ae76@sentry.org> <67606352-8546-7016-5e65-0f5c4dec8a19@sentry.org> Content-Language: en-GB From: Pete French In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-spamc-toybox: true X-transport-toybox: lookuphost X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[twisted.org.uk,none]; R_SPF_ALLOW(-0.20)[+ip4:178.250.76.50/32]; R_DKIM_ALLOW(-0.20)[twisted.org.uk:s=tbx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ASN(0.00)[asn:12290, ipnet:178.250.72.0/21, country:GB]; MLMMJ_DEST(0.00)[stable@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[twisted.org.uk:+]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4SgX6R11dJz4jH4 X-Spamd-Bar: --- On 29/11/2023 20:58, Trev wrote: > Trev wrote on 28/11/23 12:09 am: > ARGH. After a few days the issue has again returned. top -S -H shows > that the process which continues to consume 100% of one CPU core is > arc_prune, the same process that causes a kernel panic on shutdown. > > Any ideas on how to resolve this? > I would think you are seeing this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274698 Which was fixed by this: https://freshbsd.org/freebsd/src?q=commit%3A3ec4ea68d* So upgrade to a 14-STABLE after that and you will be fine. You probably want to upfrade as far as today though, to pick up some of the other fixes which went into OpenZFS. -pete. From nobody Wed Nov 29 23:20:17 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sgb0G0lwCz52PfX for ; Wed, 29 Nov 2023 23:20:30 +0000 (UTC) (envelope-from SRS0=QE77=HK=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sgb0F5nWYz3D6S for ; Wed, 29 Nov 2023 23:20:29 +0000 (UTC) (envelope-from SRS0=QE77=HK=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 3C2C4D7895; Thu, 30 Nov 2023 00:20:21 +0100 (CET) Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 64788D7894; Thu, 30 Nov 2023 00:20:18 +0100 (CET) Message-ID: Date: Thu, 30 Nov 2023 00:20:17 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Missing disk partition devices and GPT lables in /dev/ Content-Language: cs-Cestina, en-US To: Michael Proto Cc: stable@freebsd.org, Warner Losh References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Queue-Id: 4Sgb0F5nWYz3D6S On 29/11/2023 19:23, Michael Proto wrote: > This has always been the case on my systems going back to at least 11.0 > (when I first started using ZFS) using diskid over gpt. When I import a > vdev via /dev/disk/SERIALp1 its associated /dev/ada1p1 disappears. I never created / imported pool by /dev/diskid/ serials and I don't understand why I was still getting /dev/diskid/ entries even if they were disabled by sysctl (may be sysctl.conf is applied to late?) I fixed it by booting from USB flash drive, manually forced import with gpt labels, export and reboot: # zpool import -f -d /dev/gpt/ -R /tank1 tank1 # zpool import -f -d /dev/gpt/ -R /sys0 sys # cp /etc/zfs/zpool.cache /sys0/etc/zfs/ # zpool export sys # zpool export tank1 Now after reboot I have these aliases for partitions: # ls -1 /dev/{diskid,gpt,gptid,ada*} ls: /dev/diskid: No such file or directory /dev/ada0 /dev/ada0p1 /dev/ada0p2 /dev/ada0p3 /dev/ada0p4 /dev/ada0p5 /dev/ada1 /dev/ada1p1 /dev/ada1p2 /dev/ada1p3 /dev/ada1p4 /dev/ada1p5 /dev/gpt: bootC90 bootG8V diskC90sys diskC90tank0 diskG8Vsys diskG8Vtank0 efiC90 efiG8V swapC90 swapG8V /dev/gptid: 700a2d08-8be4-11ee-8287-98f2b3f71a30 719acc14-8be4-11ee-8287-98f2b3f71a30 ebaf351d-8bec-11ee-8287-98f2b3f71a30 ec5f6315-8be4-11ee-8287-98f2b3f71a30 ee024e16-8be4-11ee-8287-98f2b3f71a30 fd55c994-8bec-11ee-8287-98f2b3f71a30 And pool is made of GPT lables as I like # zpool list -v | cut -c1-28 NAME SIZE sys 39.5G mirror-0 39.5G gpt/diskG8Vsys 40G gpt/diskC90sys 40G tank1 3.58T mirror-0 3.58T gpt/diskG8Vtank0 3.59T gpt/diskC90tank0 3.59T Kind regards Miroslav Lachman From nobody Thu Nov 30 02:16:47 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sgfvv5P6Cz52blZ for ; Thu, 30 Nov 2023 02:16:59 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp2.bway.net (smtp2.bway.net [216.220.96.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sgfvv3cggz3Zxs for ; Thu, 30 Nov 2023 02:16:59 +0000 (UTC) (envelope-from spork@bway.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (pool-71-187-164-80.nwrknj.fios.verizon.net [71.187.164.80]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp2.bway.net (Postfix) with ESMTPSA id C9CD93FA5C; Wed, 29 Nov 2023 21:16:57 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bway.net; s=mail; t=1701310618; bh=S+4kTfTArJfhNXBoavXoxOa/DTReRP5y3hwWFkEYOAw=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=CD80gy0A+Idh5EF2wnEaVTZeXI42goNA5PpGJojyGuKjUAA0AU02AHBMAOKZ6vmb5 ICmt/pwRviUyBfXdagKEI98c7APsQLEC++tohe0mAI37OYhY8isNctBpZ5t3/Y9W0C eBPAaLoN1DVHz46aZkmvhNOmwjmo6MD6XjJOswRI= Content-Type: text/plain; charset=utf-8 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: Re: Missing disk partition devices and GPT lables in /dev/ From: Charles Sprickman In-Reply-To: Date: Wed, 29 Nov 2023 21:16:47 -0500 Cc: Michael Proto , stable@freebsd.org, Warner Losh Content-Transfer-Encoding: quoted-printable Message-Id: <847A9C4A-54A9-4226-A78A-B9BFCF66EBF6@bway.net> References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> <6b713a9c-c778-45fb-9626-9ad05df28d50@quip.cz> To: Miroslav Lachman <000.fbsd@quip.cz> X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8059, ipnet:216.220.96.0/19, country:US] X-Rspamd-Queue-Id: 4Sgfvv3cggz3Zxs > On Nov 29, 2023, at 6:20=E2=80=AFPM, Miroslav Lachman = <000.fbsd@quip.cz> wrote: >=20 > On 29/11/2023 19:23, Michael Proto wrote: >=20 >> This has always been the case on my systems going back to at least = 11.0 (when I first started using ZFS) using diskid over gpt. When I = import a vdev via /dev/disk/SERIALp1 its associated /dev/ada1p1 = disappears. >=20 > I never created / imported pool by /dev/diskid/ serials and I don't = understand why I was still getting /dev/diskid/ entries even if they = were disabled by sysctl (may be sysctl.conf is applied to late?) Yeah, as far as I know, those go in /boot/loader.conf - at least that's = where the installer puts them... I suspect that's the root of all this.. Charles >=20 > I fixed it by booting from USB flash drive, manually forced import = with gpt labels, export and reboot: >=20 > # zpool import -f -d /dev/gpt/ -R /tank1 tank1 > # zpool import -f -d /dev/gpt/ -R /sys0 sys >=20 > # cp /etc/zfs/zpool.cache /sys0/etc/zfs/ >=20 > # zpool export sys > # zpool export tank1 >=20 > Now after reboot I have these aliases for partitions: >=20 > # ls -1 /dev/{diskid,gpt,gptid,ada*} > ls: /dev/diskid: No such file or directory > /dev/ada0 > /dev/ada0p1 > /dev/ada0p2 > /dev/ada0p3 > /dev/ada0p4 > /dev/ada0p5 > /dev/ada1 > /dev/ada1p1 > /dev/ada1p2 > /dev/ada1p3 > /dev/ada1p4 > /dev/ada1p5 >=20 > /dev/gpt: > bootC90 > bootG8V > diskC90sys > diskC90tank0 > diskG8Vsys > diskG8Vtank0 > efiC90 > efiG8V > swapC90 > swapG8V >=20 > /dev/gptid: > 700a2d08-8be4-11ee-8287-98f2b3f71a30 > 719acc14-8be4-11ee-8287-98f2b3f71a30 > ebaf351d-8bec-11ee-8287-98f2b3f71a30 > ec5f6315-8be4-11ee-8287-98f2b3f71a30 > ee024e16-8be4-11ee-8287-98f2b3f71a30 > fd55c994-8bec-11ee-8287-98f2b3f71a30 >=20 > And pool is made of GPT lables as I like >=20 > # zpool list -v | cut -c1-28 > NAME SIZE > sys 39.5G > mirror-0 39.5G > gpt/diskG8Vsys 40G > gpt/diskC90sys 40G > tank1 3.58T > mirror-0 3.58T > gpt/diskG8Vtank0 3.59T > gpt/diskC90tank0 3.59T >=20 >=20 > Kind regards > Miroslav Lachman >=20 >=20 From nobody Fri Dec 1 20:38:10 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ShlJ43WPlz535Kb for ; Fri, 1 Dec 2023 20:38:12 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ShlJ31t1Nz4RJR for ; Fri, 1 Dec 2023 20:38:11 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net; dmarc=none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 3B1Kc99G090040 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL) for ; Fri, 1 Dec 2023 15:38:09 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:5d0d:96a4:9d50:8405] ([IPv6:2607:f3e0:0:4:5d0d:96a4:9d50:8405]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 3B1Kc83Y018699 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Fri, 1 Dec 2023 15:38:08 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: Date: Fri, 1 Dec 2023 15:38:10 -0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: FreeBSD-STABLE Mailing List From: mike tancsa Subject: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 X-Spamd-Result: default: False [-3.38 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.989]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FREEFALL_USER(0.00)[mike]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[sentex.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4ShlJ31t1Nz4RJR X-Spamd-Bar: --- Was testing out a zfs mirror install and noticed that the install process correctly sets up the same partition scheme on both drives, but only formats and installs the efi boot files on the one disk. For a proper mirror and protection, should not the second disk also have the ability to boot ? Also a little problematic is that the fstab has /dev/gtp/efiboot0 mount automatically.  I would think it would be better to optionally mount this. Otherwise, if that disk died, it would be stuck in single user mode since it cant mount that disk.     ---Mike From nobody Fri Dec 1 21:53:49 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ShmzK2Jllz53KGX for ; Fri, 1 Dec 2023 21:53:49 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ShmzJ4GZsz4f6r for ; Fri, 1 Dec 2023 21:53:48 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net; dmarc=none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 3B1Lrmng098220 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL) for ; Fri, 1 Dec 2023 16:53:48 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:5d0d:96a4:9d50:8405] ([IPv6:2607:f3e0:0:4:5d0d:96a4:9d50:8405]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 3B1Lrlvt025619 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Fri, 1 Dec 2023 16:53:47 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: Date: Fri, 1 Dec 2023 16:53:49 -0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) Content-Language: en-US From: mike tancsa To: FreeBSD-STABLE Mailing List References: Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FREEFALL_USER(0.00)[mike]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[sentex.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4ShmzJ4GZsz4f6r X-Spamd-Bar: --- On 12/1/2023 3:38 PM, mike tancsa wrote: > Was testing out a zfs mirror install and noticed that the install > process correctly sets up the same partition scheme on both drives, > but only formats and installs the efi boot files on the one disk. For > a proper mirror and protection, should not the second disk also have > the ability to boot ? > > Also a little problematic is that the fstab has /dev/gtp/efiboot0 > mount automatically.  I would think it would be better to optionally > mount this. Otherwise, if that disk died, it would be stuck in single > user mode since it cant mount that disk. > Should have looked at open PRs. There is one from a while ago https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987     ---Mike From nobody Fri Dec 1 22:01:20 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Shn8G5TSnz53Lcw for ; Fri, 1 Dec 2023 22:01:34 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Shn8G3r48z4gq6 for ; Fri, 1 Dec 2023 22:01:34 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x532.google.com with SMTP id 4fb4d7f45d1cf-54c4433e98bso2100192a12.3 for ; Fri, 01 Dec 2023 14:01:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701468092; x=1702072892; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=fiq1F6X272YH9AuhrFu2PCpXzNXC2XFu5HfiywXcgCA=; b=euWJHUQuBaBc/bVaWySPCaGQH4idlXmvBRIwLKByfNxuqDfpVUK/L6E8pc7N+T2W6R AjTd3vNz31I5dXDmp9ydYKzhcTkSXWVhbsVITpj4wmWNawUVBuQhSrEEfmsbaiqX5m2A FcAW6WPxuCOEv5tc9KPNp4kAMEp73YCtEMCWcV4TpKZl+VYddgGtrnn67A1VTP2a6/Dj IGG3Gtrwi/aaAGXXfmDdoM0Iu0o5GscqZTyA55vqnS4Ts6KLspROP3rAHq9SUg7ev7X7 MTVj50HvXqOaUEbPU19N6Uy/aQEV/mbXoFYLwA+DdlmhaGywvH0wrfMsVbkk8xIvBXFY FZaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701468092; x=1702072892; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fiq1F6X272YH9AuhrFu2PCpXzNXC2XFu5HfiywXcgCA=; b=IzYn8UhnnltBxh8qcekLboUfF7hgHabgNu044BKlBWoGmWyKk5XfyHgHdBJHjniaC7 76Ae4qSGLfYMMW2nDLRF5SgRB+1ZvLtkOnNoVn6JfTucELWBPt8mECu+WcaNRyPs/tQ9 VHjfIHQRnjZBiI6GEkIr6unPIAYqC6v3xrokTzzR7xVyRyx9Y3ckzecjHRE4yN4HQx1c Daev8XRGgCj3KszW+01J5fUk1syOdSayo4xx/fdv5fbJ4bKbjmXTRGRdp4rrvVws44Id M2o+8EqFqQ/ZCPyb06LEeRFN28bKYZhYwKKKVdFqlGVfXHfNN6A90GkekdWaBATK/q6r x+uA== X-Gm-Message-State: AOJu0YzTtUkBobj9wNeoSfH0k+S0XR3Is6HsT/kOG3S4Ou8Jf/s+YGND erXmur58ZBzQV29Awz0+PUPa6fRvsE5zgQ9aSUan3U7ieQOgwws6 X-Google-Smtp-Source: AGHT+IGJHg+83Y2qd1rFp0+H2b332X7+PNfS8gzNpvXyV61xAHJky+LMCBT2J78Cb1VuvUP3f6P3jw1TKPvDGZdFZiI= X-Received: by 2002:a50:bb05:0:b0:54c:4fec:de with SMTP id y5-20020a50bb05000000b0054c4fec00demr509315ede.109.1701468092342; Fri, 01 Dec 2023 14:01:32 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 1 Dec 2023 15:01:20 -0700 Message-ID: Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) To: mike tancsa Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="0000000000003db61e060b79e91b" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4Shn8G3r48z4gq6 --0000000000003db61e060b79e91b Content-Type: text/plain; charset="UTF-8" On Fri, Dec 1, 2023, 2:54 PM mike tancsa wrote: > On 12/1/2023 3:38 PM, mike tancsa wrote: > > Was testing out a zfs mirror install and noticed that the install > > process correctly sets up the same partition scheme on both drives, > > but only formats and installs the efi boot files on the one disk. For > > a proper mirror and protection, should not the second disk also have > > the ability to boot ? > > > > Also a little problematic is that the fstab has /dev/gtp/efiboot0 > > mount automatically. I would think it would be better to optionally > > mount this. Otherwise, if that disk died, it would be stuck in single > > user mode since it cant mount that disk. > > > Should have looked at open PRs. There is one from a while ago > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987 I think you are right. Getting redundancy right is hard. We've not set a pattern to use and updated our tooling to cope with it... Warner > > --0000000000003db61e060b79e91b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Dec 1, 2023, 2:54 PM mike tancsa <mike@sente= x.net> wrote:
On 12/1/2023 3= :38 PM, mike tancsa wrote:
> Was testing out a zfs mirror install and noticed that the install
> process correctly sets up the same partition scheme on both drives, > but only formats and installs the efi boot files on the one disk. For =
> a proper mirror and protection, should not the second disk also have <= br> > the ability to boot ?
>
> Also a little problematic is that the fstab has /dev/gtp/efiboot0
> mount automatically.=C2=A0 I would think it would be better to optiona= lly
> mount this. Otherwise, if that disk died, it would be stuck in single =
> user mode since it cant mount that disk.
>
Should have looked at open PRs. There is one from a while ago

https://bugs.freebs= d.org/bugzilla/show_bug.cgi?id=3D258987

I think you are right. Getting redun= dancy right is hard. We've not set a pattern to use and updated our too= ling to cope with it...

= Warner



--0000000000003db61e060b79e91b-- From nobody Fri Dec 1 23:57:01 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Shqjh5YDCz52Zfm for ; Fri, 1 Dec 2023 23:57:12 +0000 (UTC) (envelope-from pete@twisted.org.uk) Received: from toybox.twisted.org.uk (toybox.twisted.org.uk [178.250.76.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Shqjf4bhKz4V4d for ; Fri, 1 Dec 2023 23:57:10 +0000 (UTC) (envelope-from pete@twisted.org.uk) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=twisted.org.uk header.s=tbx header.b=lfpOg0g+; spf=pass (mx1.freebsd.org: domain of pete@twisted.org.uk designates 178.250.76.50 as permitted sender) smtp.mailfrom=pete@twisted.org.uk; dmarc=pass (policy=none) header.from=twisted.org.uk DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=twisted.org.uk; s=tbx; h=Content-Transfer-Encoding:Content-Type:In-Reply-To :From:References:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To:Cc: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=y/6diezPM8Sfv2cho7XGKFY1V71et10MxTxrkcxBhBo=; t=1701475030; x=1702339030; b=lfpOg0g+jfwW3j3eDZmtilKLPVwp52B/9uLsyZEc/hKci727DV8bIviWbYzv+JhCOeK9P2K7s4T kqTlqW9iI3gikc8LH0tB3x5L9y8vBk918y8j5lDIPTCAs+qJAHiNV1Y4RXxpllzY7fpBD3gT2ooy5 f5G+75bgI0Brmijc8CMSgqr0aLj1jFIDDP8EZ4u5LizGOls8wS/wP+Hsg2lB7hwgvxeDJVHagmF9a iEoGctan9wPpL6ssTB1ShwgVun4IFYu55ViqPEykaD0pwXeATYcx30XYlbdTZis/eVEepmHt3b2OP tYz7PxzmRL0X2G05poIaRmaNeg41SEL6N25A==; Received: from mailnull by toybox.twisted.org.uk with spamc-scanned (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r9DNW-000J15-0W for stable@freebsd.org; Fri, 01 Dec 2023 23:57:02 +0000 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on toybox.twisted.org.uk X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=ALL_TRUSTED,TW_ZF, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=4.0.0 X-Spam-Score: -0.9 () Received: from balta.twisted.org.uk ([2001:470:6cc4:1::57]) by toybox.twisted.org.uk with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r9DNW-000J12-0J for stable@freebsd.org; Fri, 01 Dec 2023 23:57:02 +0000 Message-ID: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> Date: Fri, 1 Dec 2023 23:57:01 +0000 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) Content-Language: en-GB To: stable@freebsd.org References: From: Pete French In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-spamc-toybox: true X-transport-toybox: lookuphost X-Spamd-Result: default: False [-3.96 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.973]; DMARC_POLICY_ALLOW(-0.50)[twisted.org.uk,none]; R_SPF_ALLOW(-0.20)[+ip4:178.250.76.50/32]; R_DKIM_ALLOW(-0.20)[twisted.org.uk:s=tbx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ASN(0.00)[asn:12290, ipnet:178.250.72.0/21, country:GB]; MLMMJ_DEST(0.00)[stable@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[twisted.org.uk:+]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Shqjf4bhKz4V4d X-Spamd-Bar: --- On 01/12/2023 21:53, mike tancsa wrote: > Should have looked at open PRs. There is one from a while ago > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987 > > Was thinking about this, and I was wondering if it would be possible to make the EFI partition a gmirror. So its across all discs, mounted only once, but would still boot from any of them. My understanding is geom has the label at the end, yes ? So the firmware would see the filesystem on a single partition quite happily ? -pete. From nobody Sat Dec 2 00:45:53 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Shrp748QNz52ldj for ; Sat, 2 Dec 2023 00:46:07 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Shrp66Zjxz4c8K for ; Sat, 2 Dec 2023 00:46:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-54c1cd8d239so2803545a12.0 for ; Fri, 01 Dec 2023 16:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701477965; x=1702082765; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5dXTJ7DluiWWzieo8EKKxqyZhmH90fXw7RRSDWVdm2U=; b=yk6cVNf+6I9bl7Zc0mkIRSIiAR7DOv0fc8pJiHTMRee06eOKGrS0u4pPy+ttm3D9Oy V5Y1sTkijLHPK0Zkp/JzVWq7tLiSPuyOi7W2KtkUtyoycAjsb1IT82XvvWp5Cfsg6HDd BuBbk4Jx0wJpbpLmF374RXO48sOQrEJebFknTfzx05BnNGMpy+f+wIYfYBO8CMOnOMYX ig71X1L8eOxsLiIKjp36NfYC8hASzeOZP4T3udFeAajh8CUI8r9ItSdvsEBVkoVLRbQj MZ7+eqQPD1B3e4bg/aUrVuSJwXU1EW2MnJ1ZJrs+9IC5YosCperxXej+/Eynbsza2iPG 8D2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701477965; x=1702082765; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5dXTJ7DluiWWzieo8EKKxqyZhmH90fXw7RRSDWVdm2U=; b=pqa2Kd/MsQ1qq9gywrYBXgPwEXGrDnEFFwHlZ3S6IkQ8cJBCi5VSBKHLiMsTpgzbfW MgtFXwItp30g5cOUVRsOZJCG386GCGy/VW4WWyp3/y/kz5b2iyxboGxYCn1P+AVpHX0V HKZ0yxJ1b0m5kovQWVd8sM7h3BBLTFG/fqVv6JykK7PwHwi92bMOtYZKsdURpldA0Yc5 Ras7l+vFQ1Zl99NkfxCPE+uTLMwoj/i0dEnlawHEtzRiq9wLhnxYPVklTRB+UTvWJLRT 2pfJWisaC2jHU+eXokq3H8S+skXkSFuPMW270GIZSwNggJkmWzS2g5Ex0Eds6/SPU6FY pOCQ== X-Gm-Message-State: AOJu0Yxx6P6OG9YDV14YInK6VFChj+Mp/EnSx1nFQu+Avoa5Hp5JLyzk uYIDekHaeo6WUG3asSAGYpLUXWXo00bOUjTBtLQ2Yl5gvkFZs6mW X-Google-Smtp-Source: AGHT+IFcSQUcyfkuZpuuw50NVFIc/TLhcrJN/+Q9Gd8eSYR3o8fA4+LyNSBxnHcIppnVXWpLET1NdvCp6oFVYxP8TqA= X-Received: by 2002:a05:6402:28af:b0:54c:4837:9fd9 with SMTP id eg47-20020a05640228af00b0054c48379fd9mr500243edb.48.1701477965320; Fri, 01 Dec 2023 16:46:05 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> In-Reply-To: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> From: Warner Losh Date: Fri, 1 Dec 2023 17:45:53 -0700 Message-ID: Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) To: Pete French Cc: FreeBSD Stable ML Content-Type: multipart/alternative; boundary="000000000000b76411060b7c3535" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4Shrp66Zjxz4c8K --000000000000b76411060b7c3535 Content-Type: text/plain; charset="UTF-8" On Fri, Dec 1, 2023, 4:57 PM Pete French wrote: > > On 01/12/2023 21:53, mike tancsa wrote: > > Should have looked at open PRs. There is one from a while ago > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987 > > > > > > Was thinking about this, and I was wondering if it would be possible to > make the EFI partition a gmirror. So its across all discs, mounted only > once, but would still boot from any of them. My understanding is geom > has the label at the end, yes ? So the firmware would see the filesystem > on a single partition quite happily ? > I've done this. It works ok. But I don't run like this in production. If I write a new file, that has so many writes to the different disks. If they all go through then life is good (this is what gets us to OK). BUT, if there is a power failure or crash and only some of them make it to disk, then you have a corrupt ESP and the BIOS may pick that ESP to boot off of, booting corrupt data. Since this is infrequently updated, you can use a safe sequence to update things one partition a time, then you might lose the file entirely, but it will either be there and good. Or it will be gone. You can't get into a bad situation. Either you boot old or new loader and can just quit from the boot loader if it's the old one and it can't boot. Efi will try the next one on the list. Here manual mirroring, if scripted, can be more reliable than gmirror. Warner -pete. > > > --000000000000b76411060b7c3535 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Dec 1, 2023, 4:57 PM Pete French <pete@twisted.org.uk> wrote:

On 01/12/2023 21:53, mike tancsa wrote:
> Should have looked at open PRs. There is one from a while ago
>
> https://bugs.freebsd.org/b= ugzilla/show_bug.cgi?id=3D258987
>
>

Was thinking about this, and I was wondering if it would be possible to make the EFI partition a gmirror. So its across all discs, mounted only once, but would still boot from any of them. My understanding is geom
has the label at the end, yes ? So the firmware would see the filesystem on a single partition quite happily ?

I've done this. It works ok. But = I don't run like this in production. If I write a new file, that has so= many writes to the different disks. If they all go through then life is go= od (this is what gets us to OK).

BUT, if there is a power failure or crash and only some of them m= ake it to disk, then you have a corrupt ESP and the BIOS may pick that ESP = to boot off of, booting corrupt data.

Since this is infrequently updated, you can use a safe sequen= ce to update things one partition a time, then you might lose the file enti= rely, but it will either be there and good. Or it will be gone. You can'= ;t get into a bad situation. Either you boot old or new loader and can just= quit from the boot loader if it's the old one and it can't boot. E= fi will try the next one on the list.

Here manual mirroring, if scripted, can be more reliable than= gmirror.

Warner

-pete.


--000000000000b76411060b7c3535-- From nobody Sat Dec 2 04:26:30 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Shxhh1vy1z52X81 for ; Sat, 2 Dec 2023 04:26:44 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Shxhd4BDlz4HLX for ; Sat, 2 Dec 2023 04:26:40 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-22-158.area1b.commufa.jp [123.1.22.158]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 3B24QUIu039047 for ; Sat, 2 Dec 2023 13:26:31 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Sat, 2 Dec 2023 13:26:30 +0900 From: Tomoaki AOKI To: stable@freebsd.org Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) Message-Id: <20231202132630.9bcbd8588479b8955bafd252@dec.sakura.ne.jp> In-Reply-To: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> References: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [2.11 / 15.00]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.90)[0.901]; NEURAL_HAM_SHORT(-0.65)[-0.650]; MV_CASE(0.50)[]; NEURAL_SPAM_LONG(0.36)[0.360]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[sakura.ne.jp]; R_SPF_NA(0.00)[no SPF record]; TO_DN_NONE(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Shxhd4BDlz4HLX X-Spamd-Bar: ++ On Fri, 1 Dec 2023 23:57:01 +0000 Pete French wrote: > > On 01/12/2023 21:53, mike tancsa wrote: > > Should have looked at open PRs. There is one from a while ago > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987 > > > > > > Was thinking about this, and I was wondering if it would be possible to > make the EFI partition a gmirror. So its across all discs, mounted only > once, but would still boot from any of them. My understanding is geom > has the label at the end, yes ? So the firmware would see the filesystem > on a single partition quite happily ? > > > -pete. This is just (casuall, though) discussed in Forums [1]. I've written comment #19. [1] https://forums.freebsd.org/threads/upgrading-bootloader-on-zfs-mirror.91198/ -- Tomoaki AOKI From nobody Sat Dec 2 04:36:08 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Shxvd4Km6z52Yhm for ; Sat, 2 Dec 2023 04:36:13 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Shxvc4BKHz4K5P for ; Sat, 2 Dec 2023 04:36:12 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-22-158.area1b.commufa.jp [123.1.22.158]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 3B24a9Hg040840 for ; Sat, 2 Dec 2023 13:36:10 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Sat, 2 Dec 2023 13:36:08 +0900 From: Tomoaki AOKI To: stable@freebsd.org Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) Message-Id: <20231202133608.389d3572b4ecaee6a2a38e93@dec.sakura.ne.jp> In-Reply-To: References: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [1.03 / 15.00]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.89)[0.892]; NEURAL_HAM_SHORT(-0.72)[-0.722]; NEURAL_HAM_LONG(-0.64)[-0.643]; MV_CASE(0.50)[]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[sakura.ne.jp]; HAS_ORG_HEADER(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Shxvc4BKHz4K5P X-Spamd-Bar: + On Fri, 1 Dec 2023 17:45:53 -0700 Warner Losh wrote: > On Fri, Dec 1, 2023, 4:57 PM Pete French wrote: > > > > > On 01/12/2023 21:53, mike tancsa wrote: > > > Should have looked at open PRs. There is one from a while ago > > > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987 > > > > > > > > > > Was thinking about this, and I was wondering if it would be possible to > > make the EFI partition a gmirror. So its across all discs, mounted only > > once, but would still boot from any of them. My understanding is geom > > has the label at the end, yes ? So the firmware would see the filesystem > > on a single partition quite happily ? > > > > I've done this. It works ok. But I don't run like this in production. If I > write a new file, that has so many writes to the different disks. If they > all go through then life is good (this is what gets us to OK). > > BUT, if there is a power failure or crash and only some of them make it to > disk, then you have a corrupt ESP and the BIOS may pick that ESP to boot > off of, booting corrupt data. > > Since this is infrequently updated, you can use a safe sequence to update > things one partition a time, then you might lose the file entirely, but it > will either be there and good. Or it will be gone. You can't get into a bad > situation. Either you boot old or new loader and can just quit from the > boot loader if it's the old one and it can't boot. Efi will try the next > one on the list. > > Here manual mirroring, if scripted, can be more reliable than gmirror. > > Warner > > -pete. It looks reasonable if it is auto-generated by bsdinstall, used on initial installation, and kept for later use by `make installworld`. bsdinstall should know how many, and which disk is configured for FreeBSD. And, maybe impossible for now, but if "Actually booted from at this time" ESP is auto-mounted on default location, it could be helpful for admins. Just a thought. -- Tomoaki AOKI From nobody Sat Dec 2 05:34:45 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ShzCS0dYDz52mlH for ; Sat, 2 Dec 2023 05:35:00 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ShzCR5jxFz4SML for ; Sat, 2 Dec 2023 05:34:59 +0000 (UTC) (envelope-from zbeeble@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-50bc8a9503fso3971908e87.3 for ; Fri, 01 Dec 2023 21:34:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701495298; x=1702100098; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=VwSBomomza4Cst1yNVK5IOAviVrsTpFPLIZwPQuuRAE=; b=ndfaatvEOD+5Q8P3NjnP7EfEF7BQCQhZ0hlJWNV/c66S0+av2CVnRW0qHUfMSXvYAz l7t5D8PiUry+tiLFsP/8KICiFSeVfxSIP6Ltgo4FlrpL8dGGxLEkUO5L+zdW6PGLJqXe MCDowHUnNcyBMC5wZFts8HLTkyOzETLFuBTKr8DE0sxBWBPhSJ4kL7xLWMcdYpcpy/IJ NBTfyY9+d5dt/AvfYZBG+uSH31P3rWeX7n421d1msdTfajeFNvkEuOFRGzHHuIWO3lbY UTqsCKXq0rPLB/MqBu/uo4ArwY48fb2Tte1MP7copZ/yT3PTts0aVYmzlbknuwXcVFIz frgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701495298; x=1702100098; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VwSBomomza4Cst1yNVK5IOAviVrsTpFPLIZwPQuuRAE=; b=sGVTy0yyVSdktqYMhYnkxnK7xDKjXp5AT7tnERyh1++1MPkIK3RclyCNbnJvjpmCv6 BC61G0Vb69CdhqKAO/xHckVpi0jACCMFOrdEuKuwWxjAqMr+ZXRRLG6C2HxmjEziBQM7 GghDQCNOxYVZZzcVsK61kflrb6CrUsVsXoiFjWWNkzQ+b1lsTlBegL2pn7ZCeFq2aTgN SYbDq0yydM9mnrEc++hN+82JqOAPAKbCUVSStImpO3AzE40AEK6JpFExIfEbpO/eBw/H fftJed3bSZk68jZWyocsrFiBGIWkBm6jsrzL60mKLa2R0H0RTS6omBOM6q6aEN+9Cx41 dNAQ== X-Gm-Message-State: AOJu0YyHpb+D8JULK/w4DDID5gFXBhomqJijjwSIlycjUfuKSURXSplG zqNgSnZ3Y/4aiF3QL6k09Wzo7NpaQxAfxAcJxOud7hg= X-Google-Smtp-Source: AGHT+IF+33tgSA/TF9OXVeKpUvLKN0TuU1RMrWtwa3kAB42znGLL1gGlqeFaNQL9blyJi41zJDc9rsnqCoEct+gnHg8= X-Received: by 2002:a05:6512:e98:b0:50b:d764:8810 with SMTP id bi24-20020a0565120e9800b0050bd7648810mr1523682lfb.92.1701495297561; Fri, 01 Dec 2023 21:34:57 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> In-Reply-To: From: Zaphod Beeblebrox Date: Sat, 2 Dec 2023 00:34:45 -0500 Message-ID: Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) To: Warner Losh Cc: Pete French , FreeBSD Stable ML Content-Type: multipart/alternative; boundary="000000000000cc5789060b803e27" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ShzCR5jxFz4SML --000000000000cc5789060b803e27 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable It can be more straightforward to update the gmirror, however. I've done this with UFS --- old boot, pair of UFS/GMIRROR usb sticks that then boot to a ZFS array that the BIOS couldn't see (so UFS only contained /boot and /rescue). It's easier to know that the boot is updated identically if gmirrored. Gmirror also has tools to verify, etc. On Fri, Dec 1, 2023 at 7:46=E2=80=AFPM Warner Losh wrote: > > > On Fri, Dec 1, 2023, 4:57 PM Pete French wrote: > >> >> On 01/12/2023 21:53, mike tancsa wrote: >> > Should have looked at open PRs. There is one from a while ago >> > >> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258987 >> > >> > >> >> Was thinking about this, and I was wondering if it would be possible to >> make the EFI partition a gmirror. So its across all discs, mounted only >> once, but would still boot from any of them. My understanding is geom >> has the label at the end, yes ? So the firmware would see the filesystem >> on a single partition quite happily ? >> > > I've done this. It works ok. But I don't run like this in production. If = I > write a new file, that has so many writes to the different disks. If they > all go through then life is good (this is what gets us to OK). > > BUT, if there is a power failure or crash and only some of them make it t= o > disk, then you have a corrupt ESP and the BIOS may pick that ESP to boot > off of, booting corrupt data. > > Since this is infrequently updated, you can use a safe sequence to update > things one partition a time, then you might lose the file entirely, but i= t > will either be there and good. Or it will be gone. You can't get into a b= ad > situation. Either you boot old or new loader and can just quit from the > boot loader if it's the old one and it can't boot. Efi will try the next > one on the list. > > Here manual mirroring, if scripted, can be more reliable than gmirror. > > Warner > > -pete. >> >> >> --000000000000cc5789060b803e27 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It can be more straightforward to update the gmirror, howe= ver.=C2=A0 I've done this with UFS --- old boot, pair of UFS/GMIRROR us= b sticks that then boot to a ZFS array that the BIOS couldn't see (so U= FS only contained /boot and /rescue).=C2=A0 It's easier to know that th= e boot is updated identically if gmirrored.=C2=A0 Gmirror also has tools to= verify, etc.

On Fri, Dec 1, 2023 at 7:46=E2=80=AFPM Warner Losh <imp@bsdimp.com> wrote:

On Fri, D= ec 1, 2023, 4:57 PM Pete French <pete@twisted.org.uk> wrote:

On 01/12/2023 21:53, mike tancsa wrote:
> Should have looked at open PRs. There is one from a while ago
>
> https://bugs.freebsd.org/b= ugzilla/show_bug.cgi?id=3D258987
>
>

Was thinking about this, and I was wondering if it would be possible to make the EFI partition a gmirror. So its across all discs, mounted only once, but would still boot from any of them. My understanding is geom
has the label at the end, yes ? So the firmware would see the filesystem on a single partition quite happily ?

I've done this. It works ok. But = I don't run like this in production. If I write a new file, that has so= many writes to the different disks. If they all go through then life is go= od (this is what gets us to OK).

BUT, if there is a power failure or crash and only some of them m= ake it to disk, then you have a corrupt ESP and the BIOS may pick that ESP = to boot off of, booting corrupt data.

Since this is infrequently updated, you can use a safe sequen= ce to update things one partition a time, then you might lose the file enti= rely, but it will either be there and good. Or it will be gone. You can'= ;t get into a bad situation. Either you boot old or new loader and can just= quit from the boot loader if it's the old one and it can't boot. E= fi will try the next one on the list.

Here manual mirroring, if scripted, can be more reliable than= gmirror.

Warner

-pete.


--000000000000cc5789060b803e27-- From nobody Sat Dec 2 05:57:12 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ShzjD2yfYz52rhQ for ; Sat, 2 Dec 2023 05:57:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ShzjD0JWlz4XQG for ; Sat, 2 Dec 2023 05:57:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-50be4f03b06so536802e87.0 for ; Fri, 01 Dec 2023 21:57:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701496636; x=1702101436; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=SLKfOAiKLJ3SNlrYC1cJBCRxs7o+Nur9x9DWAtAAisg=; b=kpeWAQeOr8C0Qxkf3lh5GoXbViDH9CysYM9aaov3l66A0dyfsE4od8mJ4t3p+csdB5 gov+3bCNolteorjxCvuHuLhKlMLup2YbN1Pp4dj6TeR+39cifFEi4yPO36lm7BvyAHWM G+xDaU/g/CDvxmTXb4w+VRa+olpg90+CfM3m44pH7VO5l4G5Ri4bIVBJtKzI/36XOc2j MQnhWpF1QViww+7L4aHAlggkkWhhQIbI+GLQdFqlln8dcXFxysl3I+7EGfACLJFD/enG TQC1kIg/bJmqaFNtYIKNZQOvYf9a5Us7kzt5rjAvZEO7ZFc3S1eRlpNlyKdqGmpAF+Fv 4Tww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701496636; x=1702101436; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SLKfOAiKLJ3SNlrYC1cJBCRxs7o+Nur9x9DWAtAAisg=; b=FUvI+MVz9FvlwDZuXMLHDLvdssZkGHnirTf1Wob9BzQr+a3+mZQAsAW0oG9UIVsU38 eX0yoo3Sd8xPJJ/DZTYPG5zZxlwVMsWvbt+6RLIjTNOqANzRBeNIg4CLtf4rzMLUzkM1 IcmzGf0x+9DLSBaoUPmM68JwBARrFWrnwKMKWIk+rfgE768p9QdE6PxybYevSRinbFQo cHrYCqmSk9cNNM1ace25j1Py4URCBqfh/9gajDBzWQBmDYtACdNOfKIROFIYt0sf2o7K P+xUbZZ1IM9fZeZ2IdZY2NLDhZtppu+4aPkJxKtP4lwoSNM/JskjOl72s9+AmD38jwE7 GwvQ== X-Gm-Message-State: AOJu0YzHVBT0UmtAF34MgzVBv93rZrKMdm98weXrMwq12gG+p7bpS5hP 5KxpTOaiAZhpLigvVpyhF59/9pZgLTkz+oS6mcK6kdlWI+/zEojB X-Google-Smtp-Source: AGHT+IHKqeYzLt0KrJ54Wjg8T3MqJ6/WesU8ST7xJO3BhgQbHbQIIvVs7hvJOEeakb/Q4Dh6UXKEe/hEW9TQm+NnkYc= X-Received: by 2002:a05:6512:3e0c:b0:50b:ba1d:70e7 with SMTP id i12-20020a0565123e0c00b0050bba1d70e7mr1379436lfv.28.1701496636247; Fri, 01 Dec 2023 21:57:16 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <86d04457-5018-45f9-849f-eb20ed5cf380@twisted.org.uk> In-Reply-To: From: Warner Losh Date: Fri, 1 Dec 2023 22:57:12 -0700 Message-ID: Subject: Re: EFI and zfs raid mirror partial fail (14.0 and RELENG_13) To: Zaphod Beeblebrox Cc: Pete French , FreeBSD Stable ML Content-Type: multipart/alternative; boundary="000000000000972971060b808e3d" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ShzjD0JWlz4XQG --000000000000972971060b808e3d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Dec 1, 2023 at 10:34=E2=80=AFPM Zaphod Beeblebrox wrote: > It can be more straightforward to update the gmirror, however. I've done > this with UFS --- old boot, pair of UFS/GMIRROR usb sticks that then boot > to a ZFS array that the BIOS couldn't see (so UFS only contained /boot an= d > /rescue). It's easier to know that the boot is updated identically if > gmirrored. Gmirror also has tools to verify, etc. > Yes. More straight forward, not as safe. BIOS runs before FreeBSD, and doesn't use gmirror at all, so it can't know if one copy is good or not. IT has to assume that the copies are always good. If you are a single user, then the convenience is likely worth it. It's going ot be fine and if you have a power failure while updating, then you are going to be right there to cope with whatever fallout by choosing the right device to boot from if the primary is corrupted. Once you reboot FreeBSD, the gmirror will resilver (usually) and life will be good. But you have to make absolutely sure that the gmirror never degrades (which happens sometimes on crashes) so that it always will update when you write a new loader. If the mirror is degraded, it will boot the old loader if the degraded side is the primary boot device for the BIOS. If you are deploying a redundant EFI booting system for lots of machines, some of which are in the middle of nowhere without remote hands available, then you can't rely on gmirror to always be right (because it can create corrupted partitions while updating each copy that can pose problems when you lose power. And there's the broken mirror problem that has to be constantly monitored. At work, we cope with this by having lots of monitor scripts for gmirror-based system and then take corrective actions when bad things happen to a gmirror element. But for our multiple, redundant ESPes, we manually update them one at a time because we can't take a chance on the gmirror being broken. If we have a drive that's the primary boot fail read-only and we can't change the BIOS boot order, then we RMA the box (though that's rare: we can usually move the primary and arrange a different drive to be the backup booting device). When you have tens of thousands of machines, even low failure rates can cause big expenses... Though the broken mirror and the BIOS boots the wrong disk that can't be fixed problem is way more common than having gmirror break due to a crash during an upgrade (but the latter does happen). So yea, gmirror is convenient. But you have to watch it like a hawk to make sure the mirror isn't broken before you do the update. And to make sure that you can get hands on the system if an update breaks badly due to a ill-timed power failure or system panic. Warner > On Fri, Dec 1, 2023 at 7:46=E2=80=AFPM Warner Losh wrote= : > >> >> >> On Fri, Dec 1, 2023, 4:57 PM Pete French wrote: >> >>> >>> On 01/12/2023 21:53, mike tancsa wrote: >>> > Should have looked at open PRs. There is one from a while ago >>> > >>> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258987 >>> > >>> > >>> >>> Was thinking about this, and I was wondering if it would be possible to >>> make the EFI partition a gmirror. So its across all discs, mounted only >>> once, but would still boot from any of them. My understanding is geom >>> has the label at the end, yes ? So the firmware would see the filesyste= m >>> on a single partition quite happily ? >>> >> >> I've done this. It works ok. But I don't run like this in production. If >> I write a new file, that has so many writes to the different disks. If t= hey >> all go through then life is good (this is what gets us to OK). >> >> BUT, if there is a power failure or crash and only some of them make it >> to disk, then you have a corrupt ESP and the BIOS may pick that ESP to b= oot >> off of, booting corrupt data. >> >> Since this is infrequently updated, you can use a safe sequence to updat= e >> things one partition a time, then you might lose the file entirely, but = it >> will either be there and good. Or it will be gone. You can't get into a = bad >> situation. Either you boot old or new loader and can just quit from the >> boot loader if it's the old one and it can't boot. Efi will try the next >> one on the list. >> >> Here manual mirroring, if scripted, can be more reliable than gmirror. >> >> Warner >> >> -pete. >>> >>> >>> --000000000000972971060b808e3d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Dec 1, 2023 at 10:34=E2=80=AF= PM Zaphod Beeblebrox <zbeeble@gmail= .com> wrote:
It can be more straightforward to update the gmirror, = however.=C2=A0 I've done this with UFS --- old boot, pair of UFS/GMIRRO= R usb sticks that then boot to a ZFS array that the BIOS couldn't see (= so UFS only contained /boot and /rescue).=C2=A0 It's easier to know tha= t the boot is updated identically if gmirrored.=C2=A0 Gmirror also has tool= s to verify, etc.

Yes. More strai= ght forward, not as safe. BIOS runs before FreeBSD, and doesn't use gmi= rror at all, so it can't know if one copy is good or not. IT has to ass= ume that the copies are always good. If you are a single user, then the con= venience is likely worth it. It's going ot be fine and if you have a po= wer failure while updating, then you are going to be right there to cope wi= th whatever fallout by choosing the right device to boot from if the primar= y is corrupted. Once you reboot FreeBSD, the gmirror will resilver (usually= ) and life will be good. But you have to make absolutely sure that the gmir= ror never degrades (which happens sometimes on crashes) so that it always w= ill update when you write a new loader. If the mirror is degraded, it will = boot the old loader if the degraded side is the primary boot device for the= BIOS.

If you are deploying a redundant EFI bo= oting system for lots of machines, some of which are in the middle of nowhe= re without remote hands available, then you can't rely on gmirror to al= ways be right (because it can create corrupted partitions while updating ea= ch copy that can pose problems when you lose power. And there's the bro= ken mirror problem that has to be constantly monitored. At work, we cope wi= th this by having lots of monitor scripts for gmirror-based system and then= take corrective actions when bad things happen to a gmirror element.
=

But for our multiple, redundant ESPes, we manually upda= te them one at a time because we can't take a chance on the gmirror bei= ng broken. If we have a drive that's the primary boot fail read-only an= d we can't change the BIOS boot order, then we RMA the box (though that= 's rare: we can usually move the primary and arrange a different drive = to be the backup booting device). When you have tens of thousands of machin= es, even low failure rates can cause big expenses... Though the broken mirr= or and the BIOS boots the wrong disk that can't be fixed problem is way= more common than having gmirror break due to a crash during an upgrade (bu= t the latter does happen).

So yea, gmirror is = convenient. But you have to watch it like a hawk to make sure the mirror is= n't broken before you do the update. And to make sure that you can get = hands on the system if an update breaks badly due to a ill-timed power fail= ure or system panic.

Warner
=C2=A0
On Fri, = Dec 1, 2023 at 7:46=E2=80=AFPM Warner Losh <imp@bsdimp.com> wrote:


On Fri, Dec 1, 2= 023, 4:57 PM Pete French <pete@twisted.org.uk> wrote:

On 01/12/2023 21:53, mike tancsa wrote:
> Should have looked at open PRs. There is one from a while ago
>
> https://bugs.freebsd.org/b= ugzilla/show_bug.cgi?id=3D258987
>
>

Was thinking about this, and I was wondering if it would be possible to make the EFI partition a gmirror. So its across all discs, mounted only once, but would still boot from any of them. My understanding is geom
has the label at the end, yes ? So the firmware would see the filesystem on a single partition quite happily ?

I've done this. It works ok. But = I don't run like this in production. If I write a new file, that has so= many writes to the different disks. If they all go through then life is go= od (this is what gets us to OK).

BUT, if there is a power failure or crash and only some of them m= ake it to disk, then you have a corrupt ESP and the BIOS may pick that ESP = to boot off of, booting corrupt data.

Since this is infrequently updated, you can use a safe sequen= ce to update things one partition a time, then you might lose the file enti= rely, but it will either be there and good. Or it will be gone. You can'= ;t get into a bad situation. Either you boot old or new loader and can just= quit from the boot loader if it's the old one and it can't boot. E= fi will try the next one on the list.

Here manual mirroring, if scripted, can be more reliable than= gmirror.

Warner

-pete.


--000000000000972971060b808e3d-- From nobody Sat Dec 2 10:37:21 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sj5wT2K8Sz52tj4 for ; Sat, 2 Dec 2023 10:37:29 +0000 (UTC) (envelope-from thomas.e.zander@googlemail.com) Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sj5wR70N8z4CYN for ; Sat, 2 Dec 2023 10:37:27 +0000 (UTC) (envelope-from thomas.e.zander@googlemail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=googlemail.com header.s=20230601 header.b=K99D9o7J; spf=pass (mx1.freebsd.org: domain of thomas.e.zander@googlemail.com designates 2a00:1450:4864:20::32a as permitted sender) smtp.mailfrom=thomas.e.zander@googlemail.com; dmarc=none Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-40b4746ae3bso23610025e9.0 for ; Sat, 02 Dec 2023 02:37:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20230601; t=1701513445; x=1702118245; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=MRoI9JysxEQtCOYsBQ/wP7QbuOK4g46h2UQT4QYb0BY=; b=K99D9o7JjiZ6ivTnNL0gpTShWMEuOP624gNSct0TFCxVjg497ghalt6riRpvDpgr/Q QEU/+ZywFMPxT1y4vIiIreJXq1NAEtdO6NX9KvngQq5LkrRArCqgjbT9fIpJsA2mm7wU de5U9DnHcZqqEFCB9INpF58eO4wLhmkz7+RkINvFs2rErJwg9YDyQ/2akssoja0C+mhB 0G0TTH2ZoV0Zl2+4a5LhVjeHCsH8auN6tTF3gTZVi6wRnsxGLavy0sysXpczaJcdIavS 5axOk4vSb+hhLOKFinw9iq5d4yzO923UaNBUi1LvnQ1gp4aLOQhaPMRDMICaGTM+tM0M 01hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701513445; x=1702118245; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:sender:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=MRoI9JysxEQtCOYsBQ/wP7QbuOK4g46h2UQT4QYb0BY=; b=XWRYC682w7042ZKKqrDek0RxfJbkw8mWlMgbhW1HkBBYV49GD5mZCRyEHm5QKYujv1 gBD5JkT5P2qX8lvCzxMu7i2K7JBL5lI/4DGNqBekDRTlqtV1PzxSHDvZLwrSPiwhCAPj ojBxF7X//RosouldjMCTBMV78NjrI21M8kmuMejv/OB1ZhkfQ4kdqC0Mvjx1nWlgS9nE zwRa9znYHcP/BNm8rd8bNPaswdrbAKGbQnQY/KHlDyoee/sTpuXGVXfCWcIpjKVuKDAA NshEp9cBgk4SGpvTlG1M9kykyEzGwES8YpYjvL08ImNcd3r9e7+XsGoYgi1q54gdH+hy tFiA== X-Gm-Message-State: AOJu0YyH8IZ/xWD1Pcbt1E8sOLnEzpLEGmlHeahzsbhBuJZAhqemdOGe mcdgzoeDiUcP+fyE+gij21Xch7tAGoU= X-Google-Smtp-Source: AGHT+IErkHmGtQ8u8AIvFrx34E9QInoT2bw1mZrEjly+bv3udB91aXhTZJeglkEebq3RBcc6a8dU7A== X-Received: by 2002:a05:600c:458f:b0:40b:5e1e:b3bb with SMTP id r15-20020a05600c458f00b0040b5e1eb3bbmr1077151wmo.57.1701513444344; Sat, 02 Dec 2023 02:37:24 -0800 (PST) Received: from localhost (103-167-142-46.pool.kielnet.net. [46.142.167.103]) by smtp.gmail.com with ESMTPSA id d4-20020a05600c3ac400b0040b538047b4sm11671772wms.3.2023.12.02.02.37.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Dec 2023 02:37:21 -0800 (PST) Date: Sat, 2 Dec 2023 11:37:21 +0100 From: Thomas Zander To: freebsd-stable@freebsd.org Subject: Re: FreeBSD Errata Notice FreeBSD-EN-23:16.openzfs Message-ID: References: <20231201031737.DF0231B942@freefall.freebsd.org> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="vDC2tTTeH1iTISVB" Content-Disposition: inline In-Reply-To: <20231201031737.DF0231B942@freefall.freebsd.org> X-PGP-Fingerprint: B8B5 09A4 A0F5 2002 2FF1 71B5 0D76 6192 C7F7 8C63 X-Spamd-Result: default: False [-5.30 / 15.00]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; FORGED_SENDER(0.30)[riggs@freebsd.org,thomasezander@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20230601]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DOM_EQ_FROM_DOM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32a:from]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_NEQ_ENVFROM(0.00)[riggs@freebsd.org,thomasezander@gmail.com]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[googlemail.com:+]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[googlemail.com:dkim]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Sj5wR70N8z4CYN X-Spamd-Bar: ----- --vDC2tTTeH1iTISVB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, Dec 01, 2023 at 03:17:37AM +0000, FreeBSD Errata Notices wrote: > Topic: OpenZFS data corruption > ... > I. Background > II. Problem Description > III. Impact > IV. Workaround > V. Solution IMHO, what is missing between sections III and IV when the bug is about persistency (file systems, data storage, file format conversion, ...) is a subsection "Is there a way to find out which files are affected?" For eaxmple, in the case of ZFS, it would be helpful to answer "Will 'zpool scrub' find and / or repair affected files?" There will be cases (this one? I don't know. Anyone?) in which the answer is unfortunately "Nope, sorry.", but this would still be helpful information, since it would save people from browsing the net for hours, looking for an answer to that question. Thanks for considering for future errata Riggs --vDC2tTTeH1iTISVB Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEENHOllt3Sb7Zab+O4hW2O1Hx+r6UFAmVrCOAACgkQhW2O1Hx+ r6UjKQ/8Cfjiw20IbmEudd6FRfWEU4zkINNs2rd1XKs2g6upeDXi3/qfQcdlrHtD Vu/bGTBa5OohxphCiJM7yFcV6egWEtSeUVC4yfh6qux7TOd8j9Sex8t7R7yW9HLs uWYdIKv8w94rhel98kCRtDxIarHujTOYSExL5kAl4lUbucFBsruSxzSRkzSnmRzP hd5MxLmUtI50mETqMEEgG6Mp2+irgJFb0b82vpdhBO+gD9+bOf0a/huOQUx+f9rF wEEpI2B3H0CwsGaYPfEFqXRcSeZZ+eElym+/XxQh4FRmEnhJJ93cpXK1etC4aSta XvXpg3aAGh2sISfECnQxQct0pCxs6xfJNx0FUZVoQ9IOOeZW4ZMet39cEon+t3v4 tgCf0U4umk/RuXxcTi+aswye15Wd8ep0MRfyK7h2nDd/Y3o4lY0TWlwr9VtmamEV Y3BC6B9n4wy3b43fkTZnNuRccBIWMw6scH7+EJ6LPR3RsdlnJWvxFOtVGW5zhJy9 gX1OSZ5cLy97kHATRjU5loFMMrRm+japCetGi7tM6CZmodGacGkkNmODKZQJDU78 9soXsrdrfNufiQR1NVpK/0kXzF2gNbTHgNr+e7G4/ONm9VeAE6uloWi4f0IOxv5x d4WlTCPUtfUkhgiaPX/1WIXZ1A4/yl3zVA5jyd5ES9LMpuXWrWM= =rSge -----END PGP SIGNATURE----- --vDC2tTTeH1iTISVB--