From nobody Tue Nov 28 19:55:30 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SftVV2Fk5z52SjJ for ; Tue, 28 Nov 2023 19:55:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SftVV0YLXz4YxD for ; Tue, 28 Nov 2023 19:55:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a03a900956dso27858466b.1 for ; Tue, 28 Nov 2023 11:55:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701201342; x=1701806142; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/BtFeTNvmDJ/yRUrrr6GSWHxzprurqsmsCHS8r/JK14=; b=C+SdPd6ceJiO8lSulQjmrZv1hTUaXLb8kW/NTKO8QEqOyEoO+AU6ReH+Br8LWt+b93 RzC4Cs042UiANM41xkbtwosfsZqs94qxKeXwZ89MozI0gfJOzDqVxABq1f9j4XpF8ER+ vxbj+LxJq0LJ+H/0OA6I+4QeiKEgbtegmOJtDF4cVQBp/7gbtQqZ7AmWjKzacNRS9yo5 7yRjAxc3k4WtolZa0K8696c6t+yUFPBYOUgMKIV3PyYUqIhUTUiDvDHk7HfOX/OSG9An /aXBrd4RjrxH93wbpHrZG3kW3dPJqpJ44jMAWKNDYLATjDckoqoTB09UDcmmFXrXefzF FaVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701201342; x=1701806142; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/BtFeTNvmDJ/yRUrrr6GSWHxzprurqsmsCHS8r/JK14=; b=WFDLIqMgDC94fYOnd04zgRcESy0fR48Jf+tXoywKvwo2gvVz+bpl1Yb/CRQqEAPzAv w1MgB9SWVxgST4CrwubTJGMV9s6lqEhAXVfuqbh9s/GEIv6gK8E0rJvt/BJvV/X0LF3d n9U2W8uvngPtN8RQj5PPO6wNcHFispR5gP/xDX/59wJIiMTVBb+yhRicuHQGFu6Uez0d Bf/Cw53tIdAjXoh0jhv/3PNXv+igmtNBGbVvDphyFHiNBtIxUvgUCw5dXtEvaZUFdRsx TYub/J4dOkE09SkNSfgET/Go89yssyqlHUu05Bge1XJA74WLjAFOcAJIgnp4ek5oKGo4 /R3A== X-Gm-Message-State: AOJu0Yy+MiNhEYJT4Wko33U4B94qrj3/1HGWERXGtqiS5NXy+ad1KvtH sXwLaRBNPiGQua52Q5UXi9HkuBh5P5bTBWdCAntakQ== X-Google-Smtp-Source: AGHT+IFm4yNCHEFMOkSKE2p6iGXWvl2mdIc9uKtubs6EpqCADp888Q/eTmbM2yTkPjfACY1+sngnBaR/ramJeYmWCp8= X-Received: by 2002:a17:907:765b:b0:9e3:b158:6fb8 with SMTP id kj27-20020a170907765b00b009e3b1586fb8mr15214082ejc.32.1701201341704; Tue, 28 Nov 2023 11:55:41 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> In-Reply-To: <34a54774-004d-40de-bf0d-2985da5bc908@quip.cz> From: Warner Losh Date: Tue, 28 Nov 2023 12:55:30 -0700 Message-ID: Subject: Re: Missing disk partition devices and GPT lables in /dev/ To: Miroslav Lachman <000.fbsd@quip.cz> Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000a9fd16060b3bcd43" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SftVV0YLXz4YxD --000000000000a9fd16060b3bcd43 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 28, 2023 at 12:51=E2=80=AFPM Miroslav Lachman <000.fbsd@quip.cz= > wrote: > I have strange problem on one of my machines - the disk devices in /dev/ > are only shown as ada0 and ada1 but no partition devices, no devices in > /dev/gpt/ or /dev/gptid/, these directories are completely missing even > though there are GPT labels. > This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 > Seagate NAS 4TB, manually partitioned with gpart utility, zpool created > and then all the date migrated by zfs send and zfs receive from old 2TB > disks. Everything works, system is running but device nodes in /dev/ are > missing. I noticed this only when I want to create gmirrored swap from > ada0p4 and ada1p4 but got "No such file or directory." > > There are only diskid style devices in /dev/diskid/ > > gpart show -l shows correct GPT labels for partitions > > # gpart show -l > =3D> 40 7814037088 diskid/DISK-Z3010G8V GPT (3.6T) > 40 1024 1 bootG8V (512K) > 1064 40960 2 efiG8V (20M) > 42024 83886080 3 diskG8Vsys (40G) > 83928104 20971520 4 swapG8V (10G) > 104899624 7707033600 5 diskG8Vtank0 (3.6T) > 7811933224 2103904 - free - (1.0G) > > =3D> 40 7814037088 diskid/DISK-Z302YC90 GPT (3.6T) > 40 1024 1 bootC90 (512K) > 1064 40960 2 efiC90 (20M) > 42024 83886080 3 diskC90sys (40G) > 83928104 20971520 4 swapC90 (10G) > 104899624 7707033600 5 diskC90tank0 (3.6T) > 7811933224 2103904 - free - (1.0G) > > > But gpart show -p does not show ada0p1 ada0p2 ..ada0pN, it shows diskid > style entries > > # gpart show -p > =3D> 40 7814037088 diskid/DISK-Z3010G8V GPT (3.6T) > 40 1024 diskid/DISK-Z3010G8Vp1 freebsd-boot (512K) > 1064 40960 diskid/DISK-Z3010G8Vp2 efi (20M) > 42024 83886080 diskid/DISK-Z3010G8Vp3 freebsd-zfs (40G) > 83928104 20971520 diskid/DISK-Z3010G8Vp4 freebsd-swap (10G) > 104899624 7707033600 diskid/DISK-Z3010G8Vp5 freebsd-zfs (3.6T) > 7811933224 2103904 - free - (1.0G) > > =3D> 40 7814037088 diskid/DISK-Z302YC90 GPT (3.6T) > 40 1024 diskid/DISK-Z302YC90p1 freebsd-boot (512K) > 1064 40960 diskid/DISK-Z302YC90p2 efi (20M) > 42024 83886080 diskid/DISK-Z302YC90p3 freebsd-zfs (40G) > 83928104 20971520 diskid/DISK-Z302YC90p4 freebsd-swap (10G) > 104899624 7707033600 diskid/DISK-Z302YC90p5 freebsd-zfs (3.6T) > 7811933224 2103904 - free - (1.0G) > > > # gpart list | grep label > label: bootG8V > label: efiG8V > label: diskG8Vsys > label: swapG8V > label: diskG8Vtank0 > label: bootC90 > label: efiC90 > label: diskC90sys > label: swapC90 > label: diskC90tank0 > > > # ll /dev/gpt/ > ls: /dev/gpt/: No such file or directory > > > # ll /dev/ada* > crw-r----- 1 root operator 0x73 Nov 27 00:00 /dev/ada0 > crw-r----- 1 root operator 0x81 Nov 27 00:00 /dev/ada1 > > > Sysctl seems normal to me. > > # sysctl kern.geom.label > kern.geom.label.disk_ident.enable: 1 > kern.geom.label.gptid.enable: 1 > kern.geom.label.gpt.enable: 1 > kern.geom.label.ufs.enable: 1 > kern.geom.label.ufsid.enable: 1 > kern.geom.label.reiserfs.enable: 1 > kern.geom.label.ntfs.enable: 1 > kern.geom.label.msdosfs.enable: 1 > kern.geom.label.iso9660.enable: 1 > kern.geom.label.flashmap.enable: 1 > kern.geom.label.ext2fs.enable: 1 > kern.geom.label.debug: 0 > > > This is the only way I can access disk partitions > > # ll /dev/diskid/ > total 0 > crw-r----- 1 root operator 0x7f Nov 27 00:00 DISK-Z3010G8V > crw-r----- 1 root operator 0x99 Nov 27 00:00 DISK-Z3010G8Vp1 > crw-r----- 1 root operator 0x9b Nov 27 00:00 DISK-Z3010G8Vp2 > crw-r----- 1 root operator 0x9d Nov 27 00:00 DISK-Z3010G8Vp3 > crw-r----- 1 root operator 0x9f Nov 27 00:00 DISK-Z3010G8Vp4 > crw-r----- 1 root operator 0xa1 Nov 27 00:00 DISK-Z3010G8Vp5 > crw-r----- 1 root operator 0xad Nov 27 00:00 DISK-Z302YC90 > crw-r----- 1 root operator 0xc5 Nov 27 00:00 DISK-Z302YC90p1 > crw-r----- 1 root operator 0xc7 Nov 27 00:00 DISK-Z302YC90p2 > crw-r----- 1 root operator 0xc9 Nov 27 00:00 DISK-Z302YC90p3 > crw-r----- 1 root operator 0xcb Nov 27 00:00 DISK-Z302YC90p4 > crw-r----- 1 root operator 0xcd Nov 27 00:00 DISK-Z302YC90p5 > > > I created two pools (sys and tank1) with adaN devices when there were 4 > disks and the system was booted from USB flash drive with FreeBSD 13.2; > current ada0 was ada2 and current ada1 was ada3. > > These commands were used > > # zpool create -m none sys mirror ada2p3 ada3p3 > > # zpool create -m none tank1 mirror ada2p5 ada3p5 > > But now, when the system is booted of these disks, they are reported > only by diskids. > > # zpool list -v > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ > FRAG CAP DEDUP HEALTH ALTROOT > sys 39.5G 15.9G 23.6G - - > 12% 40% 1.00x ONLINE - > mirror-0 39.5G 15.9G 23.6G - - > 12% 40.2% - ONLINE > diskid/DISK-Z3010G8Vp3 40G - - - - > - - - ONLINE > diskid/DISK-Z302YC90p3 40G - - - - > - - - ONLINE > tank1 3.58T 1.22T 2.36T - - > 0% 34% 1.00x ONLINE - > mirror-0 3.58T 1.22T 2.36T - - > 0% 34.0% - ONLINE > diskid/DISK-Z3010G8Vp5 3.59T - - - - > - - - ONLINE > diskid/DISK-Z302YC90p5 3.59T - - - - > - - - ONLINE > > > I did similar setup and migration many times and never seen this problem. > > What can be wrong with this system? > The problem is a geom design point. When you have multiple aliases that you can open, the others are removed from the filesystem view when one is opened. It looks like from the above that the mirror-0 was created with the diskid, so that's the device that zfs opens, leaving the ada2 devices removed. Warner --000000000000a9fd16060b3bcd43 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Nov 28, 2023 at 12:51=E2=80= =AFPM Miroslav Lachman <000.fbsd@qui= p.cz> wrote:
I have strange problem on one of my machines - the disk devices in /dev= /
are only shown as ada0 and ada1 but no partition devices, no devices in /dev/gpt/ or /dev/gptid/, these directories are completely missing even though there are GPT labels.
This is FreeBSD 13.2-RELEASE-p5 amd64 on HP Microserver Gen8. Disk are 2 Seagate NAS 4TB, manually partitioned with gpart utility, zpool created and then all the date migrated by zfs send and zfs receive from old 2TB disks. Everything works, system is running but device nodes in /dev/ are missing. I noticed this only when I want to create gmirrored swap from
ada0p4 and ada1p4 but got "No such file or directory."

There are only diskid style devices in /dev/diskid/

gpart show -l shows correct GPT labels for partitions

# gpart show -l
=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 diskid/DISK-Z3= 010G8V=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A01=C2=A0 bootG8V=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02= =C2=A0 efiG8V=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 diskG8Vsys= =C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04=C2=A0 swapG8V=C2=A0 (= 10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05=C2=A0 diskG8Vtank0=C2=A0 (3.6T)<= br> =C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -=C2=A0 = (1.0G)

=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 diskid/DISK-Z3= 02YC90=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A01=C2=A0 bootC90=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02= =C2=A0 efiC90=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 diskC90sys= =C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04=C2=A0 swapC90=C2=A0 (= 10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05=C2=A0 diskC90tank0=C2=A0 (3.6T)<= br> =C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -=C2=A0 = (1.0G)


But gpart show -p does not show ada0p1 ada0p2 ..ada0pN, it shows diskid style entries

# gpart show -p
=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 =C2=A0 diskid/= DISK-Z3010G8V=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 diskid/DISK-Z3010G8Vp1=C2=A0 freebsd-boot=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 diskid/DISK-Z3010G8Vp2=C2=A0 efi=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 diskid/DISK-Z= 3010G8Vp3=C2=A0 freebsd-zfs=C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 diskid/DISK-Z3010G= 8Vp4=C2=A0 freebsd-swap=C2=A0 (10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 diskid/DISK-Z3010G8Vp5=C2=A0= freebsd-zfs=C2=A0 (3.6T)
=C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -= =C2=A0 (1.0G)

=3D>=C2=A0 =C2=A0 =C2=A0 =C2=A0 40=C2=A0 7814037088=C2=A0 =C2=A0 diskid/= DISK-Z302YC90=C2=A0 GPT=C2=A0 (3.6T)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A040=C2=A0 =C2=A0 =C2=A0 =C2=A0 1024= =C2=A0 diskid/DISK-Z302YC90p1=C2=A0 freebsd-boot=C2=A0 (512K)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01064=C2=A0 =C2=A0 =C2=A0 =C2=A040960=C2= =A0 diskid/DISK-Z302YC90p2=C2=A0 efi=C2=A0 (20M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 42024=C2=A0 =C2=A0 83886080=C2=A0 diskid/DISK-Z= 302YC90p3=C2=A0 freebsd-zfs=C2=A0 (40G)
=C2=A0 =C2=A0 =C2=A083928104=C2=A0 =C2=A0 20971520=C2=A0 diskid/DISK-Z302YC= 90p4=C2=A0 freebsd-swap=C2=A0 (10G)
=C2=A0 =C2=A0 104899624=C2=A0 7707033600=C2=A0 diskid/DISK-Z302YC90p5=C2=A0= freebsd-zfs=C2=A0 (3.6T)
=C2=A0 =C2=A07811933224=C2=A0 =C2=A0 =C2=A02103904=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - free -= =C2=A0 (1.0G)


# gpart list | grep label
=C2=A0 =C2=A0 label: bootG8V
=C2=A0 =C2=A0 label: efiG8V
=C2=A0 =C2=A0 label: diskG8Vsys
=C2=A0 =C2=A0 label: swapG8V
=C2=A0 =C2=A0 label: diskG8Vtank0
=C2=A0 =C2=A0 label: bootC90
=C2=A0 =C2=A0 label: efiC90
=C2=A0 =C2=A0 label: diskC90sys
=C2=A0 =C2=A0 label: swapC90
=C2=A0 =C2=A0 label: diskC90tank0


# ll /dev/gpt/
ls: /dev/gpt/: No such file or directory


# ll /dev/ada*
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x73 Nov 27 00:00 /dev/a= da0
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x81 Nov 27 00:00 /dev/a= da1


Sysctl seems normal to me.

# sysctl kern.geom.label
kern.geom.label.disk_ident.enable: 1
kern.geom.label.gptid.enable: 1
kern.geom.label.gpt.enable: 1
kern.geom.label.ufs.enable: 1
kern.geom.label.ufsid.enable: 1
kern.geom.label.reiserfs.enable: 1
kern.geom.label.ntfs.enable: 1
kern.geom.label.msdosfs.enable: 1
kern.geom.label.iso9660.enable: 1
kern.geom.label.flashmap.enable: 1
kern.geom.label.ext2fs.enable: 1
kern.geom.label.debug: 0


This is the only way I can access disk partitions

# ll /dev/diskid/
total 0
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x7f Nov 27 00:00 DISK-Z= 3010G8V
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x99 Nov 27 00:00 DISK-Z= 3010G8Vp1
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x9b Nov 27 00:00 DISK-Z= 3010G8Vp2
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x9d Nov 27 00:00 DISK-Z= 3010G8Vp3
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00x9f Nov 27 00:00 DISK-Z= 3010G8Vp4
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xa1 Nov 27 00:00 DISK-Z= 3010G8Vp5
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xad Nov 27 00:00 DISK-Z= 302YC90
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xc5 Nov 27 00:00 DISK-Z= 302YC90p1
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xc7 Nov 27 00:00 DISK-Z= 302YC90p2
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xc9 Nov 27 00:00 DISK-Z= 302YC90p3
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xcb Nov 27 00:00 DISK-Z= 302YC90p4
crw-r-----=C2=A0 1 root=C2=A0 operator=C2=A0 =C2=A00xcd Nov 27 00:00 DISK-Z= 302YC90p5


I created two pools (sys and tank1) with adaN devices when there were 4 disks and the system was booted from USB flash drive with FreeBSD 13.2; current ada0 was ada2 and current ada1 was ada3.

These commands were used

# zpool create -m none sys mirror ada2p3 ada3p3

# zpool create -m none tank1 mirror ada2p5 ada3p5

But now, when the system is booted of these disks, they are reported
only by diskids.

# zpool list -v
NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2=A0 CKPOINT=C2=A0 E= XPANDSZ
FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP=C2=A0 =C2=A0 HEALTH=C2=A0 ALTROOT
sys=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A039.5G=C2=A0 15.9G=C2=A0 23.6G=C2=A0 =C2=A0 =C2=A0 =C2= =A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
12%=C2=A0 =C2=A0 40%=C2=A0 1.00x=C2=A0 =C2=A0 ONLINE=C2=A0 -
=C2=A0 =C2=A0mirror-0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 39.5G=C2=A0 15.9G=C2=A0 23.6G=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0-
12%=C2=A0 40.2%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z3010G8Vp3=C2=A0 =C2=A0 40G=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z302YC90p3=C2=A0 =C2=A0 40G=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
tank1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A03.58T=C2=A0 1.22T=C2=A0 2.36T=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
0%=C2=A0 =C2=A0 34%=C2=A0 1.00x=C2=A0 =C2=A0 ONLINE=C2=A0 -
=C2=A0 =C2=A0mirror-0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 3.58T=C2=A0 1.22T=C2=A0 2.36T=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0-
0%=C2=A0 34.0%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z3010G8Vp5=C2=A0 3.59T=C2=A0 =C2=A0 =C2=A0 = -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE
=C2=A0 =C2=A0 =C2=A0diskid/DISK-Z302YC90p5=C2=A0 3.59T=C2=A0 =C2=A0 =C2=A0 = -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0-
-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE


I did similar setup and migration many times and never seen this problem.
What can be wrong with this system?

The= problem is a geom design point. When you have multiple aliases that you ca= n open, the others are removed from the filesystem view when one is opened.= It looks like from the above that the mirror-0 was created with the diskid= , so that's the device that zfs opens, leaving the ada2 devices removed= .

Warner
--000000000000a9fd16060b3bcd43--