From nobody Wed Jul 3 23:31:20 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WDwys5z74z5QZCK for ; Wed, 03 Jul 2024 23:31:33 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WDwys1Gkbz563n for ; Wed, 3 Jul 2024 23:31:33 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=gEVWCkaN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::531 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-6e7b121be30so11321a12.1 for ; Wed, 03 Jul 2024 16:31:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720049492; x=1720654292; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=l8UE2EgcfqqPVXD+R4ruGzA3Kfy38QtiCd8m4Q8wPbU=; b=gEVWCkaNc7ZcxuiBQGbukR1aSN0cqifph61L5VlCRNq93/xAg2HatNg8L+XIsj8KWW qmBqfuRFDoId3yeXf6oB5FRhGew24JJ4E6FkW9YxYGr2QlR7QQlQwLQNcc2dLRMQTGDX 1cHHfS+VmyF9MZSqMSbvqdAa3pD7y9wVFgqfXP39XfjMHh8l07gHZLlUBjXR3u3I4wbi C9lgPqzdD+zhg0jAmmW/pv/iui3p7NKt+qvCPpYGzt7iPhlfFqMaSEySViYbtPoRPHff YwpK0XtPOqYYQF+Q8NSDWu2SNMcdZBn/l9umFgBr0JBnZQHl6IA8VZuIqlU9VFeAWHbb PkpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720049492; x=1720654292; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l8UE2EgcfqqPVXD+R4ruGzA3Kfy38QtiCd8m4Q8wPbU=; b=HDkFUs6YKi64j76yOWM/hs8oG+DFqr4DLPWQ0C5wRKrITWc9/gF/zdogu4PV4eM6DR wOR5jn4YdTPi5rslvbCNOMidSF+4WYYr+NF8WhcEi9MsSYQP4H2+X/fwVD0ltCe8xiJw JRh2EEXM0jufZUU04z5de61FezIC7WA+HVKu9BY2KZcjCvjhVqc+JAtiFsfoT5k8GiKg LuvF1l6rfZsp7hddXBCLR6EUrdbwQw5TWe5agtu7gA4hTqF5e+UqnOEhRwdF77yKxGKK ZBZx8JSjbPgR2zMe2/nhPhzlykAQaipC1x2oFII+1oXIn+gzhyzT5NItz873zvkjbAZa z3Dw== X-Gm-Message-State: AOJu0Yxhte0gt+jEa/xo4sr+rPUxnrksRzaRtomy0TKwZe31TUgOzT3R wkaBU1ECuLWPa70zB2qndXOnzzVSPeTHqJvIvVDVTtJ/dPlcPKCFNulDkE5SY0s47NBPspKQvsv 9vqwJwHWXZnLT/XJB11LZlfWE8hNgwH4= X-Google-Smtp-Source: AGHT+IFVLB+e/krc9HEnpfPHcMBgPPtyYTqwmtdTC8auyaA2oHp4a/aNs3T2j6T5aNpSNczbBlX9PwG5IhEUu4ianfQ= X-Received: by 2002:a05:6a21:3286:b0:1be:c97f:2445 with SMTP id adf61e73a8af0-1bef6101ca6mr14274230637.22.1720049491469; Wed, 03 Jul 2024 16:31:31 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <202407021430.462EUI7s017611@gndrsh.dnsmgr.net> In-Reply-To: <202407021430.462EUI7s017611@gndrsh.dnsmgr.net> From: Rick Macklem Date: Wed, 3 Jul 2024 16:31:20 -0700 Message-ID: Subject: Re: Booting in bhyve always sets currdev to ZFS To: "Rodney W. Grimes" Cc: FreeBSD CURRENT Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::531:from] X-Rspamd-Queue-Id: 4WDwys1Gkbz563n On Tue, Jul 2, 2024 at 7:30=E2=80=AFAM Rodney W. Grimes wrote: > > > On Sun, Jun 30, 2024 at 8:23?AM Rodney W. Grimes > > wrote: > > > > > > > On Fri, Jun 28, 2024 at 6:26?PM Rick Macklem wrote: > > > > > > > > > > Hi, > > > > > > > > > > I've installed FreeBSD current in a bhvye instance. Everything > > > > > went ok, with UFS as a root partition. > > > > > Then I created a zpool in another partition... > > > > > - Now, every time I boot it I have to > > > > > OK set currdev=3Ddisk0s1a: > > > > > to get it to boot. > > > > > > > > > > What is the trick to keep ZFS from messing up the boot variables? > > > > I've been poking around, but haven't learned much. > > > > I think it is userboot (although there are so many boot programs in= /boot, > > > > I am not 100% sure?) that sets currdev=3Dzfs:example: since it sees= there is ZFS > > > > in a partition on the drive. It is not the boot partition and doesn= 't have > > > > any boot stuff in it. > > > > > > > > When I look at userboot, it appears that it always sets userboot_zf= s_found > > > > to 1 whenever userboot_zfs_probe() is called, given that there is a= ZFS > > > > partition with a pool on it. This makes extract_currdev get set to = the > > > > ZFS stuff, > > > > assuming I am reading the code correctly. > > > > What I do not understand is why I have not seen this before? > > > > (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?) > > > > > > > > It seems that userboot_zfs_probe() should check for boot files on t= he volume > > > > and not just that a pool exists on the partition, maybe? > > > > > > > > Anyhow, manually setting currdev=3Ddisk0s1a: gets around the proble= m. > > > > > > You should be able to set that in your ufs partition /boot/loader.con= f > > > file and at least then your not having to drop to the OK prompt and > > > manually entering this. > > Doesn't help, because it is not reading files from the boot ufs file sy= stem. > > (I could copy all the boot stuff over into the ZFS file system, but tha= t kinda > > defeats the purpose of having UFS as the boot fs.) > > It had to of loaded userboot from the ufs, right? I assume it is reading some boot file from UFS. Way back when I thought a primary boot was read from 8K at the begining of the partition, but that's how out of date I on this. -> Put another way, I don't know if "userboot" is the one I am seeing try to use the ZFS partition. > > What type of disk is this? gpt or mbr? It's old MBR. Although I'll assume bhyve knows other stuff, most of my old hardware only does MBR and not EFI, so I'll still in the MBR habit. > > Can you show the output of gpart show and gpart list? root@nfsv4-foo:~ # gpart show =3D> 63 67108801 vtbd0 MBR (32G) 63 1 - free - (512B) 64 67108800 1 freebsd [active] (32G) =3D> 0 67108800 vtbd0s1 BSD (32G) 0 33554432 1 freebsd-ufs (16G) 33554432 12582912 2 freebsd-swap (6.0G) 46137344 20971456 4 freebsd-zfs (10G) root@nfsv4-foo:~ # gpart list Geom name: vtbd0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 67108863 first: 63 entries: 4 scheme: MBR Providers: 1. Name: vtbd0s1 Mediasize: 34359705600 (32G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r3w3e5 efimedia: HD(1,MBR,0x90909090,0x40,0x3ffffc0) attrib: active rawtype: 165 length: 34359705600 offset: 32768 type: freebsd index: 1 end: 67108863 start: 64 Consumers: 1. Name: vtbd0 Mediasize: 34359738368 (32G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r3w3e8 Geom name: vtbd0s1 modified: false state: OK fwheads: 255 fwsectors: 63 last: 67108799 first: 0 entries: 8 scheme: BSD Providers: 1. Name: vtbd0s1a Mediasize: 17179869184 (16G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r1w1e1 rawtype: 7 length: 17179869184 offset: 0 type: freebsd-ufs index: 1 end: 33554431 start: 0 2. Name: vtbd0s1b Mediasize: 6442450944 (6.0G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r1w1e0 rawtype: 1 length: 6442450944 offset: 17179869184 type: freebsd-swap index: 2 end: 46137343 start: 33554432 3. Name: vtbd0s1d Mediasize: 10737385472 (10G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r1w1e1 rawtype: 27 length: 10737385472 offset: 23622320128 type: freebsd-zfs index: 4 end: 67108799 start: 46137344 Consumers: 1. Name: vtbd0s1 Mediasize: 34359705600 (32G) Sectorsize: 512 Stripesize: 32768 Stripeoffset: 0 Mode: r3w3e5 > > > > > For example, it always fails with: > > ERROR: cannot open /boot/lua/loader.lua: no such file or directory. > > > > When I look in stand/efi/loader/main.c, there is a function that checks > > for boot files called sanity_check_currdev(). However, userboot does no= t > > have a similar function and just seems to assume that the ZFS partition > > in the bootable one, if it finds one. > > That sounds like a wrong assumption by the code. > > > --> I am wondering if sanity_check_currdev() shoud be added to userboot > > to handle this case? > > Warner?? > > > rick > > > > > > > > There is much magic in boot code that bites, and not just in FreeBSD. > > > Often assumptions are made that "this" boot code, and "this" OS are > > > the only things on the disk(s). > > > > > > The fact that Linux uses a "apple-zfs" uuid, and FreeBSD a "FreeBSD-z= fs" > > > drove me nuts for 2 days until I found this little tid bit. > > > > > > Zpool import doesnt care, but the boot code does! > > > > > > -- > > > Rod Grimes rgrimes@fr= eebsd.org > -- > Rod Grimes rgrimes@freebs= d.org