ribe@freebsd.org> X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MW4PR11MB7032:EE_|DS7PR11MB6269:EE_ X-MS-Office365-Filtering-Correlation-Id: a28938e7-9b63-4b68-3d3a-08de3b70d27f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|10070799003; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?F2PhcVDjUuqeXqQXwKqIEfN47Jg5nP4jSRy79C0kC/Wg3Q990BXeCZfuFnj7?= =?us-ascii?Q?Ri0QMxLg2pcHWaiVTYnzTrDgtkskXxcYb4N0+a+YOnvhBi4GGV3PGF6Kw2rK?= =?us-ascii?Q?8vKS2pFEY8kAFuNnx10VkocKS8caoLarSEJCpLh9V+YJ5vg2QuCCrwS7h1fK?= =?us-ascii?Q?/z+uThOEmGdOGtfvcrJUSPUea+nRA0y/UGFWMFJbDm8tWegaGF7Oj/ZZIVTh?= =?us-ascii?Q?v1K2xHERU95LSQGfOB3jmgbxyduXtAfc2KNl01rkKwt9NHazZIW9Zi7E9owT?= =?us-ascii?Q?Rg2oFCXQ4f4QDs/OCjrc+KWGkH1z4H9P4w4CCv2/X9uh39qin+z4mcG3bEYh?= =?us-ascii?Q?t4wrrjx9AQdr5bHFZ9gLvkeAzUkP0R3LaQYHP0JPkFKAOn7Lx86BlkXVWhJ0?= =?us-ascii?Q?5ws62mcx9UqOle9ECReP8cBbXHtWIlr4pQZKtQiMUEkhgTNGzPWvzQmhM4hd?= =?us-ascii?Q?OAtsO95r8KIXd1O8I+viYfXjuSJVIZYWVp2bxGUmTjdfcruPWX2RGV/kPl7K?= =?us-ascii?Q?3HquHud5LIrc7MLCN3qEOijvt9+q78fUv8UHLqpSZvVnH4pPL4GjSfWhbQVJ?= =?us-ascii?Q?vNuqLLhjl1OHRdMfIsrA2f0pFxsXlRvbjRJj3V0F55PN9xz9Plbv023DWRZy?= =?us-ascii?Q?i0pSQ8EDmNx+fkW1cXvuLSr/n9xoEz4xeJyFUHCFG4/f3QRWdjRNRFJhlM6E?= =?us-ascii?Q?gFUVjrA1Jc5HDo274UN4B9GLXcaKgylwm/2VKHL1ripiMFvPzQwPup5Insx5?= =?us-ascii?Q?HrUbIR0p19+FVVecyA3mY8+Q1XLneETTMjAdTLOVLciy2ZCyT+XQjPeaF5nA?= =?us-ascii?Q?C5v2TXObFoWztJlfRFSHzpqY2WPXu1RWjzwUMNB7X1kt4sIayY61IxLJhj5X?= =?us-ascii?Q?MYVPO4J4Mb+QpHpK+GE06eD82jEDHmxTrV5Z08NO1GMwYY27UXa1hCgtXbpg?= =?us-ascii?Q?nTg8LTQPet2FyctkThgV4zlCxweQM4WP6tOHu/DUNHCkNJr4+c1ASbhuQ1OH?= =?us-ascii?Q?1mEzpYzluTI7bxUAxAp53q62+RqRU84Lz5TacSCEDmLJ5Kr3KyKRzLTAhYnO?= =?us-ascii?Q?SEYbkdDs1xgfmk59cSZn/zuSKNg7VyMKxJFMTCv/1zd8gL81C8Pm/E8j0wkg?= =?us-ascii?Q?IxGcKamo+ZwM/JGI8rx9t6KldjNBkYPhPOYPEitE3VPMSShVAD6BbRoYr9fe?= =?us-ascii?Q?lWsU1q8SWRUloAvYt2NuMS2hErZjxjsLt472YjvjW/aBpkc0WBu0UHc1XFQR?= =?us-ascii?Q?leeeaw5PfO+Ko35vjzJg1AQgXS8LdwZnlUFEBAWyX93YxU4VvaBzLs+bAk5o?= =?us-ascii?Q?dufVDoO8un4cr/3Uymw7gxxmZpsLAA3TE2JsclFz81QphrTadQ4XlDKaaIvi?= =?us-ascii?Q?8opJ+W3a9RNpAWM5uI1wxc/tlEErMBIPSduDrZIsgc0RzlLzH1IslNc6QHDd?= =?us-ascii?Q?g303b/nRv5gGW+SHLRaz4/4hFRRHhrCB?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW4PR11MB7032.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(10070799003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FUWtSjsrZqOIYcMY4LDAgX/HTnTnLmWPSCl3IU8rOlBbxT6SxkBiy/ALkm45?= =?us-ascii?Q?qs7YtjtFQ3Q5B9SZ2Bc1MW+nUqGUjBrc6fJ8YqZQbsWE0Prwfv9PhaB+qCPW?= =?us-ascii?Q?9V9JGuyFj5LAZuXBdRk+rGxIV5mcire8D7Y6oucyO1cPua8/AltelTEO4o+K?= =?us-ascii?Q?lZD6fXrxyHn8d7MACuDYmhBoCsbuJRPBhcV5GsWFSecqBTOFUJxD9uuY8L/a?= =?us-ascii?Q?umky2cnyso5fW+EoLIWXlUT3eSUSMb4Pgu7mssfjEG6CdRalYTmZ05hjODtC?= =?us-ascii?Q?EsamJ0c229oFK1FgBPpUcaVQQMopuEMSsR36dQ7uf3+qn9cnt75JsoNdVoOU?= =?us-ascii?Q?H8HDXeXG/Du6RROd0PStkIAWJdMBar5EdclzzfT9C2Hi1I4DUVykIVxbpT+V?= =?us-ascii?Q?o14jeE1HrAnRjzrtnC2rNSxpf1JtrIVoW8i4p8R4p3BdyymIp1POdoWBmNmo?= =?us-ascii?Q?fJ9r/tHJa3FtoAwQmvfEc7PxYTAIPeifAy+LzdjVu0/2KOb39+etiCzl7eQ+?= =?us-ascii?Q?1evrp2gwth1BA/W2607xNBjpX8yddZtm4NGNxtxU+woLjsdM2hwjPsWrXxbT?= =?us-ascii?Q?vC4b8V/wSsqImTAAl2xH4S2B11Avv38Vkqx5J6XnPFy5Qwr2giMHxF4XSFoW?= =?us-ascii?Q?+eaTiwq9VXnDcextFich6Iplwo8Bj4jB+CNfGHH+bvp8js5odsmOrQrsVHdh?= =?us-ascii?Q?QArKi6FuTtqHEgy+/xHQtpDlCpE7Q6OCYFby47Fhkig/kFZl6HgASUlz4Xui?= =?us-ascii?Q?O9y7hV26ddJDur0+P9dASbdy8OrGXgCxDswTDPuNAprAEU7dW9w943AmUTEt?= =?us-ascii?Q?oJOHIt81tlMWbteA83VPltT1LmfI8n6i5eAqm2HBQFTVjueVrxcOIVOgmOU1?= =?us-ascii?Q?g1d3QSgk0rYNWet/H4HAHc7upLKsx45ptLegaF7zTB3R73xb5bbtnNk0cKd1?= =?us-ascii?Q?78YjNvungs0yb8g4QWMXeqhv6+Qgb8vfmUgYrQgCwSL0G+cEpGTZSuhgu2i1?= =?us-ascii?Q?O9AEpXhZBNW0AXbJungYVJJ+OJ2bYKVFXclpGl9uS36FtjGEm1RT3ogt5Q3u?= =?us-ascii?Q?fEtOID9EteYNHpczz3d2VmodHGebUBOuZXajKwQ75uS4Njj4F6oU29227TvT?= =?us-ascii?Q?EbD8+kYU2ElkdVEPyx6R4l2ZOmephnlJXJo+smpiFczHV07O+6pF8jKCjEEI?= =?us-ascii?Q?djNkTesBiGB7x82j2K/RZ5xAQemNjhgKRZmD+FJUU7/fCeDESXMB4V4eVu2r?= =?us-ascii?Q?YQlDEu7LvYtYK+MV7H/sTAs5bGgXmgiOXufc3rvpOCEQpyu5r5GS9RtIUMnI?= =?us-ascii?Q?pV8rMGeLP6TFH90RqgF3EeoaRMUXmiayj0cYYsDIPoqMbG7/Lpi/WJBLu4a1?= =?us-ascii?Q?beWwrf6bUrXPsJtF8oC8THBKG7TvLoaorQFOpZuVUxsca78cUJddnAYPe3w9?= =?us-ascii?Q?dnkI9yW/Ie/ljpw0VzHvffjez/sB35mBqtiOYBmIQrPREXny/TQqEiCYlscY?= =?us-ascii?Q?5aRod/s9Je3FTttRtsKG0ogAZy8SamLDkmp+ChVwTeGimn2lpIjrj3X26V1q?= =?us-ascii?Q?akaVBUIBi+J8unbG1s9aj8uMyzmvjybwKwr5aJBdMgpKlC1pbiafbfujn4Di?= =?us-ascii?Q?BkjD6lxLLa8oRA9hhFLL3oQJn+sbQq5Cp+/Hr6raF78zv2nC6GGyjiYRtWwJ?= =?us-ascii?Q?2F/C3g=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: a28938e7-9b63-4b68-3d3a-08de3b70d27f X-MS-Exchange-CrossTenant-AuthSource: MW4PR11MB7032.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Dec 2025 00:28:09.8822 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5217e0e7-539d-4563-b1bf-7c6dcf074f91 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4bvzc7TW6xCii7d5mX1CWm1jl9EDHIBhXHtOimGjhBfjvA1sxFqF1VbUopd9NkyhTYR2ah6ZcOlBwGDNQg4mwA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB6269 X-OriginatorOrg: cs.utah.edu X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.50 / 15.00]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector10001:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[cs.utah.edu,none]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:155.97.144.0/24]; R_DKIM_ALLOW(-0.20)[utah.edu:s=UniversityOfUtah,UofUtah.onmicrosoft.com:s=selector2-UofUtah-onmicrosoft-com]; MIME_GOOD(-0.10)[text/plain]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:17055, ipnet:155.97.0.0/16, country:US]; FROM_EQ_ENVFROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_SEVEN(0.00)[7]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.93.20.103:received]; DKIM_TRACE(0.00)[utah.edu:+,UofUtah.onmicrosoft.com:+] X-Rspamd-Queue-Id: 4dV1G83knzz3tNC I've run into what appears to be a regression in vmm/bhyve in FreeBSD 15, and could use some help tracking it down. I upgraded my AMD Zen 4 machine (via source) from stable/14 (sometime post-14.3) to stable/15 (currently running -stable from today). I had some VMs (Linux guests of various flavors) that were running fine before the update. After the update, they won't start. The behavior is: the bootloader seems to work fine (eg. I can run grub-bhyve, it runs to completion and exits). However once I start bhyve itself, it just burns up a CPU (note: just one, no matter how many vCPUs) - I get nothing on serial console, nothing on the fbuf if I enable it. I've tried the following, with no change in behavior: Here are example command lines, though I've tried many variations: sudo grub-bhyve \ -m /trilogy/vms/diomira/device.map \ -M 4G \ -r hd0,msdos1 \ -d /grub \ -g grub.cfg \ diomira .... then bhyve \ -c 1 \ -m 4G \ -AHPw \ -u \ -U 4e9cecd8-d716-484c-b469-d17d00ad7ed4 \ -s 0,hostbridge \ -s 31,lpc \ -s 0:4:0,virtio-blk,/dev/zvol/trilogy/vms/diomira/disk0 \ -s 0:5:0,virtio-net,tap0,mac=58:9c:fc:03:99:50 \ -l com1,stdio \ diomira I've tried all of the following, with no change in behavior: * Leaving it for hours, never makes any progress * Different Linux kernels * Using ed2k-bhyve as the bootloader instead of grub-bhyve * Setting hw.vmm.svm.disable_npf_assist=1 * Changing the number of vCPUs * Removing all devices other than the disk * Running with and without ACPI * Building a release/15 kernel in case something got added in resent stable The best clue I have so far is that it's taking a ton of NPT exits: vcpu0 stats: vcpu total runtime 268191920934 vcpu migration across host cpus 20 total number of vm exits 36277381 vm exits due to external interrupt 28971 number of times hlt was intercepted 0 number of times %cr access was intercepted 0 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 0 number of monitor trap exits 0 number of times pause was intercepted 0 vm exits due to interrupt window opening 0 vm exits due to nmi window opening 0 number of times in/out was intercepted 0 number of times cpuid was intercepted 0 vm exits due to nested page fault 36248410 vm exits for instruction emulation 0 number of vm exits for unknown reason 0 number of times astpending at exit 2573 number of times idle requested at exit 48777 number of vm exits handled in userspace 51354 number of times rendezvous pending at exit 0 number of vm exits due to exceptions 0 number of ticks vcpu was idle 0 number of NMIs delivered to vcpu 0 number of ExtINTs delivered to vcpu 0 Resident memory 2109440 Wired memory 0 EOI without any in-service interrupt 0 error interrupts generated by vlapic 0 timer interrupts generated by vlapic 0 corrected machine check interrupts generated by vlapic 0 lvts triggered[0] 0 lvts triggered[1] 0 lvts triggered[2] 0 lvts triggered[3] 0 lvts triggered[4] 0 lvts triggered[5] 0 lvts triggered[6] 0 ipis sent from vcpu 0 ipis received by vcpu 0 VM exits during event delivery 0 Events pending at VM entry 0 VM exits due to interrupt window 0 The NPT exits keeps going up at a rate of about 150k/second. So my assumption is that nested page table faults are not getting handled correctly and it's just repeatedly faulting, likely on the same address. But at this point, I'm stuck, I don't know exactly what's failing or what might have changed from FreeBSD 14 to 15 to cause this change in behavior. If anyone has any suggestions of other things to try to track this down, I'd appreciate it! dmesg: ---<>--- Copyright (c) 1992-2025 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 15.0-STABLE stable/15-n281506-b94c971dd0cf GENERIC amd64 FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) VT(efifb): resolution 1024x768 CPU: AMD Ryzen Threadripper 7960X 24-Cores (4193.85-MHz K8-class CPU) Origin="AuthenticAMD" Id=0xa10f81 Family=0x19 Model=0x18 Stepping=1 Features=0x178bfbff Features2=0x7efa320b AMD Features=0x2e500800 AMD Features2=0x75c237ff Structured Extended Features=0xf1bf97a9 Structured Extended Features2=0x415fde Structured Extended Features3=0x10000010 XSAVE Features=0xf AMD Extended Feature Extensions ID EBX=0x79bef25f SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics real memory = 137434759168 (131068 MB) avail memory = 133773152256 (127576 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 48 CPUs FreeBSD/SMP: 1 package(s) x 4 cache groups x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key Seed random: fast provider: "Intel Secure Key Seed" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 120-151 ioapic2 irqs 88-119 ioapic3 irqs 24-55 ioapic4 irqs 56-87 Launching APs: 22 23 1 10 24 11 25 21 19 16 14 13 29 33 26 31 35 39 40 45 36 47 43 17 32 28 12 18 27 34 30 20 7 2 4 9 5 46 3 41 37 38 42 44 15 8 6 random: entropy device external interface kbd0 at kbdmux0 efirtc0: efirtc0: registered as a time-of-day clock, resolution 1.000000s smbios0: at iomem 0x6e01d000-0x6e01d017 smbios0: Entry point: v3 (64-bit), Version: 3.6 aesni0: acpi0: acpi0: Power Button (fixed) AMD-Vi: IVRS Info VAsize = 64 PAsize = 52 GVAsize = 3 flags:0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 350 Event timer "HPET2" frequency 14318180 Hz quality 350 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_ec0: port 0x62,0x66 on acpi0 pcib0: on acpi0 pci0: on pcib0 amdviiommu0 at device 0.2 on pci0 pci0: at device 1.1 (no driver attached) pci0: at device 1.2 (no driver attached) pci0: at device 1.3 (no driver attached) pci0: at device 1.4 (no driver attached) pci0: at device 1.5 (no driver attached) pci0: at device 1.6 (no driver attached) pci0: at device 1.7 (no driver attached) pci0: at device 2.1 (no driver attached) pci0: at device 2.2 (no driver attached) pci0: at device 3.1 (no driver attached) pci0: at device 3.2 (no driver attached) pci0: at device 3.3 (no driver attached) pci0: at device 3.4 (no driver attached) pci0: at device 3.5 (no driver attached) pci0: at device 3.6 (no driver attached) pci0: at device 3.7 (no driver attached) pci0: at device 4.1 (no driver attached) pci0: at device 4.2 (no driver attached) pcib1: at device 7.1 on pci0 pci1: on pcib1 xhci0: mem 0xf3600000-0xf36fffff at device 0.4 on pci1 xhci0: 64 bytes context size, 64-bit DMA xhci0: xECP capabilities usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pcib2: on acpi0 pci2: on pcib2 amdviiommu1 at device 0.2 on pci2 pcib3: at device 1.1 on pci2 pci3: on pcib3 nvme0: mem 0xf2500000-0xf2503fff at device 0.0 on pci3 pcib4: at device 1.2 on pci2 pci4: on pcib4 nvme1: mem 0xf2400000-0xf2403fff at device 0.0 on pci4 pcib5: at device 1.3 on pci2 pci5: on pcib5 ix0: port 0x3020-0x303f mem 0xf2080000-0xf20fffff,0xf2104000-0xf2107fff at device 0.0 on pci5 ix0: Using 2048 TX descriptors and 2048 RX descriptors ix0: Using 16 RX queues 16 TX queues ix0: Using MSI-X interrupts with 17 vectors ix0: allocated for 16 queues ix0: allocated for 16 rx queues ix0: Ethernet address: 98:b7:85:1f:f8:0e ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: Option ROM V1-b2960-p0 eTrack 0x800006d1 ix0: netmap queues/slots: TX 16/2048, RX 16/2048 ix1: port 0x3000-0x301f mem 0xf1f80000-0xf1ffffff,0xf2100000-0xf2103fff at device 0.1 on pci5 ix1: Using 2048 TX descriptors and 2048 RX descriptors ix1: Using 16 RX queues 16 TX queues ix1: Using MSI-X interrupts with 17 vectors ix1: allocated for 16 queues ix1: allocated for 16 rx queues ix1: Ethernet address: 98:b7:85:1f:f8:0f ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: Option ROM V1-b2960-p0 eTrack 0x800006d1 ix1: netmap queues/slots: TX 16/2048, RX 16/2048 pcib6: at device 3.1 on pci2 pci6: on pcib6 mpr0: port 0x2000-0x20ff mem 0x10020900000-0x100209fffff,0x10020800000-0x100208fffff,0xf2200000-0xf22fffff at device 0.0 on pci6 mpr0: Firmware: 06.00.16.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: fa84c pcib7: at device 7.1 on pci2 pci7: on pcib7 pci2: at device 7.2 (no driver attached) pcib8: port 0xcf8-0xcff on acpi0 pci8: on pcib8 amdviiommu2 at device 0.2 on pci8 pcib9: at device 1.1 on pci8 pci9: on pcib9 vgapci0: port 0x1000-0x10ff mem 0x10090000000-0x1009fffffff,0x80320000-0x8033ffff at device 0.0 on pci9 hdac0: mem 0x80340000-0x80343fff at device 0.1 on pci9 pci8: at device 3.1 (no driver attached) pci8: at device 3.2 (no driver attached) pci8: at device 3.3 (no driver attached) pci8: at device 3.4 (no driver attached) pci8: at device 3.5 (no driver attached) pci8: at device 3.6 (no driver attached) pci8: at device 3.7 (no driver attached) pci8: at device 4.1 (no driver attached) pci8: at device 4.2 (no driver attached) pcib10: at device 5.1 on pci8 pci10: on pcib10 pcib11: at device 5.2 on pci8 pci11: on pcib11 pcib12: at device 5.3 on pci8 pci12: on pcib12 pcib13: at device 5.4 on pci8 pci13: on pcib13 pcib14: at device 7.1 on pci8 pci14: on pcib14 xhci1: mem 0x80100000-0x801fffff at device 0.4 on pci14 xhci1: 64 bytes context size, 64-bit DMA xhci1: xECP capabilities usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 pci14: at device 0.5 (no driver attached) hdac1: mem 0x80200000-0x80207fff at device 0.7 on pci14 isab0: at device 20.3 on pci8 isa0: on isab0 pcib15: on acpi0 pci15: on pcib15 amdviiommu3 at device 0.2 on pci15 pcib16: at device 3.2 on pci15 pci16: on pcib16 pci16: at device 0.0 (no driver attached) pcib17: at device 3.3 on pci15 pci17: on pcib17 igc0: mem 0xf0c00000-0xf0cfffff,0xf0d00000-0xf0d03fff at device 0.0 on pci17 igc0: EEPROM V2.17-0 eTrack 0x80000307 igc0: Using 1024 TX descriptors and 1024 RX descriptors igc0: Using 4 RX queues 4 TX queues igc0: Using MSI-X interrupts with 5 vectors igc0: Ethernet address: 60:cf:84:d8:9c:b4 igc0: netmap queues/slots: TX 4/1024, RX 4/1024 pcib18: at device 3.4 on pci15 pci18: on pcib18 pci18: at device 0.0 (no driver attached) pcib19: at device 3.5 on pci15 pci19: on pcib19 xhci2: mem 0xf0e00000-0xf0e07fff at device 0.0 on pci19 xhci2: 32 bytes context size, 64-bit DMA xhci2: xECP capabilities usbus2 on xhci2 usbus2: 5.0Gbps Super Speed USB v3.0 pcib20: at device 3.6 on pci15 pci20: on pcib20 pcib21: at device 0.0 on pci20 pci21: on pcib21 pcib22: at device 12.0 on pci21 ACPI Warning: \134_SB.PCI1.GPPE.UP00.DP60._PRT: Return Package has no elements (empty) (20250807/nsprepkg-251) pci22: on pcib22 xhci3: mem 0xf0b00000-0xf0b07fff at device 0.0 on pci22 xhci3: 32 bytes context size, 64-bit DMA xhci3: xECP capabilities usbus3 on xhci3 usbus3: 5.0Gbps Super Speed USB v3.0 pcib23: at device 13.0 on pci21 pci23: on pcib23 ahci0: mem 0xf0a80000-0xf0a803ff at device 0.0 on pci23 ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pcib24: at device 7.1 on pci15 pci24: on pcib24 pci15: at device 7.2 (no driver attached) acpi_button0: on acpi0 acpi_syscontainer0: on acpi0 apei0: on acpi0 cpu0: on acpi0 uart: ns8250: UART FCR is broken (0x1) uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ivhd0: on acpi0 ivhd0: Flag:30 ivhd0: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 4 ivhd0: Extended features[31:0]:a2295afe,NXSup,GTSup,,IASup,GASup,PCSup> HATS = 0x2 GATS = 0x1 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd0: Extended features[62:32]:25bf732f Max PASID: 0x2f DevTblSegSup = 0x2 MarcSup = 0x0 ivhd0: supported paging level:7, will use only: 4 ivhd0: device [0xc003 - 0xfffe] config:0 ivhd0: PCI cap 0x190b640f@0x40 feature:19 ivhd1: on acpi0 ivhd1: Flag:30 ivhd1: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 4 ivhd1: Extended features[31:0]:a2295afe,NXSup,GTSup,,IASup,GASup,PCSup> HATS = 0x2 GATS = 0x1 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd1: Extended features[62:32]:25bf732f Max PASID: 0x2f DevTblSegSup = 0x2 MarcSup = 0x0 ivhd1: supported paging level:7, will use only: 4 ivhd1: device [0x8003 - 0xbffe] config:0 ivhd1: PCI cap 0x190b640f@0x40 feature:19 ivhd2: on acpi0 ivhd2: Flag:30 ivhd2: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 4 ivhd2: Extended features[31:0]:a2295afe,NXSup,GTSup,,IASup,GASup,PCSup> HATS = 0x2 GATS = 0x1 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd2: Extended features[62:32]:25bf732f Max PASID: 0x2f DevTblSegSup = 0x2 MarcSup = 0x0 ivhd2: supported paging level:7, will use only: 4 ivhd2: device [0x3 - 0x3ffe] config:0 ivhd2: device [0xff00 - 0xffff] config:0 ivhd2: PCI cap 0x190b640f@0x40 feature:19 ivhd3: on acpi0 ivhd3: Flag:30 ivhd3: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 4 ivhd3: Extended features[31:0]:a2295afe,NXSup,GTSup,,IASup,GASup,PCSup> HATS = 0x2 GATS = 0x1 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1 DualPortLogSup = 0x2 DualEventLogSup = 0x2 ivhd3: Extended features[62:32]:25bf732f Max PASID: 0x2f DevTblSegSup = 0x2 MarcSup = 0x0 ivhd3: supported paging level:7, will use only: 4 ivhd3: device [0x4003 - 0x7ffe] config:0 ivhd3: PCI cap 0x190b640f@0x40 feature:19 hwpstate0: on cpu0 cpufreq0: on cpu0 Timecounter "TSC-low" frequency 2096849600 Hz quality 1000 Timecounters tick every 1.000 msec ugen1.1: at usbus1 ugen0.1: at usbus0 ugen2.1: <(0x1b21) XHCI root HUB> at usbus2 ZFS filesystem version: 5 uhub0 on usbus0 uhub0: on usbus0 ugen3.1: at usbus3 ZFS storage pool version: features support (5000) uhub1 on usbus3 uhub1: on usbus3 uhub2 on usbus1 uhub2: on usbus1 uhub3 on usbus2 uhub3: <(0x1b21) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 mpr0: Found device ,End Device> <12.0Gbps> handle<0x0012> enclosureHandle<0x0001> slot 12 pcm0: mpr0: At enclosure level 0 and connector name (C3 ) at nid 3 on hdaa0 mpr0: Found device <881,End Device> <6.0Gbps> handle<0x0015> enclosureHandle<0x0001> slot 8 mpr0: At enclosure level 0 and connector name (C2 ) hdacc1: mpr0: Found device ,End Device> <12.0Gbps> handle<0x0014> enclosureHandle<0x0001> slot 13 at cad 0 on hdac1 hdaa1: mpr0: At enclosure level 0 and connector name (C3 ) at nid 1 on hdacc1 mpr0: Found device <881,End Device> <6.0Gbps> handle<0x0016> enclosureHandle<0x0001> slot 9 mpr0: At enclosure level 0 and connector name (C2 ) mpr0: Found device ,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0001> slot 14 mpr0: At enclosure level 0 and connector name (C3 ) pcm1: at nid 20 and 26 on hdaa1 mpr0: Found device <881,End Device> <6.0Gbps> handle<0x0017> enclosureHandle<0x0001> slot 10 mpr0: At enclosure level 0 and connector name (C2 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x0018> enclosureHandle<0x0001> slot 11 pcm2: mpr0: At enclosure level 0 and connector name (C2 ) at nid 27 and 25 on hdaa1 mpr0: Found device <881,End Device> <6.0Gbps> handle<0x0019> enclosureHandle<0x0001> slot 4 mpr0: At enclosure level 0 and connector name (C1 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x001a> enclosureHandle<0x0001> slot 0 pcm3: mpr0: At enclosure level 0 and connector name (C0 ) at nid 30 on hdaa1 mpr0: Found device <881,End Device> <6.0Gbps> handle<0x001b> enclosureHandle<0x0001> slot 5 mpr0: At enclosure level 0 and connector name (C1 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x001c> enclosureHandle<0x0001> slot 1 mpr0: At enclosure level 0 and connector name (C0 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x001d> enclosureHandle<0x0001> slot 6 mpr0: At enclosure level 0 and connector name (C1 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x001e> enclosureHandle<0x0001> slot 2 mpr0: At enclosure level 0 and connector name (C0 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x001f> enclosureHandle<0x0001> slot 7 mpr0: At enclosure level 0 and connector name (C1 ) mpr0: Found device <881,End Device> <6.0Gbps> handle<0x0020> enclosureHandle<0x0001> slot 3 mpr0: At enclosure level 0 and connector name (C0 ) mpr0: Found device <5c01,End Device> <1.5Gbps> handle<0x0011> enclosureHandle<0x0001> slot 16 mpr0: At enclosure level 0 and connector name (C3 ) nda0 at nvme0 bus 0 scbus5 target 0 lun 1 nda0: nda0: Serial Number BTNR031414DG1P0B nda0: nvme version 1.3 nda0: 976762MB (2000409264 512 byte sectors) nda1 at nvme1 bus 0 scbus6 target 0 lun 1 nda1: nda1: Serial Number BTNR0314130D1P0B nda1: nvme version 1.3 nda1: 976762MB (2000409264 512 byte sectors) ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ACS-4 ATA SATA 3.x device ada0: Serial Number S757NL0Y100358M ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada0: Command Queueing enabled ada0: 3815447MB (7814037168 512 byte sectors) ada0: quirks=0x3<4K,NCQ_TRIM_BROKEN> ada1 at ahcich1 bus 0 scbus2 target 0 lun 0 ada1: ACS-4 ATA SATA 3.x device ada1: Serial Number S757NL0Y100330N ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada1: Command Queueing enabled ada1: 3815447MB (7814037168 512 byte sectors) ada1: quirks=0x3<4K,NCQ_TRIM_BROKEN> ada2 at ahcich2 bus 0 scbus3 target 0 lun 0 ada2: ACS-4 ATA SATA 3.x device ada2: Serial Number S3Z8NB0M131938Z ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada2: Command Queueing enabled ada2: 953869MB (1953525168 512 byte sectors) ada2: quirks=0x3<4K,NCQ_TRIM_BROKEN> ada3 at ahcich3 bus 0 scbus4 target 0 lun 0 ada3: ACS-4 ATA SATA 3.x device ada3: Serial Number S3Z8NB0M123206K ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada3: Command Queueing enabled ada3: 953869MB (1953525168 512 byte sectors) ada3: quirks=0x3<4K,NCQ_TRIM_BROKEN> ses0 at mpr0 bus 0 scbus0 target 16 lun 0 ses0: Fixed Enclosure Services SPC-4 SCSI device ses0: Serial Number 500605B00DCA3730 ses0: 150.000MB/s transfers ses0: SES Device ses0: da12,pass12,da13,pass13 in Array Device Slot 1, SAS Slot: 1 phys at slot 12 ses0: phy 0: SAS device type 1 phy 0 Target ( SSP ) ses0: phy 0: parent 500605b00dca3730 addr 5000c500da929165 ses0: da8,pass8 in Array Device Slot 2, SAS Slot: 1 phys at slot 8 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca3731 addr 300605b00dca3731 ses0: da14,pass14,da15,pass15 in Array Device Slot 3, SAS Slot: 1 phys at slot 13 ses0: phy 0: SAS device type 1 phy 0 Target ( SSP ) ses0: phy 0: parent 500605b00dca3732 addr 5000c500da939f4d ses0: da9,pass9 in Array Device Slot 4, SAS Slot: 1 phys at slot 9 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca3733 addr 300605b00dca3733 ses0: da16,pass16,da17,pass17 in Array Device Slot 5, SAS Slot: 1 phys at slot 14 ses0: phy 0: SAS device type 1 phy 0 Target ( SSP ) ses0: phy 0: parent 500605b00dca3734 addr 5000c500da928881 ses0: da10,pass10 in Array Device Slot 6, SAS Slot: 1 phys at slot 10 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca3735 addr 300605b00dca3735 ses0: da11,pass11 in Array Device Slot 8, SAS Slot: 1 phys at slot 11 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca3737 addr 300605b00dca3737 ses0: da4,pass4 in Array Device Slot 9, SAS Slot: 1 phys at slot 4 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca3738 addr 300605b00dca3738 ses0: da0,pass0 in Array Device Slot 10, SAS Slot: 1 phys at slot 0 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca3739 addr 300605b00dca3739 ses0: da5,pass5 in Array Device Slot 11, SAS Slot: 1 phys at slot 5 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca373a addr 300605b00dca373a ses0: da1,pass1 in Array Device Slot 12, SAS Slot: 1 phys at slot 1 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca373b addr 300605b00dca373b ses0: da6,pass6 in Array Device Slot 13, SAS Slot: 1 phys at slot 6 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca373c addr 300605b00dca373c ses0: da2,pass2 in Array Device Slot 14, SAS Slot: 1 phys at slot 2 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca373d addr 300605b00dca373d ses0: da7,pass7 in Array Device Slot 15, SAS Slot: 1 phys at slot 7 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca373e addr 300605b00dca373e ses0: da3,pass3 in Array Device Slot 16, SAS Slot: 1 phys at slot 3 ses0: phy 0: SATA device ses0: phy 0: parent 500605b00dca373f addr 300605b00dca373f da0 at mpr0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SPC-4 SCSI device da0: Serial Number ZXA0KEPF da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 20981760MB (42970644480 512 byte sectors) da1 at mpr0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SPC-4 SCSI device da1: Serial Number ZXA0E9KT da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 20981760MB (42970644480 512 byte sectors) da2 at mpr0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SPC-4 SCSI device da2: Serial Number ZXA0BK40 da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 20981760MB (42970644480 512 byte sectors) da3 at mpr0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SPC-4 SCSI device da3: Serial Number ZXA0EDD5 da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 20981760MB (42970644480 512 byte sectors) da4 at mpr0 bus 0 scbus0 target 4 lun 0 da4: Fixed Direct Access SPC-4 SCSI device da4: Serial Number ZXA0AMBM da4: 600.000MB/s transfers da4: Command Queueing enabled da4: 20981760MB (42970644480 512 byte sectors) da5 at mpr0 bus 0 scbus0 target 5 lun 0 da5: Fixed Direct Access SPC-4 SCSI device da5: Serial Number ZXA05EMN da5: 600.000MB/s transfers da5: Command Queueing enabled da5: 20981760MB (42970644480 512 byte sectors) da6 at mpr0 bus 0 scbus0 target 6 lun 0 da6: Fixed Direct Access SPC-4 SCSI device da6: Serial Number ZXA0NBJA da6: 600.000MB/s transfers da6: Command Queueing enabled da6: 20981760MB (42970644480 512 byte sectors) da7 at mpr0 bus 0 scbus0 target 7 lun 0 da7: Fixed Direct Access SPC-4 SCSI device da7: Serial Number ZXA0KP07 da7: 600.000MB/s transfers da7: Command Queueing enabled da7: 20981760MB (42970644480 512 byte sectors) da8 at mpr0 bus 0 scbus0 target 8 lun 0 da8: Fixed Direct Access SPC-4 SCSI device da8: Serial Number ZXA0Q3VB da8: 600.000MB/s transfers da8: Command Queueing enabled da8: 20981760MB (42970644480 512 byte sectors) da9 at mpr0 bus 0 scbus0 target 9 lun 0 da9: Fixed Direct Access SPC-4 SCSI device da9: Serial Number ZXA0E2J4 da9: 600.000MB/s transfers da9: Command Queueing enabled da9: 20981760MB (42970644480 512 byte sectors) da10 at mpr0 bus 0 scbus0 target 10 lun 0 da10: Fixed Direct Access SPC-4 SCSI device da10: Serial Number ZXA0FXB0 da10: 600.000MB/s transfers da10: Command Queueing enabled da10: 20981760MB (42970644480 512 byte sectors) da11 at mpr0 bus 0 scbus0 target 11 lun 0 da11: Fixed Direct Access SPC-4 SCSI device da11: Serial Number ZXA0FNE6 da11: 600.000MB/s transfers da11: Command Queueing enabled da11: 20981760MB (42970644480 512 byte sectors) da12 at mpr0 bus 0 scbus0 target 12 lun 0 da12: Fixed Direct Access SPC-5 SCSI device da12: Serial Number ZKL2JWRD0000C2514FDE da12: 1200.000MB/s transfers da12: Command Queueing enabled da12: 6675968MB (13672382464 512 byte sectors) da14 at mpr0 bus 0 scbus0 target 13 lun 0 da14: Fixed Direct Access SPC-5 SCSI device da14: Serial Number ZKL2K4AB0000C2514EZ8 da14: 1200.000MB/s transfers da14: Command Queueing enabled da14: 6675968MB (13672382464 512 byte sectors) da16 at mpr0 bus 0 scbus0 target 14 lun 0 da16: Fixed Direct Access SPC-5 SCSI device da16: Serial Number ZKL2JWTN0000C2514FPL da16: 1200.000MB/s transfers da16: Command Queueing enabled da16: 6675968MB (13672382464 512 byte sectors) da13 at mpr0 bus 0 scbus0 target 12 lun 1 da13: Fixed Direct Access SPC-5 SCSI device da13: Serial Number ZKL2JWRD0000C2514FDE da13: 1200.000MB/s transfers da13: Command Queueing enabled da13: 6675968MB (13672382464 512 byte sectors) da15 at mpr0 bus 0 scbus0 target 13 lun 1 da15: Fixed Direct Access SPC-5 SCSI device da15: Serial Number ZKL2K4AB0000C2514EZ8 da15: 1200.000MB/s transfers da15: Command Queueing enabled da15: 6675968MB (13672382464 512 byte sectors) da17 at mpr0 bus 0 scbus0 target 14 lun 1 da17: Fixed Direct Access SPC-5 SCSI device da17: Serial Number ZKL2JWTN0000C2514FPL da17: 1200.000MB/s transfers da17: Command Queueing enabled da17: 6675968MB (13672382464 512 byte sectors) Trying to mount root from zfs:twins []... uhub3: 2 ports with 2 removable, self powered uhub2: 4 ports with 4 removable, self powered uhub0: 4 ports with 4 removable, self powered ugen0.2: at usbus0 usbhid0 on uhub0 usbhid0: on usbus0 hidbus0: on usbhid0 hkbd0: on hidbus0 kbd1 at hkbd0 usbhid1 on uhub0 usbhid1: on usbus0 hidbus1: on usbhid1 uhub1: 18 ports with 18 removable, self powered ugen3.2: at usbus3 intsmb0: at device 20.0 on pci8 smbus0: on intsmb0 acpi_wmi0: on acpi0 acpi_wmi0: Embedded MOF found acpi_wmi1: on acpi0 acpi_wmi1: Embedded MOF found acpi_wmi2: on acpi0 acpi_wmi2: Embedded MOF found ig4iic0: iomem 0xfedc3000-0xfedc3fff irq 11 on acpi0 iicbus0: on ig4iic0 ig4iic1: iomem 0xfedc4000-0xfedc4fff irq 16 on acpi0 iicbus1: on ig4iic1 ig4iic2: iomem 0xfedc6000-0xfedc6fff irq 22 on acpi0 iicbus2: on ig4iic2 ig4iic3: iomem 0xfedcb000-0xfedcbfff irq 23 on acpi0 iicbus3: on ig4iic3 iicbus3: at addr 0x40 iicbus3: at addr 0x40 hcons0: on hidbus1 hsctrl0: on hidbus1 lo0: link state changed to UP ix0: link state changed to UP bridge0: Ethernet address: 58:9c:fc:10:e9:e0 bridge0: changing name to 'vm-public'