Date: Mon, 29 Apr 2019 17:24:10 +1000 From: Alexey Kardashevskiy <aik@ozlabs.ru> To: Justin Hibbits <chmeeedalf@gmail.com> Cc: freebsd-ppc@freebsd.org Subject: Re: Fail to boot FreeBSD12 or 13 on POWER9 under PowerKVM Message-ID: <113863b2-3a7e-e709-252d-c7ec8c7919a0@ozlabs.ru> In-Reply-To: <20190426111659.61c4a447@titan.knownspace> References: <a7a06170-396d-4896-3915-c8a4420aec3e@ozlabs.ru> <20190426111659.61c4a447@titan.knownspace>
next in thread | previous in thread | raw e-mail | index | archive | help
On 27/04/2019 02:16, Justin Hibbits wrote: > Hi Alexey, > > On Fri, 5 Apr 2019 15:07:58 +1100 > Alexey Kardashevskiy <aik@ozlabs.ru> wrote: > >> Hi! >> >> I am trying a freebsd guest on a POWER9 (pvr=004e1201) host with >> linux+kvm (5.1.0-rc2) and qemu (upstream, 4.0) and something goes >> wrong >> - it crashes as (the full output is below): >> >> ===== >> run_interrupt_driven_hooks: still waiting after 300 seconds for >> xpt_config random: unblocking device. >> panic: run_interrupt_driven_config_hooks: waited too long >> cpuid = 0 >> time = 361 >> KDB: stack backtrace: >> 0xe000000000008660: at .kdb_backtrace+0x5c >> 0xe000000000008790: at .vpanic+0x1b4 >> 0xe000000000008850: at .panic+0x38 >> 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194 >> 0xe0000000000089e0: at .mi_startup+0x1f8 >> 0xe000000000008a80: at btext+0xc4 >> KDB: enter: panic >> [ thread pid 0 tid 100000 ] >> Stopped at .kdb_enter+0x60: ld r2, r1, 0x28 >> db> >> ===== >> >> >> The systems were installed from: >> FreeBSD-12.0-RELEASE-powerpc-powerpc64-dvd1.iso >> FreeBSD-13.0-CURRENT-powerpc-powerpc64-20190307-r344854-disc1.iso >> >> Everything by default (vt100 terminal, etc), IBM vio-scsi disk and >> cdrom, 8GB RAM, 16MB backing huge pages, the host is running in the >> hash (HPT) mode. >> >> However the exact same disk images + qemu/slof binary + qemu command >> line + linux kernel do boot on POWER8E (pvr=004b0201) and POWER8NVL >> (pvr=004c0100) to the login prompt. >> >> >> I told QEMU to enforce XICS interrupt controller mode, POWER8 >> compatibility (although it does not make sense as FreeBSD does not do >> "client-architecture-support" RTAS call), what else can I try? > > That seems pretty bizarre. Is the peripheral list on the working and > non-working the same? Yes. >> >> >> While at it, FreeBSD is aware of 004b0201 and 004e1201 but it fails to >> recognize 004c0100 (the FreeBSD guest still boots just fine): >> >> cpu0: Unknown PowerPC CPU revision 0x0100, 3259.00 MHz >> cpu0: Features c4000000<PPC32,PPC64,MMU> >> >> but in fact architecturally it behaves exactly as IBMPOWER8 (004bxxxx >> or 004d0000). > > This just needs another entry in the CPU table. I can add that > tonight. It's not entirely cosmetic, because we do make certain > decisions and set capabilities based on the CPU found. > >> >> >> >> build/qemu-aikrhel74alt-ppc64/ppc64-softmmu/qemu-system-ppc64 \ >> -nodefaults \ >> -chardev stdio,id=STDIO0,signal=off,mux=on \ >> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ >> -mon id=MON0,chardev=STDIO0,mode=readline -nographic -vga none \ >> img/freebsd12-64G.qcow2 -enable-kvm \ >> -smp 1 -mem-prealloc -mem-path qemu_hp_16M_node0 -m 8G \ >> -machine \ >> pseries,cap-hpt-max-page-size=16M,cap-cfpc=broken,max-cpu-compat=power8,ic-mode=xics >> \ >> -snapshot -bios ./slof.bin \ >> -L /home/aik/t/qemu-ppc64-bios/ \ >> -trace events=qemu_trace_events -d guest_errors \ >> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.8324 \ >> -mon chardev=SOCKET0,mode=control >> >> >> SLOF >> ********************************************************************** >> QEMU Starting Build Date = Apr 5 2019 13:01:51 >> FW Version = git-a5b428e1c1eae703 >> Press "s" to enter Open Firmware. >> >> Populating /vdevice methods >> Populating /vdevice/nvram@71000000 >> Populating /vdevice/v-scsi@71000001 >> SCSI: Looking for devices >> 8000000000000000 DISK : "QEMU QEMU HARDDISK 2.5+" >> Populating /vdevice/vty@71000110 >> Populating /pci@800000020000000 >> No NVRAM common partition, re-initializing... >> Scanning USB >> Using default console: /vdevice/vty@71000110 >> >> Welcome to Open Firmware >> >> Copyright (c) 2004, 2017 IBM Corporation All rights reserved. >> This program and the accompanying materials are made available >> under the terms of the BSD License available at >> http://www.opensource.org/licenses/bsd-license.php >> >> >> Trying to load: >> from: /vdevice/v-scsi@71000001/disk@8000000000000000 ... Successfully >> loaded >> >>>> FreeBSD/powerpc Open Firmware boot block >> Boot path: /vdevice/v-scsi@71000001/disk@8000000000000000 >> Boot loader: /boot/loader >> Boot volume: /vdevice/v-scsi@71000001/disk@8000000000000000:2 >> Consoles: Open Firmware console >> >> FreeBSD/powerpc64 Open Firmware loader, Revision 0.1 >> Memory: 8388608KB >> Booted from: /vdevice/v-scsi@71000001/disk@8000000000000000 >> >> Loading /boot/defaults/loader.conf >> /boot/kernel/kernel data=0x136c550+0x4aa1f0 >> syms=[0x8+0x165c78+0x8+0x1643cb] /boot/entropy size=0x1000 >> >> Hit [Enter] to boot immediately, or any other key for command prompt. >> Booting [/boot/kernel/kernel]... >> Kernel entry at 0x1025d0 ... >> ---<<BOOT>>--- >> Copyright (c) 1992-2018 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 12.0-RELEASE r341666 GENERIC powerpc >> gcc version 4.2.1 20070831 patched [FreeBSD] >> VT: init without driver. >> cpu0: IBM POWER9 revision 2.1, 2250.00 MHz >> cpu0: Features >> dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE> >> cpu0: Features2 >> eee00000<ARCH207,HTM,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN> >> real memory = 8544436224 (8148 MB) >> avail memory = 8258940928 (7876 MB) >> random: unblocking device. >> random: entropy device external interface >> kbd0 at kbdmux0 >> random: registering fast source PowerISA DARN random number generator >> random: fast provider: "PowerISA DARN random number generator" >> ofwbus0: <Open Firmware Device Tree> on nexus0 >> xicp0: <External Interrupt Presentation Controller> on ofwbus0 >> xicp0: Handling CPUs 0-7 >> vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0 >> uart0: <POWER Hypervisor Virtual Serial Port> irq 16781585 on vdevice0 >> vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781570 on vdevice0 >> vscsi0: Queue depth 22 commands >> pcib0: <RTAS Host-PCI bridge> on ofwbus0 >> pci0: <POWER Hypervisor PCI bus> on pcib0 >> cpulist0: <Open Firmware CPU Group> on ofwbus0 >> cpu0: <Open Firmware CPU> on cpulist0 >> rtas0: <Run-Time Abstraction Services> on ofwbus0 >> rtas0: registered as a time-of-day clock, resolution 0.002000s >> Timecounter "timebase" frequency 512000000 Hz quality 0 >> Event timer "decrementer" frequency 512000000 Hz quality 1000 >> Timecounters tick every 1.000 msec >> usb_needs_explore_all: no devclass >> run_interrupt_driven_hooks: still waiting after 60 seconds for >> xpt_config run_interrupt_driven_hooks: still waiting after 120 >> seconds for xpt_config run_interrupt_driven_hooks: still waiting >> after 180 seconds for xpt_config run_interrupt_driven_hooks: still >> waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: >> still waiting after 300 seconds for xpt_config random: unblocking >> device. panic: run_interrupt_driven_config_hooks: waited too long >> cpuid = 0 >> time = 361 >> KDB: stack backtrace: >> 0xe000000000008660: at .kdb_backtrace+0x5c >> 0xe000000000008790: at .vpanic+0x1b4 >> 0xe000000000008850: at .panic+0x38 >> 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194 >> 0xe0000000000089e0: at .mi_startup+0x1f8 >> 0xe000000000008a80: at btext+0xc4 >> KDB: enter: panic >> [ thread pid 0 tid 100000 ] >> Stopped at .kdb_enter+0x60: ld r2, r1, 0x28 >> db> >> >> >> > > It's possible FreeBSD is making a decision based on certain assumptions > of the CPU type in the VM, but I'm not sure at all. I hope someone > else might know more about the pseries FreeBSD code than I do. Well, I was kinda hoping that since boot_run_interrupt_driven_config_hooks is in the common code, it might look familiar to someone, may be. Or give some clues. Can FreeBSD start from an arbitrary memory location? I can compile the kernel, modify QEMU to load it into the guest RAM and start guest from where the kernel is loaded, and this works for Linux-PPC64 (it detects endianness and then relocates-readjusts itself) but will this work with FreeBSD? -- Alexey
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?113863b2-3a7e-e709-252d-c7ec8c7919a0>