Date: Tue, 10 Jul 2018 16:12:17 -0700 From: Cameron Berkenpas <cam@neo-zeon.de> To: freebsd-ppc@freebsd.org Subject: KVM working on Talos2/POWER9! Message-ID: <f3392131-3486-1964-35db-5b5be2a82125@neo-zeon.de>
index | next in thread | raw e-mail
Hello, FreeBSD is running in KVM on the Talos II/POWER9 with some caveats. Before doing anything, you need to append "disable_radix=yes" to the Linux kernel command line and reboot. I am able to install it by manually running: qemu-system-ppc64 -enable-kvm -m 8192 -nographic -vga none -mem-prealloc -mem-path /dev/hugepages -cdrom FreeBSD-12.0-CURRENT-powerpc-powerpc64-20180709-r336134-disc1.iso -drive file=freebsd-ppc.qcow2,format=qcow2 -smp 2 Not the most pressing issue, but during the install, I learned the hard way that the GPT partition table format doesn't work. I was able to get it running with MBR (which is the default option anyway). I haven't tried any of the other options. With GPT, it fails with the same messages as found here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211599 I am then able to run the VM manually with the following: qemu-system-ppc64 -enable-kvm -m 8192 -nographic -vga none -mem-prealloc -mem-path /dev/hugepages -drive file=freebsd-ppc.qcow2,format=qcow2 -smp 2 I haven't tried earlier snapshots of 12.0-CURRENT or 11.2 since figuring out how to get it running yet. When attempting to use KVM via libvirt, FreeBSD begins to boot, and then is unable to find the filesystem to mount root. This happens when booting off of the install disc, as well as off of the hard disk image itself even after I successfully installed running kvm manually. Here I am booting off of the virtual hard disk: http://vasteel.neo-zeon.de/~hiryu/freebsd-power9.png Here is how libvirt tries to run this VM: /usr/bin/qemu-system-ppc64 -name guest=freebsd-ppc,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-31-freebsd-ppc/master-key.aes -machine pseries-2.12,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 4096 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/31-freebsd-ppc -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid ffd4b906-7af7-48d4-8275-f990c97faaa4 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-31-freebsd-ppc/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device ahci,id=sata0,bus=pci.0,addr=0x7 -drive file=/var/lib/libvirt/images/freebsd-ppc.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-scsi0-0-0-1,readonly=on -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:34:14,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -device usb-kbd,id=input0,bus=usb.0,port=1 -device usb-mouse,id=input1,bus=usb.0,port=2 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on I see a backtrace in dmesg that doesn't always seem occur. Pastebin appears to be currently down for me, so here's the full dmesg in case other details are helpful: Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #0 r336134: Mon Jul 9 18:14:23 UTC 2018 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 powerpc gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. cpu0: IBM POWER9 revision 2.2, 2234.00 MHz cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE> cpu0: Features2 eee00000<ARCH207,HTM,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN> real memory = 8544436224 (8148 MB) avail memory = 8249282560 (7867 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs random: unblocking device. random: entropy device external interface kbd0 at kbdmux0 ofwbus0: <Open Firmware Device Tree> on nexus0 xicp0: <External Interrupt Presentation Controller> on ofwbus0 xicp0: Handling CPUs 0-15 vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0 vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781322 on vdevice0 vscsi0: Queue depth 22 commands llan0: <POWER Hypervisor Virtual Ethernet> irq 16781321 on vdevice0 llan0: Ethernet address: 00:12:34:56:00:00 uart0: <POWER Hypervisor Virtual Serial Port> irq 16781315 on vdevice0 pcib0: <RTAS Host-PCI bridge> on ofwbus0 pci0: <POWER Hypervisor PCI bus> on pcib0 cpulist0: <Open Firmware CPU Group> on ofwbus0 cpu0: <Open Firmware CPU> on cpulist0 cpu1: <Open Firmware CPU> on cpulist0 rtas0: <Run-Time Abstraction Services> on ofwbus0 rtas0: registered as a time-of-day clock, resolution 0.002000s Timecounter "timebase" frequency 512000000 Hz quality 0 Event timer "decrementer" frequency 512000000 Hz quality 1000 Timecounters tick every 1.000 msec llan0: link state changed to UP usb_needs_explore_all: no devclass cd0 at vscsi0 bus 0 scbus0 target 0 lun 8200 cd0: <QEMU QEMU CD-ROM 2.5+> Removable CD-ROM SPC-3 SCSI device cd0: 150.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present da0 at vscsi0 bus 0 scbus0 target 0 lun 0 da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device da0: 150.000MB/s transfers da0: 32768MB (67108864 512 byte sectors) SMP: AP CPU #1 launched Trying to mount root from ufs:/dev/da0s2a [rw]... WARNING: WITNESS option enabled, expect reduced performance. lock order reversal: 1st 0xe000000000c8a3b8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3916 2nd 0xc000000007521600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0xc0000000006c937c at .witness_debugger+0x90 #1 0xc0000000006cc650 at .witness_checkorder+0xa20 #2 0xc00000000064f870 at ._sx_xlock+0xbc #3 0xc000000000989f74 at .ufsdirhash_acquire+0x3c #4 0xc00000000098a6e4 at .ufsdirhash_add+0x30 #5 0xc00000000098e250 at .ufs_direnter+0x7e8 #6 0xc0000000009975ac at .ufs_rename+0xb5c #7 0xc000000000a49a88 at .VOP_RENAME_APV+0x164 #8 0xc0000000007553fc at .kern_renameat+0x3c0 #9 0xc0000000007555f4 at .sys_rename+0x2c #10 0xc000000000a2db70 at .trap+0x65c #11 0xc000000000a22824 at .powerpc_interrupt+0x29c lo0: link state changed to UP During shutdown, I saw another backtrace, this one seems to occur all the time or at least more frequently: All buffers synced. lock order reversal: 1st 0xc00000000747c9c8 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1343 2nd 0xc00000000747d248 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2572 stack backtrace: #0 0xc0000000006c937c at .witness_debugger+0x90 #1 0xc0000000006cc650 at .witness_checkorder+0xa20 #2 0xc0000000006179c8 at .lockmgr_lock_fast_path+0x1a0 #3 0xc00000000073407c at .vop_stdlock+0x30 #4 0xc000000000a4850c at .VOP_LOCK1_APV+0x158 #5 0xc00000000075cd98 at ._vn_lock+0xac #6 0xc00000000074d15c at .vget+0xac #7 0xc0000000004dfafc at .devfs_allocv+0x110 #8 0xc0000000004dcec0 at .devfs_root+0x54 #9 0xc00000000074e4c4 at .vflush+0x5e8 #10 0xc0000000004dcfd4 at .devfs_unmount+0x64 #11 0xc00000000073ea3c at .dounmount+0xaec #12 0xc000000000746570 at .unmount_or_warn+0x28 #13 0xc000000000746668 at .vfs_unmountall+0x88 #14 0xc000000000723b28 at .bufshutdown+0x308 #15 0xc000000000642ffc at .kern_reboot+0x2f0 #16 0xc000000000643de0 at .sys_reboot+0x454 #17 0xc000000000a2db70 at .trap+0x65c Hope this helps! -Cameronhelp
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f3392131-3486-1964-35db-5b5be2a82125>