From owner-freebsd-virtualization@freebsd.org Mon Feb 11 13:32:11 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8203614D73DE for ; Mon, 11 Feb 2019 13:32:11 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E651B6D851 for ; Mon, 11 Feb 2019 13:32:10 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: by mailman.ysv.freebsd.org (Postfix) id 9DEF814D73DC; Mon, 11 Feb 2019 13:32:10 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78E3E14D73DB for ; Mon, 11 Feb 2019 13:32:10 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mx1.cksoft.de (mx1.cksoft.de [IPv6:2001:67c:24f8:1::25:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.cksoft.de", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D01536D84E for ; Mon, 11 Feb 2019 13:32:06 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from m.cksoft.de (m.cksoft.de [IPv6:2001:67c:24f8:2003::25:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.cksoft.de (Postfix) with ESMTPSA id AED291E9EB9 for ; Mon, 11 Feb 2019 14:32:03 +0100 (CET) Received: from amavisfra1.cksoft.de (unknown [IPv6:2001:67c:24f8:2003::25:a1]) by m.cksoft.de (Postfix) with ESMTP id CBB4163017 for ; Mon, 11 Feb 2019 14:32:02 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from m.cksoft.de ([IPv6:2001:67c:24f8:2003::25:3]) by amavisfra1.cksoft.de (amavisfra1.cksoft.de [IPv6:2001:67c:24f8:2003::25:a1]) (amavisd-new, port 10051) with ESMTP id tuHrVz-NIubL for ; Mon, 11 Feb 2019 14:32:01 +0100 (CET) Received: from nocfra1.cksoft.de (nocfra1.cksoft.de [IPv6:2001:67c:24f8:2001::53:1]) by m.cksoft.de (Postfix) with ESMTP id 90D5863001 for ; Mon, 11 Feb 2019 14:32:01 +0100 (CET) Received: by nocfra1.cksoft.de (Postfix, from userid 1000) id 0238313D6E; Mon, 11 Feb 2019 14:32:00 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nocfra1.cksoft.de (Postfix) with ESMTP id F282D139EB for ; Mon, 11 Feb 2019 14:32:00 +0100 (CET) Date: Mon, 11 Feb 2019 14:32:00 +0100 (CET) From: Christian Kratzer X-X-Sender: ck@nocfra1.cksoft.de Reply-To: Christian Kratzer To: virtualization@FreeBSD.org Subject: 12.0-RELEASE und kvm/qemu using on AMD EPYC Message-ID: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) X-NCC-RegID: de.cksoft X-Spammer-Kill-Ratio: 75% MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2019 13:32:11 -0000 Hi, I am running freebsd vm on debian 10 buster with libvirt/kvm/qemu. I have several kvm hosts in the cluster. Some with various intel xeon and others with AMD EPYC 7301 cpu. FreeBSD vms upto 11.2-RELEASE-p9 boo fine on all systems when passing through the host cpu using following libvirt xml FreeBSD 12.0-RELEASE and later hang after boot when swithcing to usermode in start_init: trying /sbin/init Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on Intel CPU Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE-p3 GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT(vga): text 80x25 CPU: QEMU Virtual CPU version 2.1.0 (2400.13-MHz K8-class CPU) Origin="GenuineIntel" Id=0x663 Family=0x6 Model=0x6 Stepping=3 Features=0x783fbfd Features2=0x80a02001 AMD Features=0x20100800 AMD Features2=0x1 Hypervisor: Origin = "KVMKVMKVM" real memory = 1073741824 (1024 MB) avail memory = 1000058880 (953 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 4 package(s) x 1 core(s) ... Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on the qemu virtual cpu Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE-p3 GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT(vga): text 80x25 CPU: QEMU Virtual CPU version 2.1.0 (2200.06-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x663 Family=0x6 Model=0x6 Stepping=3 Features=0x783fbfd Features2=0x80a02001 AMD Features=0x20100800 AMD Features2=0x65 SVM: NAsids=16 Hypervisor: Origin = "KVMKVMKVM" real memory = 4294967296 (4096 MB) avail memory = 4099080192 (3909 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 4 package(s) x 1 core(s) Following is dmesg from a succesfull boot of 11.2-RELEASE using host-model on AMD EPYC Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5 15:30:36 UTC 2019 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) VT(vga): text 80x25 CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f12 Family=0x17 Model=0x1 Stepping=2 Features=0x783fbff Features2=0xfff83203 AMD Features=0x2e500800 AMD Features2=0x8003f7 Structured Extended Features=0x201c01ab XSAVE Features=0x7 AMD Extended Feature Extensions ID EBX=0x2001000 SVM: NAsids=16 Hypervisor: Origin = "KVMKVMKVM" real memory = 4294967296 (4096 MB) avail memory = 4088360960 (3898 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 4 package(s) ... And finally following is the failing verbose boot of 12.0-RELEASE on AMD-EPYC Table 'FACP' at 0xbffe104e Table 'APIC' at 0xbffe10c2 APIC: Found table at 0xbffe10c2 APIC: Using the MADT enumerator. Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE-p3 GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) Table 'FACP' at 0xbffe104e Table 'APIC' at 0xbffe10c2 Table 'HPET' at 0xbffe1152 ACPI: No SRAT table found PPIM 0: PA=0xb8000, VA=0xffffffff82810000, size=0x8000, mode=0 pmap: large map 8 PML4 slots (4096 Gb) VT(vga): text 80x25 Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff8263e000. Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff826471d0. Table 'FACP' at 0xbffe104e FACP: Found table at 0xbffe104e Calibrating TSC clock ... TSC clock: 2200048246 Hz CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f12 Family=0x17 Model=0x1 Stepping=2 Features=0x783fbff Features2=0xfff83203 AMD Features=0x2e500800 AMD Features2=0x8003f7 Structured Extended Features=0x209c01ab XSAVE Features=0x7 AMD Extended Feature Extensions ID EBX=0x2001000 SVM: Features=0x0 Revision=1, ASIDs=16 L1 2MB data TLB: 255 entries, 1-way associative L1 2MB instruction TLB: 255 entries, 1-way associative L1 4KB data TLB: 255 entries, 1-way associative L1 4KB instruction TLB: 255 entries, 1-way associative L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L2 2MB unified TLB: 0 entries, disabled/not present L2 4KB data TLB: 512 entries, 4-way associative L2 4KB instruction TLB: 512 entries, 4-way associative L2 unified cache: 512 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative Hypervisor: Origin = "KVMKVMKVM" real memory = 4294967296 (4096 MB) Physical memory chunk(s): 0x0000000000001000 - 0x000000000009efff, 647168 bytes (158 pages) 0x0000000000103000 - 0x00000000001fffff, 1036288 bytes (253 pages) 0x0000000002800000 - 0x00000000b7d44fff, 3042201600 bytes (742725 pages) 0x0000000100000000 - 0x000000013ffe7fff, 1073643520 bytes (262120 pages) avail memory = 4099080192 (3909 MB) Table 'FACP' at 0xbffe104e Table 'APIC' at 0xbffe10c2 Table 'HPET' at 0xbffe1152 ACPI: No DMAR table found ... ... ... ata0: stat0=0x00 err=0x00 lsb=0x00 msb=0x00 ata0: stat1=0x00 err=0x00 lsb=0x00 msb=0x00 ata0: reset tp2 stat0=00 stat1=00 devices=0x0 ata1: reset tp1 mask=03 ostat0=00 ostat1=00 ata1: stat0=0x00 err=0x00 lsb=0x00 msb=0x00 ata1: stat1=0x00 err=0x00 lsb=0x00 msb=0x00 ata1: reset tp2 stat0=00 stat1=00 devices=0x0 Trying to mount root from ufs:/dev/vtbd0p2 [rw]... atrtc0: providing initial system time start_init: trying /sbin/init uhub0: 2 ports with 2 removable, self powered After that the system freezes. I can provide more complete information if somebody tells me what would be helpfull. How should I go about debugging this ??? Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/