From owner-freebsd-virtualization@freebsd.org Mon Feb 11 22:08:02 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B70C14E53E4 for ; Mon, 11 Feb 2019 22:08:02 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 716F58C100 for ; Mon, 11 Feb 2019 22:08:01 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: by mailman.ysv.freebsd.org (Postfix) id 2A84414E53D0; Mon, 11 Feb 2019 22:08:01 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E278114E53C9 for ; Mon, 11 Feb 2019 22:08:00 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [69.87.218.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7585A8C0FD for ; Mon, 11 Feb 2019 22:08:00 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from [10.3.135.13] (50-207-240-162-static.hfc.comcastbusiness.net [50.207.240.162]) (authenticated bits=0) by webmail5.jnielsen.net (8.15.2/8.15.2) with ESMTPSA id x1BM0N6V003891 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Feb 2019 15:00:27 -0700 (MST) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host 50-207-240-162-static.hfc.comcastbusiness.net [50.207.240.162] claimed to be [10.3.135.13] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: 12.0-RELEASE und kvm/qemu using on AMD EPYC From: John Nielsen In-Reply-To: Date: Mon, 11 Feb 2019 15:00:22 -0700 Cc: "virtualization@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Christian Kratzer X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 7585A8C0FD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2019 22:08:02 -0000 > On Feb 11, 2019, at 6:32 AM, Christian Kratzer = wrote: >=20 > I am running freebsd vm on debian 10 buster with libvirt/kvm/qemu. >=20 > I have several kvm hosts in the cluster. Some with various intel xeon = and others with AMD EPYC 7301 cpu. >=20 > FreeBSD vms upto 11.2-RELEASE-p9 boo fine on all systems when passing = through the host cpu using following libvirt xml >=20 > > > Probably not the same issue, but this sounds similar to this bug I = reported a few years ago: https://bugs.launchpad.net/qemu/+bug/1329956 It's just as likely to be a bug in Qemu or KVM as it is in FreeBSD IMO. = Maybe you can start by determining which CPU feature or features = trigger(s) the issue. You'll have to hand-roll either some libvirt XML = or qemu command lines to do it. Assuming you want to stick with XML, = first grab the CPU model and features list from `virsh capabilities`. = Then start with just the model without any extra features (using AMD = hardware I have access to as an example, replace "Opteron_G3" as = appropriate): Opteron_G3 If that works, then add the other features a few at a time until you = break it. Here's an example feature list from my same hardware. Opteron_G3 Once you identify the feature or features that cause things to break, = you can report back here, look for open bugs in Qemu or KVM regarding = those features, and/or open new bugs. > FreeBSD 12.0-RELEASE and later hang after boot when swithcing to = usermode in start_init: trying /sbin/init >=20 > Following is dmesg from a succesfull boot of 12.0-RELEASE using = host-model on Intel CPU >=20 > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, = 1993, 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) = (based on LLVM 6.0.1) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.0 (2400.13-MHz K8-class CPU) > Origin=3D"GenuineIntel" Id=3D0x663 Family=3D0x6 Model=3D0x6 = Stepping=3D3 > = Features=3D0x783fbfd > Features2=3D0x80a02001 > AMD Features=3D0x20100800 > AMD Features2=3D0x1 > Hypervisor: Origin =3D "KVMKVMKVM" > real memory =3D 1073741824 (1024 MB) > avail memory =3D 1000058880 (953 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) x 1 core(s) > ... >=20 > Following is dmesg from a succesfull boot of 12.0-RELEASE using = host-model on the qemu virtual cpu >=20 >=20 > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, = 1993, 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) = (based on LLVM 6.0.1) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.0 (2200.06-MHz K8-class CPU) > Origin=3D"AuthenticAMD" Id=3D0x663 Family=3D0x6 Model=3D0x6 = Stepping=3D3 > = Features=3D0x783fbfd > Features2=3D0x80a02001 > AMD Features=3D0x20100800 > AMD Features2=3D0x65 > SVM: NAsids=3D16 > Hypervisor: Origin =3D "KVMKVMKVM" > real memory =3D 4294967296 (4096 MB) > avail memory =3D 4099080192 (3909 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) x 1 core(s) >=20 > Following is dmesg from a succesfull boot of 11.2-RELEASE using = host-model on AMD EPYC >=20 > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, = 1993, 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5 15:30:36 UTC 2019 > = root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) = (based on LLVM 6.0.0) > VT(vga): text 80x25 > CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) > Origin=3D"AuthenticAMD" Id=3D0x800f12 Family=3D0x17 = Model=3D0x1 Stepping=3D2 > = Features=3D0x783fbff > = Features2=3D0xfff83203 > AMD = Features=3D0x2e500800 > AMD = Features2=3D0x8003f7 > Structured Extended = Features=3D0x201c01ab > XSAVE Features=3D0x7 > AMD Extended Feature Extensions ID EBX=3D0x2001000 > SVM: NAsids=3D16 > Hypervisor: Origin =3D "KVMKVMKVM" > real memory =3D 4294967296 (4096 MB) > avail memory =3D 4088360960 (3898 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) > ... >=20 >=20 > And finally following is the failing verbose boot of 12.0-RELEASE on = AMD-EPYC >=20 > Table 'FACP' at 0xbffe104e > Table 'APIC' at 0xbffe10c2 > APIC: Found table at 0xbffe10c2 > APIC: Using the MADT enumerator. > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, = 1993, 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) = (based on LLVM 6.0.1) > Table 'FACP' at 0xbffe104e > Table 'APIC' at 0xbffe10c2 > Table 'HPET' at 0xbffe1152 > ACPI: No SRAT table found > PPIM 0: PA=3D0xb8000, VA=3D0xffffffff82810000, size=3D0x8000, = mode=3D0 > pmap: large map 8 PML4 slots (4096 Gb) > VT(vga): text 80x25 > Preloaded elf kernel "/boot/kernel/kernel" at = 0xffffffff8263e000. > Preloaded boot_entropy_cache "/boot/entropy" at = 0xffffffff826471d0. > Table 'FACP' at 0xbffe104e > FACP: Found table at 0xbffe104e > Calibrating TSC clock ... TSC clock: 2200048246 Hz > CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) > Origin=3D"AuthenticAMD" Id=3D0x800f12 Family=3D0x17 = Model=3D0x1 Stepping=3D2 > = Features=3D0x783fbff > = Features2=3D0xfff83203 > AMD = Features=3D0x2e500800 > AMD = Features2=3D0x8003f7 > Structured Extended = Features=3D0x209c01ab > XSAVE Features=3D0x7 > AMD Extended Feature Extensions ID EBX=3D0x2001000 > SVM: Features=3D0x0 > Revision=3D1, ASIDs=3D16 > L1 2MB data TLB: 255 entries, 1-way associative > L1 2MB instruction TLB: 255 entries, 1-way associative > L1 4KB data TLB: 255 entries, 1-way associative > L1 4KB instruction TLB: 255 entries, 1-way associative > L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way = associative > L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, = 2-way associative > L2 2MB unified TLB: 0 entries, disabled/not present > L2 4KB data TLB: 512 entries, 4-way associative > L2 4KB instruction TLB: 512 entries, 4-way associative > L2 unified cache: 512 kbytes, 64 bytes/line, 1 lines/tag, 16-way = associative > Hypervisor: Origin =3D "KVMKVMKVM" > real memory =3D 4294967296 (4096 MB) > Physical memory chunk(s): > 0x0000000000001000 - 0x000000000009efff, 647168 bytes (158 = pages) > 0x0000000000103000 - 0x00000000001fffff, 1036288 bytes (253 = pages) > 0x0000000002800000 - 0x00000000b7d44fff, 3042201600 bytes = (742725 pages) > 0x0000000100000000 - 0x000000013ffe7fff, 1073643520 bytes = (262120 pages) > avail memory =3D 4099080192 (3909 MB) > Table 'FACP' at 0xbffe104e > Table 'APIC' at 0xbffe10c2 > Table 'HPET' at 0xbffe1152 > ACPI: No DMAR table found > ... > ... > ... > ata0: stat0=3D0x00 err=3D0x00 lsb=3D0x00 msb=3D0x00 > ata0: stat1=3D0x00 err=3D0x00 lsb=3D0x00 msb=3D0x00 > ata0: reset tp2 stat0=3D00 stat1=3D00 devices=3D0x0 > ata1: reset tp1 mask=3D03 ostat0=3D00 ostat1=3D00 > ata1: stat0=3D0x00 err=3D0x00 lsb=3D0x00 msb=3D0x00 > ata1: stat1=3D0x00 err=3D0x00 lsb=3D0x00 msb=3D0x00 > ata1: reset tp2 stat0=3D00 stat1=3D00 devices=3D0x0 > Trying to mount root from ufs:/dev/vtbd0p2 [rw]... > atrtc0: providing initial system time > start_init: trying /sbin/init > uhub0: 2 ports with 2 removable, self powered >=20 > After that the system freezes. >=20 > I can provide more complete information if somebody tells me what = would be helpfull. >=20 > How should I go about debugging this ??? >=20 > Greetings > Christian >=20 > --=20 > Christian Kratzer CK Software GmbH > Email: ck@cksoft.de Wildberger Weg 24/2 > Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden > Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart > Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian = Kratzer > Web: http://www.cksoft.de/ > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to = "freebsd-virtualization-unsubscribe@freebsd.org" >=20