From owner-freebsd-virtualization@freebsd.org Sun Mar 4 21:00:30 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7561CF322CD for ; Sun, 4 Mar 2018 21:00:30 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15F1A6F26A for ; Sun, 4 Mar 2018 21:00:30 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 67C0814445 for ; Sun, 4 Mar 2018 21:00:29 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w24L0TUr076248 for ; Sun, 4 Mar 2018 21:00:29 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w24L0TtV076238 for freebsd-virtualization@FreeBSD.org; Sun, 4 Mar 2018 21:00:29 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201803042100.w24L0TtV076238@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: freebsd-virtualization@FreeBSD.org Subject: Problem reports for freebsd-virtualization@FreeBSD.org that need special attention Date: Sun, 4 Mar 2018 21:00:29 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Mar 2018 21:00:30 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 212820 | FreeBSD 10-STABLE from latest HEAD and 11-RELEASE 1 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Tue Mar 6 06:52:22 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9A04F343FA for ; Tue, 6 Mar 2018 06:52:22 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C2DC8594F for ; Tue, 6 Mar 2018 06:52:19 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.119.1] (wlan-141-23-171-244.tubit.tu-berlin.de [141.23.171.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 99AEF61FA1; Tue, 6 Mar 2018 06:45:12 +0000 (UTC) From: "Fabian Freyer" To: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org Subject: rumpkernel and bhyve: triple faults Date: Tue, 06 Mar 2018 07:45:00 +0100 X-Mailer: MailMate (1.10r5443) Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 06:52:23 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello lists, I=E2=80=99m currently playing around with getting rump kernels built with= rumpkernel(7) running on FreeBSD=E2=80=99s bhyve(4). I=E2=80=99m using a= custom boot loader [1] which builds on some patches to bhyveload / user = boot [2]. To test, I=E2=80=99m using a simple =E2=80=9Chelloer=E2=80=9D unikernel f= rom the tutorial [3]: #include #include int main() { printf("Hello, Rumprun ... I'm feeling tired\n"); sleep(2); printf("much better!\n"); return 0; } I=E2=80=99m compiling this for the hw_virtio platform: x86_64-rumprun-netbsd-gcc -o helloer-rumprun helloer.c rumprun-bake hw_virtio helloer-rumprun.elf helloer-rumprun Starting this kernel in bhyve gives me the following VM Exit: =46rom the Intel SDM 3D, Appendix C [4], VM Exit 2 is a triple fault. To further debug this, I=E2=80=99m using the ktr(4) support introduced in= FreeBSD r276098 [5]. I=E2=80=99ve appended a full KTR trace further down in my mail, but I=E2=80= =99ll go along the trace here with some commentary in execution order, an= d attempt to annotate the source code lines from rumpkernel: 0x104000 is the entry address given in rump kernel=E2=80=99s multi boot h= eader. 0000000000104000 <_start>: 632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000 636 0 350886961656 vm testing[0]: Resume execution at 0x104000 This is the cpuid instruction in x86_cpuid which is called from hyperviso= r_detect: rumprun/platform/hw/arch/x86/x86_subr.c: 91 0000000000102f90 : 102f90: 49 89 d2 mov r10,rdx 102f93: 49 89 c9 mov r9,rcx 102f96: 53 push rbx 102f97: 89 f8 mov eax,edi 102f99: 0f a2 cpuid rumprun/platform/hw/arch/x86/hypervisor.c: 38 637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8 638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f= 99 639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b rumprun/platform/hw/arch/x86/hypervisor.c: 40 640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8 641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f= 99 642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b rumprun/platform/hw/arch/x86/hypervisor.c: 51 643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8 644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f= 99 645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b The next vexit is in cons_init, directly after the call to hypervisor_det= ect. 0000000000102a50 : 102a50: 53 push rbx 102a51: e8 ea 0a 00 00 call 103540 = 102a56: 66 83 3d a4 d5 ef ff cmp WORD PTR [rip-0x102a5c],0x= 0 # 2 Due to compiler optimisations, the check here isn=E2=80=99t the (hypervisor =3D=3D HYPERVISOR_XEN) check directly after the call to hyper= visor_detect, but the check (bios_crtc_base =3D=3D 0) in rumprun/platform/hw/arch/x86/cons.c:59: 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x= 102a56 Therefore, I=E2=80=99m assuming this is the origin of the fault. Tracking down bios_crtc_base, I find that it=E2=80=99s loaded in rumprun/platform/hw/arch/amd64/locore.S:70: /* save BIOS data area values */ movw BIOS_COM1_BASE, %bx movw %bx, bios_com1_base movw BIOS_CRTC_BASE, %bx movw %bx, bios_crtc_base Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the b= hyve device node in /dev/vmm with xxd(1), I find the words at these addresses = to be Uninitialised: 00000400: 0000 .. 00000483: 0000 .. I=E2=80=99m not sure where to go from here. Is this a bug in bhyve(4), sh= ould these values be initialised somehow, or should I patch rumpkernel(7) to skip th= is check when running on bhyve(4)? Fabian Full KTR trace: index cpu timestamp trace ------ --- ---------------- ----- 673 0 350887364560 vm testing[0]: vcpu state changed from frozen= to idle 672 0 350887363766 vm testing[0]: retu 0/14 671 0 350887362932 vm testing[0]: All vcpus suspended 670 0 350887357212 vm testing[0]: vcpu state changed from runnin= g to frozen 669 0 350887350444 vm testing[0]: returning from vmx_run: exitco= de 14 668 0 350887348806 vm testing: virtual machine successfully susp= ended 4 667 0 350887347712 vm testing[0]: triple fault: info1(0x80000b08= ), info2(0x80000b0e) 666 0 350887347590 vm testing[0]: Exception 14 delivered: 0x8000= 0b0e 665 0 350887346312 vm testing[0]: handled exception vmexit at 0x= 102a56 664 0 350887346178 vm testing[0]: Exception 14 pending 663 0 350887346060 vm testing[0]: Setting intr_shadow to 0 succe= eded 662 0 350887341876 vm testing[0]: Reflecting exception 14/0 into= the guest 661 0 350887339570 vm testing[0]: vm_exit_intinfo: info1(0x80000= b08) 660 0 350887315326 vm testing[0]: Resume execution at 0x102a56 659 0 350887311168 vm testing[0]: vm_entry_intinfo: info1(0x8000= 0b0e), info2(0x80000b0e), retinfo(0x80000b08) 658 0 350887310988 vm testing[0]: Exception 14 delivered: 0x8000= 0b0e 657 0 350887309700 vm testing[0]: handled exception vmexit at 0x= 102a56 656 0 350887309570 vm testing[0]: Exception 14 pending 655 0 350887309442 vm testing[0]: Setting intr_shadow to 0 succe= eded 654 0 350887305126 vm testing[0]: Reflecting exception 14/0 into= the guest 653 0 350887302436 vm testing[0]: vm_exit_intinfo: info1(0x80000= b0e) 652 0 350887248280 vm testing[0]: Resume execution at 0x102a56 651 0 350887184160 vm testing[0]: vm_entry_intinfo: info1(0), in= fo2(0x80000b0e), retinfo(0x80000b0e) 650 0 350887184040 vm testing[0]: Exception 14 delivered: 0x8000= 0b0e 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x= 102a56 648 0 350887182374 vm testing[0]: Exception 14 pending 647 0 350887182240 vm testing[0]: Setting intr_shadow to 0 succe= eded 646 0 350887176682 vm testing[0]: Reflecting exception 14/0 into= the guest 645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b 644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f= 99 643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8 642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b 641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f= 99 640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8 639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b 638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f= 99 637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8 636 0 350886961656 vm testing[0]: Resume execution at 0x104000 635 0 350886900428 vm testing[0]: vcpu state changed from frozen= to running 634 0 350886895942 vm testing[0]: vcpu state changed from idle t= o frozen 633 0 350886886630 vm testing[0]: vcpu state changed from frozen= to idle 632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000 631 0 350886802596 vm testing[0]: vcpu state changed from idle t= o frozen 630 3 350886414966 vm testing[0]: vcpu state changed from frozen= to idle 629 3 350886414184 vm testing[0]: activated 628 3 350886412730 vm testing[0]: vcpu state changed from idle t= o frozen 627 3 350886137831 vm testing[0]: vcpu state changed from frozen= to idle 626 3 350885989045 vm testing[0]: vcpu state changed from idle t= o frozen 625 3 350885773367 vm testing: RTC time set to 0x5a9e154e 624 3 350885773017 vm testing: Updating RTC base uptime from 0x3= 1c2762314 to 0x8bd6b051a2 623 3 350885772543 vm testing: Updating RTC secs from 0x5a9e14f4= to 0x5a9e154e 622 3 350885017699 vm testing: RTC nvram write 0 to offset 0x5d 621 3 350885015953 vm testing: RTC nvram write 0 to offset 0x5c 620 3 350885013951 vm testing: RTC nvram write 0 to offset 0x5b 619 3 350885010615 vm testing: RTC nvram write 0x1 to offset 0x3= 5 618 3 350885008163 vm testing: RTC nvram write 0 to offset 0x34 [1] https://github.com/fubarnetes/bhyve-multiboot [2] https://reviews.FreeBSD.org/D14473 [3] https://github.com/rumpkernel/wiki/wiki/Tutorial%3A-Building-Rumprun-= Unikernels#2-building-applications [4] https://www.intel.com/content/dam/www/public/us/en/documents/manuals/= 64-ia-32-architectures-software-developer-vol-3d-part-4-manual.pdf [5] https://svnweb.freebsd.org/base?view=3Drevision&revision=3D276098 --=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJWBAEBCgBAFiEEX6JoxdmEemcFacQZmealkcs85+YFAlqeOOwiHGZhYmlhbi5m cmV5ZXJAcGh5c2lrLnR1LWJlcmxpbi5kZQAKCRCZ5qWRyzzn5owLD/915dsZZNlz stLP2tVgPifGtY8yCD5U/WfpWznOxzDmeyBlJszTdGoKRdbTDoZHSHsEype/PjAw fn6cQKHChmsW7GwZSMZ/ETgu9xYzPaS9tDQPpRNcRpku0Npvw4Kifv4QzSibwP7e 6vfTFk59SoCkScinVXuKbjvylwnEtxbzIxWvSo24aXebcEdqSKsX8zKemGra00XM S9NSny4Sa5VOpuMsPMhuvwexEP3CJW888h5/3xPNU9N5/vbLc1FDHarS2QNQF8fB iy1F0lyIEP4iAzwO6STLtJv+tMiZh//2V/o6tKxFQMtgDlc6bb/vw40OlG/Z8Es/ Z9Gbeg6YZVdrxLGVE5sEgiC4BHkmgbFGp1yqkA23OMF3nFqTiLulftUun90S66uT Vy2vor/tnVH6FrhnZ+f8Xh2s1movStz6DpEUFlZfmOxE6Z5lftn5089KYRZeMs++ i4P5x9ajhIhGNeqkKIeeWgjZ7CNh+CcG/ZJqvWmcln/j8/lPSIktCYSHPbCOYdzb 1RigTNstbX9I+2fZU4caaMlxcDE712zdt4HA55j7UbTPjeMRcXah1H/szPBcHWQK O923YtXpiYRQaRTIAzt9++BWtjU60P2YQ4KQnZ2hzs6DlUKBSr33FFU4xUsFUt6Q FrBzA4skxpa29GEsgeL2sreOREmNY6aQwg== =qScN -----END PGP SIGNATURE----- --=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_=-- From owner-freebsd-virtualization@freebsd.org Tue Mar 6 08:06:34 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BD71F3CE41 for ; Tue, 6 Mar 2018 08:06:34 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B464968F8C for ; Tue, 6 Mar 2018 08:06:33 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w2686TF1047820; Tue, 6 Mar 2018 00:06:29 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w2686Shu047819; Tue, 6 Mar 2018 00:06:28 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803060806.w2686Shu047819@pdx.rh.CN85.dnsmgr.net> Subject: Re: rumpkernel and bhyve: triple faults In-Reply-To: To: Fabian Freyer Date: Tue, 6 Mar 2018 00:06:28 -0800 (PST) CC: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 08:06:34 -0000 > Hello lists, > > I?m currently playing around with getting rump kernels built with rumpkernel(7) running on FreeBSD?s bhyve(4). I?m using a custom boot loader [1] which builds on some patches to bhyveload / user boot [2]. > > To test, I?m using a simple ?helloer? unikernel from the tutorial [3]: > ... excelent discription of your debug process removed for breif reply ... > Due to compiler optimisations, the check here isn?t the > (hypervisor == HYPERVISOR_XEN) check directly after the call to hypervisor_detect, > but the check (bios_crtc_base == 0) in bios_crtc_base would be part of the isa legacy vga controller card. Bhyve does not, at this time, or in the near future expect to have, support for this legacy device. > rumprun/platform/hw/arch/x86/cons.c:59: > 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56 > > Therefore, I?m assuming this is the origin of the fault. > > Tracking down bios_crtc_base, I find that it?s loaded in > rumprun/platform/hw/arch/amd64/locore.S:70: > > /* save BIOS data area values */ > movw BIOS_COM1_BASE, %bx > movw %bx, bios_com1_base > movw BIOS_CRTC_BASE, %bx > movw %bx, bios_crtc_base > > Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve > device node in /dev/vmm with xxd(1), I find the words at these addresses to be > Uninitialised: > > 00000400: 0000 .. > 00000483: 0000 .. Typo here, should this be 00000463? > I?m not sure where to go from here. Is this a bug in bhyve(4), No, it is not a bug, it is an unimplemented device. > should these > values be initialised somehow, or should I patch rumpkernel(7) to skip this check > when running on bhyve(4)? rumpkernel is assuming or requiring the presence of legacy isa hardware, it should probably be taught that this may not exist. You could simply skip this check, but I expect you would then have a harder to find failure later when it tries to use the hardware it expects to be there. > > Fabian ... Full KTR trace removed for breif reply ... -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Mar 6 08:28:40 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39D1BF3F442 for ; Tue, 6 Mar 2018 08:28:40 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A101F6A389 for ; Tue, 6 Mar 2018 08:28:39 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w268Sbrm047943; Tue, 6 Mar 2018 00:28:37 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w268SbZX047942; Tue, 6 Mar 2018 00:28:37 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803060828.w268SbZX047942@pdx.rh.CN85.dnsmgr.net> Subject: Re: rumpkernel and bhyve: triple faults In-Reply-To: <201803060806.w2686Shu047819@pdx.rh.CN85.dnsmgr.net> To: "Rodney W. Grimes" Date: Tue, 6 Mar 2018 00:28:37 -0800 (PST) CC: Fabian Freyer , rumpkernel-users@freelists.org, freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 08:28:40 -0000 > > Hello lists, > > > > I?m currently playing around with getting rump kernels built with rumpkernel(7) running on FreeBSD?s bhyve(4). I?m using a custom boot loader [1] which builds on some patches to bhyveload / user boot [2]. > > > > To test, I?m using a simple ?helloer? unikernel from the tutorial [3]: > > > ... excelent discription of your debug process removed for breif reply ... > > > Due to compiler optimisations, the check here isn?t the > > (hypervisor == HYPERVISOR_XEN) check directly after the call to hypervisor_detect, > > but the check (bios_crtc_base == 0) in > > bios_crtc_base would be part of the isa legacy vga > controller card. Bhyve does not, at this time, or > in the near future expect to have, support for this > legacy device. I am wrong on this, the framebuffer device does infact have support for the legacy i/o addresses that this should point to. You should see the "vgaconf" section of the FrameBuffer section of the bhyve(8) manpage. I believe you need to be running bhyve with the uefi bios options, the with CMS version, and have vgaconf=on to get your code to work as is. > > rumprun/platform/hw/arch/x86/cons.c:59: > > 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56 > > > > Therefore, I?m assuming this is the origin of the fault. > > > > Tracking down bios_crtc_base, I find that it?s loaded in > > rumprun/platform/hw/arch/amd64/locore.S:70: > > > > /* save BIOS data area values */ > > movw BIOS_COM1_BASE, %bx > > movw %bx, bios_com1_base > > movw BIOS_CRTC_BASE, %bx > > movw %bx, bios_crtc_base > > > > Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve > > device node in /dev/vmm with xxd(1), I find the words at these addresses to be > > Uninitialised: > > > > 00000400: 0000 .. > > 00000483: 0000 .. > Typo here, should this be 00000463? > > > > I?m not sure where to go from here. Is this a bug in bhyve(4), > No, it is not a bug, it is an unimplemented device. > > > should these > > values be initialised somehow, or should I patch rumpkernel(7) to skip this check > > when running on bhyve(4)? > rumpkernel is assuming or requiring the presence of legacy isa hardware, > it should probably be taught that this may not exist. You could simply > skip this check, but I expect you would then have a harder to find > failure later when it tries to use the hardware it expects to be > there. > > > > > Fabian > > ... Full KTR trace removed for breif reply ... -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Mar 6 08:53:25 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B2C0F4136C for ; Tue, 6 Mar 2018 08:53:25 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E24A26B818 for ; Tue, 6 Mar 2018 08:53:24 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.119.1] (wlan-141-23-171-244.tubit.tu-berlin.de [141.23.171.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id EB56762069; Tue, 6 Mar 2018 08:53:21 +0000 (UTC) From: "Fabian Freyer" To: "Rodney W. Grimes" Cc: rumpkernel-users@freelists.org, freebsd-virtualization@freebsd.org Subject: Re: rumpkernel and bhyve: triple faults Date: Tue, 06 Mar 2018 09:53:21 +0100 X-Mailer: MailMate (1.10r5443) Message-ID: <256A0D22-46D6-44DA-9ACF-631FE981D0EC@physik.tu-berlin.de> In-Reply-To: <201803060828.w268SbZX047942@pdx.rh.CN85.dnsmgr.net> References: <201803060828.w268SbZX047942@pdx.rh.CN85.dnsmgr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 08:53:25 -0000 On 6 Mar 2018, at 9:28, Rodney W. Grimes wrote: >> bios_crtc_base would be part of the isa legacy vga >> controller card. Bhyve does not, at this time, or >> in the near future expect to have, support for this >> legacy device. > > I am wrong on this, the framebuffer device does > infact have support for the legacy i/o addresses > that this should point to. You should see the > "vgaconf" section of the FrameBuffer section > of the bhyve(8) manpage. > > I believe you need to be running bhyve with the > uefi bios options, the with CMS version, and > have vgaconf=on to get your code to work as is. For diskless multiboot kernels I’m going with a separate userboot.so-compatible loader. Specifying a UEFI bootrom implicitly resets the CPU. (See usr.sbin/bhyve/bhyverun.c:960) I think deciding to use the serial output (which is what most of rumpkernel’s cons_init is doing) based on the hypervisor is probably the right way to go. Something similar is already done for XEN: /* * If running under Xen use the serial console. */ if (hypervisor == HYPERVISOR_XEN) prefer_serial = 1; >>> rumprun/platform/hw/arch/x86/cons.c:59: >>> 649 0 350887182668 vm testing[0]: handled exception vmexit >>> at 0x102a56 >>> >>> Therefore, I?m assuming this is the origin of the fault. >>> >>> Tracking down bios_crtc_base, I find that it?s loaded in >>> rumprun/platform/hw/arch/amd64/locore.S:70: >>> >>> /* save BIOS data area values */ >>> movw BIOS_COM1_BASE, %bx >>> movw %bx, bios_com1_base >>> movw BIOS_CRTC_BASE, %bx >>> movw %bx, bios_crtc_base >>> >>> Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking >>> the bhyve >>> device node in /dev/vmm with xxd(1), I find the words at these >>> addresses to be >>> Uninitialised: >>> >>> 00000400: 0000 .. >>> 00000483: 0000 .. >> Typo here, should this be 00000463? Yes, sorry about that. Fabian From owner-freebsd-virtualization@freebsd.org Tue Mar 6 09:18:17 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D995CF4306B for ; Tue, 6 Mar 2018 09:18:16 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48FF86C4F2 for ; Tue, 6 Mar 2018 09:18:15 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w269ICbw048091; Tue, 6 Mar 2018 01:18:12 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w269IAl1048090; Tue, 6 Mar 2018 01:18:10 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803060918.w269IAl1048090@pdx.rh.CN85.dnsmgr.net> Subject: Re: rumpkernel and bhyve: triple faults In-Reply-To: <256A0D22-46D6-44DA-9ACF-631FE981D0EC@physik.tu-berlin.de> To: Fabian Freyer Date: Tue, 6 Mar 2018 01:18:10 -0800 (PST) CC: rumpkernel-users@freelists.org, freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 09:18:17 -0000 > On 6 Mar 2018, at 9:28, Rodney W. Grimes wrote: > > >> bios_crtc_base would be part of the isa legacy vga > >> controller card. Bhyve does not, at this time, or > >> in the near future expect to have, support for this > >> legacy device. > > > > I am wrong on this, the framebuffer device does > > infact have support for the legacy i/o addresses > > that this should point to. You should see the > > "vgaconf" section of the FrameBuffer section > > of the bhyve(8) manpage. > > > > I believe you need to be running bhyve with the > > uefi bios options, the with CMS version, and > > have vgaconf=on to get your code to work as is. > > For diskless multiboot kernels I?m going with a > separate userboot.so-compatible loader. Specifying > a UEFI bootrom implicitly resets the CPU. > (See usr.sbin/bhyve/bhyverun.c:960) Well in that case my original claim that there is not a legacy isa vga device avaliable would be correct for this environment, and you should just eliminate anything that tries to use it, or make it understand that this device may not exist. I am not sure if bhyve maps any of these legacy addresses if your not using bhyveloader or uefi bios code. 0x400 and 0x483 being unmapped could lead to your tripple fault. > > I think deciding to use the serial output (which is > what most of rumpkernel?s cons_init is doing) based > on the hypervisor is probably the right way to go. > Something similar is already done for XEN: > > /* > * If running under Xen use the serial console. > */ > if (hypervisor == HYPERVISOR_XEN) > prefer_serial = 1; > > >>> rumprun/platform/hw/arch/x86/cons.c:59: > >>> 649 0 350887182668 vm testing[0]: handled exception vmexit > >>> at 0x102a56 > >>> > >>> Therefore, I?m assuming this is the origin of the fault. > >>> > >>> Tracking down bios_crtc_base, I find that it?s loaded in > >>> rumprun/platform/hw/arch/amd64/locore.S:70: > >>> > >>> /* save BIOS data area values */ > >>> movw BIOS_COM1_BASE, %bx > >>> movw %bx, bios_com1_base > >>> movw BIOS_CRTC_BASE, %bx > >>> movw %bx, bios_crtc_base > >>> > >>> Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking > >>> the bhyve > >>> device node in /dev/vmm with xxd(1), I find the words at these > >>> addresses to be > >>> Uninitialised: > >>> > >>> 00000400: 0000 .. > >>> 00000483: 0000 .. > >> Typo here, should this be 00000463? > Yes, sorry about that. > > Fabian > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Mar 6 09:50:28 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A08DF45117 for ; Tue, 6 Mar 2018 09:50:28 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2ADEE6D935; Tue, 6 Mar 2018 09:50:24 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.119.1] (wlan-141-23-171-244.tubit.tu-berlin.de [141.23.171.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 215CE61FA1; Tue, 6 Mar 2018 09:50:24 +0000 (UTC) From: "Fabian Freyer" To: "Rodney W. Grimes" Cc: freebsd-virtualization@freebsd.org, tychon@Freebsd.org Subject: Re: rumpkernel and bhyve: triple faults Date: Tue, 06 Mar 2018 10:50:22 +0100 X-Mailer: MailMate (1.10r5443) Message-ID: <43C0E566-19BC-4AB9-96BD-EE9608DA63F5@physik.tu-berlin.de> In-Reply-To: <201803060918.w269IAl1048090@pdx.rh.CN85.dnsmgr.net> References: <201803060918.w269IAl1048090@pdx.rh.CN85.dnsmgr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 09:50:28 -0000 (un-CC’ing rumpkernel-users@, since this part of the discussion is getting very bhyve-specific) On 6 Mar 2018, at 10:18, Rodney W. Grimes wrote: >> On 6 Mar 2018, at 9:28, Rodney W. Grimes wrote: >> >>>> bios_crtc_base would be part of the isa legacy vga >>>> controller card. Bhyve does not, at this time, or >>>> in the near future expect to have, support for this >>>> legacy device. >>> >>> I am wrong on this, the framebuffer device does >>> infact have support for the legacy i/o addresses >>> that this should point to. You should see the >>> "vgaconf" section of the FrameBuffer section >>> of the bhyve(8) manpage. >>> >>> I believe you need to be running bhyve with the >>> uefi bios options, the with CMS version, and >>> have vgaconf=on to get your code to work as is. >> >> For diskless multiboot kernels I?m going with a >> separate userboot.so-compatible loader. Specifying >> a UEFI bootrom implicitly resets the CPU. >> (See usr.sbin/bhyve/bhyverun.c:960) > > Well in that case my original claim that there > is not a legacy isa vga device avaliable would > be correct for this environment, and you should > just eliminate anything that tries to use it, > or make it understand that this device may not > exist. I am not sure if bhyve maps any of these > legacy addresses if your not using bhyveloader > or uefi bios code. 0x400 and 0x483 being > unmapped could lead to your tripple fault. According to lib/libvmmapi/vmmapi.c:409, (and some code / comments in grub2-bhyve), bhyve maps up to two memory segments: [0, lowmem) and if more than 3 GiB of memory is given, [4GiB, 4GiB + highmem). According to the comments in grub2-bhyve/grub-core/kern/emu/bhyve_hostif.c:149 The area from 640kiB to 1MiB is reserved as the vga hole and BIOS. 0x400 and 0x463 are in the bios data area [1], which is mapped, but zeroed out: 40:00 word COM1 port address 40:63 word Base port address for active 6845 CRT controller I’m not sure whether it might not be useful for bhyve to populate the bios data area with the information that it does have available (e.g. serial ports). On a related note, I saw that there is a relevant GSoC Project idea [2] to be mentored by tychon@ (CC’d) that mentions VGA/VESA device emulation independent of the UEFI frame buffer. I’ll try to look into that more, and possibly submit a proposal :). Fabian [1] http://stanislavs.org/helppc/bios_data_area.html [2] https://wiki.freebsd.org/SummerOfCodeIdeas#VGA_emulation_improvements_for_bhyve From owner-freebsd-virtualization@freebsd.org Tue Mar 6 15:15:49 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7D93F39C57 for ; Tue, 6 Mar 2018 15:15:49 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) by mx1.freebsd.org (Postfix) with ESMTP id 088FE7B33A for ; Tue, 6 Mar 2018 15:15:48 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by vito.onthenet.com.au (Postfix) with ESMTP id 86A712076FB0 for ; Wed, 7 Mar 2018 01:15:46 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 6819C20B4A0C for ; Wed, 7 Mar 2018 01:15:46 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 5C210281A4B for ; Wed, 7 Mar 2018 01:15:46 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id qtSbALU1VroP for ; Wed, 7 Mar 2018 01:15:46 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 6482F2804D6; Wed, 7 Mar 2018 01:15:43 +1000 (AEST) Subject: Re: rumpkernel and bhyve: triple faults To: Fabian Freyer References: From: Peter Grehan Cc: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org Message-ID: <651856d3-3c34-9930-cda1-62d41091f91f@freebsd.org> Date: Tue, 6 Mar 2018 07:15:41 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=dNCIZtRb c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=BRlsKWOS0KQfm44lbqQA:9 a=y3tL4Oqu7xm67Kjg:21 a=9dlVhwN69FE26nys:21 a=QEXdDO2ut3YA:10 wl=host:3 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=Mvd8FVSe c=1 sm=1 tr=0 a=mJOSnoNX3k71adV6TmU0eQ==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=BRlsKWOS0KQfm44lbqQA:9 a=y3tL4Oqu7xm67Kjg:21 a=9dlVhwN69FE26nys:21 a=QEXdDO2ut3YA:10 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 15:15:50 -0000 Hi Fabian, > 657 0 350887309700 vm testing[0]: handled exception vmexit at 0x102a56 > 656 0 350887309570 vm testing[0]: Exception 14 pending > 655 0 350887309442 vm testing[0]: Setting intr_shadow to 0 succeeded > 654 0 350887305126 vm testing[0]: Reflecting exception 14/0 into the guest > 653 0 350887302436 vm testing[0]: vm_exit_intinfo: info1(0x80000b0e) > 652 0 350887248280 vm testing[0]: Resume execution at 0x102a56 > 651 0 350887184160 vm testing[0]: vm_entry_intinfo: info1(0), info2(0x80000b0e), retinfo(0x80000b0e) > 650 0 350887184040 vm testing[0]: Exception 14 delivered: 0x80000b0e > 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56 Exception 14 is a page fault (SDM Vol3 ch 6.15). The exception type is "fault" which means it is delivered at the address it was detected at. This cascaded very quickly into a triple-fault, so it looks like it could possibly be an issue with the stack. One debug tool you do have is to get a register dump on exit, with 'bhyvectl --get-all --vm='. For a page-fault, the virtual address that resulted in the fault will be in the CR2 register. From the code at the faulting address: > 0000000000102a50 : > 102a50: push rbx > 102a51: call 103540 > 102a56: cmp WORD PTR [rip-0x102a5c],0x0 # 2 It's using RIP-relative addressing here, but objdump seems to think this may be an offset in the current_lwp structure - is it possible that may have an uninitialized value ? (I don't believe this has anything to do with VGA). later, Peter. From owner-freebsd-virtualization@freebsd.org Tue Mar 6 16:22:07 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9116F3FB04 for ; Tue, 6 Mar 2018 16:22:07 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 545DF7E578; Tue, 6 Mar 2018 16:22:06 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.119.1] (wlan-141-23-171-244.tubit.tu-berlin.de [141.23.171.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 674E161FA2; Tue, 6 Mar 2018 16:22:03 +0000 (UTC) From: "Fabian Freyer" To: "Peter Grehan" Cc: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org Subject: Re: rumpkernel and bhyve: triple faults Date: Tue, 06 Mar 2018 17:22:03 +0100 X-Mailer: MailMate (1.10r5443) Message-ID: <2AAD4069-7D37-4325-8990-21C5DFD80B3B@physik.tu-berlin.de> In-Reply-To: <651856d3-3c34-9930-cda1-62d41091f91f@freebsd.org> References: <651856d3-3c34-9930-cda1-62d41091f91f@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 16:22:08 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Peter, On 6 Mar 2018, at 16:15, Peter Grehan wrote: > Exception 14 is a page fault (SDM Vol3 ch 6.15). The exception type is= "fault" which means it is delivered at the address it was detected at. > > This cascaded very quickly into a triple-fault, so it looks like it co= uld possibly be an issue with the stack. One debug tool you do have is to= get a register dump on exit, with 'bhyvectl --get-all --vm=3D'. > > For a page-fault, the virtual address that resulted in the fault will = be in the CR2 register. I don=E2=80=99t see a CR2 register in the output of bhyvectl --get-all, I= was looking for that too. > From the code at the faulting address: > > > 0000000000102a50 : > > 102a50: push rbx > > 102a51: call 103540 > > 102a56: cmp WORD PTR [rip-0x102a5c],0x0 # 2 > > It's using RIP-relative addressing here, but objdump seems to think th= is may be an offset in the current_lwp structure - is it possible that ma= y have an uninitialized value ? I=E2=80=99m pretty sure it=E2=80=99s tooling that=E2=80=99s displaying so= mething off, since hopper is showing me this as 0x0000000000102a56 cmp word [0x2], 0x0 Which is very similar to what r2 is giving me: ;-- cons_init: 0x00102a50 53 push rbx ; /arch/x86:43= 0x00102a51 e8ea0a0000 call sym.hypervisor_detect ; /arch/x86:47= 0x00102a56 66833da4d5ef. cmp word [0x00000002], 0 ; /arch/x86:62= > (I don't believe this has anything to do with VGA). Maybe I=E2=80=99m off with my analysis of the actual fault here, but how = I understand the source (assuming compilers work as I would expect, which is not alway= s true) the values here are initialised from values in the bios data area (which = is zeroed out on bhyve): #define BIOS_COM1_BASE 0x400 #define BIOS_CRTC_BASE 0x463 =2E.. movw BIOS_COM1_BASE, %bx movw %bx, bios_com1_base movw BIOS_CRTC_BASE, %bx movw %bx, bios_crtc_base =2E.. /* * If the BIOS says no CRTC is present use the serial console if * available. */ if (bios_crtc_base =3D=3D 0) prefer_serial =3D 1; Here=E2=80=99s my full output from bhyvectl --get-all: ID Length Name 0 128MB sysmem Address Length Segment Offset Prot Flags 0 128MB sysmem 0 RWX efer[0] 0x0000000000000500 cr0[0] 0x0000000080010031 cr3[0] 0x000000000010b000 cr4[0] 0x0000000000002620 dr7[0] 0x0000000000000400 rsp[0] 0x0000000000100ff0 rip[0] 0x0000000000102a56 rax[0] 0x0000000000000000 rbx[0] 0x00000000003eaa2b rcx[0] 0x0000000068622065 rdx[0] 0x0000000020657679 rsi[0] 0x0000000000100fd0 rdi[0] 0x0000000040000000 rbp[0] 0x0000000000000000 r8[0] 0x0000000000100fdc r9[0] 0x0000000000100fd8 r10[0] 0x0000000000100fd4 r11[0] 0x0000000000000000 r12[0] 0x0000000000000000 r13[0] 0x0000000000000000 r14[0] 0x0000000000000000 r15[0] 0x0000000000000000 rflags[0] 0x0000000000010006 ds desc[0] 0x0000000000000000/0xffffffff/0x0000c093 es desc[0] 0x0000000000000000/0xffffffff/0x0000c093 fs desc[0] 0x0000000000000000/0xffffffff/0x0001c001 gs desc[0] 0x0000000000000000/0xffffffff/0x0001c001 ss desc[0] 0x0000000000000000/0xffffffff/0x0000c093 cs desc[0] 0x0000000000000000/0xffffffff/0x0000a09b tr desc[0] 0x0000000000000000/0x00000000/0x0000008b ldtr desc[0] 0x0000000000000000/0x0000ffff/0x00000082 gdtr[0] 0x0000000000378040/0x0000002f idtr[0] 0x0000000000000000/0x0000ffff cs[0] 0x0008 ds[0] 0x0018 es[0] 0x0018 fs[0] 0x0000 gs[0] 0x0000 ss[0] 0x0018 tr[0] 0x0000 ldtr[0] 0x0000 cr0_mask[0] 0xffffffff60000020 cr0_shadow[0] 0x0000000000000021 cr4_mask[0] 0xffffffffffe8f800 cr4_shadow[0] 0x0000000000000000 cr3_target_count[0] 0x0000000000000000 cr3_target0[0] 0x0000000000000000 cr3_target1[0] 0x0000000000000000 cr3_target2[0] 0x0000000000000000 cr3_target3[0] 0x0000000000000000 pinbased_ctls[0] 0x000000000000003f procbased_ctls[0] 0x00000000f51865f2 procbased_ctls2[0] 0x00000000000010a2 gla[0] 0xfffffe0000c41000 gpa[0] 0x0000000000000000 entry_interruption_info[0] 0x0000000000000000 tpr_threshold[0] 0x0000000000000000 instruction_error[0] 0x0000000000000000 exit_ctls[0] 0x000000000033efff entry_ctls[0] 0x00000000000093ff host_pat[0] 0x0001050600070406 host_cr0[0] 0x000000008005003b host_cr3[0] 0x0000000038045054 host_cr4[0] 0x00000000001726e0 host_rip[0] 0xffffffff81435290 host_rsp[0] 0xfffffe003218d700 vmcs_pointer[0] 0xffffffffffffffff vmcs_exit_interruption_info[0] 0x0000000080000b0e vmcs_exit_interruption_error[0] 0x0000000000000000 vmcs_guest_interruptibility[0] 0x0000000000000000 vmcs_exit_inst_length[0] 0x00000003 vmcs_exit_qualification[0] 0x0000000000000080 x2apic_state[0] 0 eptp[0] 0x000000003817905e exception_bitmap[0] 0xffffffff io_bitmap_a[0] 0 io_bitmap_b[0] 0 tsc_offset[0] 0x0000000000000000 msr_bitmap[0] 0x1adbc000 MSR_TSC [0] R- MSR_EFER [0] RW MSR_STAR [0] RW MSR_LSTAR [0] RW MSR_CSTAR [0] RW MSR_SF_MASK [0] RW MSR_FSBASE [0] RW MSR_GSBASE [0] RW MSR_KGSBASE [0] RW MSR_SYSENTER_CS_MSR [0] RW MSR_SYSENTER_ESP_MSR[0] RW MSR_SYSENTER_EIP_MSR[0] RW vpid[0] 0x0011 guest_pat[0] 0x0000000000000000 guest_sysenter_cs[0] 0 guest_sysenter_sp[0] 0 guest_sysenter_ip[0] 0 exit_reason[0] 0 rtc nvram[000]: 0x34 rtc time 0x5a9ebfd2: Tue Mar 06 16:20:34 2018 Capability "hlt_exit" is set on vcpu 0 Capability "mtrap_exit" is not set on vcpu 0 Capability "pause_exit" is set on vcpu 0 Capability "unrestricted_guest" is set on vcpu 0 Capability "enable_invpcid" is set on vcpu 0 active cpus: 0 suspended cpus: 0 pending: n/a current: n/a vcpu0 stats: number of times in/out was intercepted 0 number of times cpuid was intercepted 3 vm exits due to nested page fault 13 vm exits for instruction emulation 0 number of vm exits for unknown reason 0 number of times astpending at exit 0 number of times idle requested at exit 0 number of vm exits handled in userspace 14 number of times rendezvous pending at exit 0 number of vm exits due to exceptions 3 number of NMIs delivered to vcpu 0 number of ExtINTs delivered to vcpu 0 Resident memory 69632 Wired memory 0 vcpu total runtime 3112708 EOI without any in-service interrupt 0 error interrupts generated by vlapic 0 timer interrupts generated by vlapic 0 corrected machine check interrupts generated by vlapic 0 lvts triggered[0] 0 lvts triggered[1] 0 lvts triggered[2] 0 lvts triggered[3] 0 lvts triggered[4] 0 lvts triggered[5] 0 lvts triggered[6] 0 ipis sent to vcpu[0] 0 ipis sent to vcpu[1] 0 ipis sent to vcpu[2] 0 ipis sent to vcpu[3] 0 ipis sent to vcpu[4] 0 ipis sent to vcpu[5] 0 ipis sent to vcpu[6] 0 ipis sent to vcpu[7] 0 ipis sent to vcpu[8] 0 ipis sent to vcpu[9] 0 ipis sent to vcpu[10] 0 ipis sent to vcpu[11] 0 ipis sent to vcpu[12] 0 ipis sent to vcpu[13] 0 ipis sent to vcpu[14] 0 ipis sent to vcpu[15] 0 number of ticks vcpu was idle 0 vcpu migration across host cpus 1 total number of vm exits 19 vm exits due to external interrupt 0 Number of vpid invalidations saved 0 Number of vpid invalidations done 1 number of times hlt was intercepted 0 number of times %cr access was intercepted 0 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 0 number of monitor trap exits 0 number of times pause was intercepted 0 vm exits due to interrupt window opening 0 vm exits due to nmi window opening 0 Fabian --=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJWBAEBCgBAFiEEX6JoxdmEemcFacQZmealkcs85+YFAlqewCsiHGZhYmlhbi5m cmV5ZXJAcGh5c2lrLnR1LWJlcmxpbi5kZQAKCRCZ5qWRyzzn5munD/9HLibTUH3S 0sRR4LvpKXUZIzoYkIEqHNDtrGh4Y0NaNAJKQx1J6FAzf5lZBHKkcDx83vSfbRs7 C0BT76kq+jxxCgMSPm1gxXmjOJ5wuuM64tFKalEIn8AVwdKfeCiqvL4COM3DbTds obw3MWifiBf4U8p7mZMm+AQmUxZgllmV4Lglzkq79rKWOxHjyUz7yP/sJHOUS+wN +0QP9iFjdv2DFuFJTfa9/d84nhzY6bbdajmwYD2jtGfLnCkQW2JDjkMyk6Q5YT0g fBDard4y2EyOsUx+RZpOWvC09MnuyTfZEVPbVCpyfH9tyRxjDw9ARnOOuWo1267u R742vuZrZumKWV6TygloUS6DDpc6P7DhYPCZKHsrpsBUWwq+HCd52SbcYe3WS+rb DqOHW800KLQlvopg2mQ2nix+f6Bb1pmYmJgvRcp14f+QDihjszJGjTXo8Y3npQqm JsHBXLJsa6Txo1SlESWPg/uTB5fiNt63hL7aI9ElWSeDLalvQwRfPlMLWLXxntIX fIFdRAF5d9nKAKYUYf9O9K94z8yFoIDTirFmm44zaWvwOosHk7WC0iCvdHZL71Xv pXnWOhU5AbOANZgS3xPRgPZNhn2DmJAtUCLTtBiugdzU5IVv92Wmo0ifiNbmihjx nK/IlfQ/CDc9+KcaFexiEkuqOdKaYTTqTw== =z1en -----END PGP SIGNATURE----- --=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_=-- From owner-freebsd-virtualization@freebsd.org Wed Mar 7 03:03:34 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D954F4804D for ; Wed, 7 Mar 2018 03:03:34 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) by mx1.freebsd.org (Postfix) with ESMTP id C248C795AF for ; Wed, 7 Mar 2018 03:03:33 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by vito.onthenet.com.au (Postfix) with ESMTP id 6A6B12095C5E for ; Wed, 7 Mar 2018 13:03:31 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 519D920B1AE1 for ; Wed, 7 Mar 2018 13:03:31 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 4BCAA281A4B for ; Wed, 7 Mar 2018 13:03:31 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 80kxVTW9g82y for ; Wed, 7 Mar 2018 13:03:31 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 6C776280A18; Wed, 7 Mar 2018 13:03:28 +1000 (AEST) Subject: Re: rumpkernel and bhyve: triple faults To: Fabian Freyer Cc: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org References: <651856d3-3c34-9930-cda1-62d41091f91f@freebsd.org> <2AAD4069-7D37-4325-8990-21C5DFD80B3B@physik.tu-berlin.de> From: Peter Grehan Message-ID: <1f1161ee-4543-6802-59a5-290e6c8c0329@freebsd.org> Date: Tue, 6 Mar 2018 19:03:24 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <2AAD4069-7D37-4325-8990-21C5DFD80B3B@physik.tu-berlin.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=dNCIZtRb c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=GcetEkhIoyJB3HI8B2UA:9 a=hRAa9tYipjSx2e2k:21 a=p7G98uB1is8chpi9:21 a=QEXdDO2ut3YA:10 wl=host:3 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=Mvd8FVSe c=1 sm=1 tr=0 a=mJOSnoNX3k71adV6TmU0eQ==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=GcetEkhIoyJB3HI8B2UA:9 a=hRAa9tYipjSx2e2k:21 a=p7G98uB1is8chpi9:21 a=QEXdDO2ut3YA:10 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 03:03:34 -0000 Hi Fabian, >> For a page-fault, the virtual address that resulted in the fault >> will be in the CR2 register. >=20 > I don=E2=80=99t see a CR2 register in the output of bhyvectl --get-all,= I > was looking for that too. Oops, I'll add that to bhyvectl. > I=E2=80=99m pretty sure it=E2=80=99s tooling that=E2=80=99s displaying = something off, since >hopper is showing me this as >=20 > 0x0000000000102a56 cmp word [0x2], 0x0 >=20 > Which is very similar to what r2 is giving me: >=20 > ;-- cons_init: > 0x00102a50 53 push rbx ; /arch/x86:= 43 > 0x00102a51 e8ea0a0000 call sym.hypervisor_detect ; /arch/x86:= 47 > 0x00102a56 66833da4d5ef. cmp word [0x00000002], 0 ; /arch/x86:= 62 This is reading the 16-bit value from memory location 0x2. Hard to see=20 why this would generate a page-fault - the zero page is often mapped=20 read-only. Perhaps rumpkernel doesn't have a mapping for it, but then,=20 the offset for the access would be incorrect (maybe a linking issue with=20 the location of variables ?). > Maybe I=E2=80=99m off with my analysis of the actual fault here, but ho= w I understand > the source (assuming compilers work as I would expect, which is not alw= ays true) > the values here are initialised from values in the bios data area (whic= h is > zeroed out on bhyve): It shouldn't matter that those were zero. Loading them into a memory=20 location shouldn't be a problem. > Here=E2=80=99s my full output from bhyvectl --get-all: >=20 > ID Length Name > 0 128MB sysmem > Address Length Segment Offset Prot Flags > 0 128MB sysmem 0 RWX > efer[0] 0x0000000000000500 Ok, the guest is in 64-bit mode - the LMA bit is set. This implies=20 that rumpkernel has set up it's own mappings, since the multiboot loader=20 entered the guest in flat 32-bit mode. > cr0[0] 0x0000000080010031 Paging is enabled (bit 31) as expected. later, PEter. From owner-freebsd-virtualization@freebsd.org Wed Mar 7 16:42:39 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A1EBF3DDB4 for ; Wed, 7 Mar 2018 16:42:39 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7348A7EF96 for ; Wed, 7 Mar 2018 16:42:38 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 5BF1B24318 for ; Wed, 7 Mar 2018 16:42:38 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 5016B143726; Wed, 7 Mar 2018 16:42:38 +0000 (UTC) Date: Wed, 7 Mar 2018 16:42:38 +0000 To: freebsd-virtualization@freebsd.org From: "fabian.freyer_physik.tu-berlin.de (Fabian Freyer)" Reply-to: D14473+333+002e492985d67ce8@reviews.freebsd.org Subject: [Differential] D14473: userboot: add callbacks to set unrestricted guest mode Message-ID: <05c3479bcf80d767afc3f065016357dd@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: X-Herald-Rules: <28>, <67>, none X-Phabricator-Projects: <#bhyve> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk Thread-Topic: PHID-DREV-z4kdgdlru2lrsvfkw4bo X-Phabricator-Mail-ID: 987791 X-Phabricator-Send-Attempt: yolnde3jxua5o4eu In-Reply-To: References: Thread-Index: OWY4MjZiMDdiMWRiNGU2MWQzZDM0N2I0N2FiIFqgFn4= X-Phabricator-Stamps: actor(@fabian.freyer_physik.tu-berlin.de) application(Differential) author(@fabian.freyer_physik.tu-berlin.de) herald(H28) herald(H67) monogram(D14473) object-type(DREV) phid(PHID-DREV-z4kdgdlru2lrsvfkw4bo) reviewer(#bhyve) reviewer(@grehan) reviewer(@imp) reviewer(@rgrimes) revision-status(needs-review) subscriber(#contributor_reviews_base) subscriber(@freebsd-virtualization-list) subscriber(@grehan) subscriber(@imp) tag(#bhyve) via(web) MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 16:42:39 -0000 ZmFiaWFuLmZyZXllcl9waHlzaWsudHUtYmVybGluLmRlIGFkZGVkIDEgYmxvY2tpbmcgcmV2aWV3 ZXIocyk6IGdyZWhhbi4KClJFVklTSU9OIERFVEFJTAogIGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNk Lm9yZy9EMTQ0NzMKCkVNQUlMIFBSRUZFUkVOQ0VTCiAgaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qu b3JnL3NldHRpbmdzL3BhbmVsL2VtYWlscHJlZmVyZW5jZXMvCgpUbzogZmFiaWFuLmZyZXllcl9w aHlzaWsudHUtYmVybGluLmRlLCBpbXAsIHJncmltZXMsICNiaHl2ZSwgZ3JlaGFuCkNjOiBncmVo YW4sIGltcCwgZnJlZWJzZC12aXJ0dWFsaXphdGlvbi1saXN0LCAjY29udHJpYnV0b3JfcmV2aWV3 c19iYXNlCg== From owner-freebsd-virtualization@freebsd.org Wed Mar 7 16:55:42 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DCA5F3F1F6 for ; Wed, 7 Mar 2018 16:55:42 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A61680827 for ; Wed, 7 Mar 2018 16:55:42 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 01D792477A for ; Wed, 7 Mar 2018 16:55:42 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id F31F8147AFA; Wed, 7 Mar 2018 16:55:41 +0000 (UTC) Date: Wed, 7 Mar 2018 16:55:41 +0000 To: freebsd-virtualization@freebsd.org From: "grehan (Peter Grehan)" Reply-to: D14473+333+002e492985d67ce8@reviews.freebsd.org Subject: [Differential] D14473: userboot: add callbacks to set unrestricted guest mode Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: X-Herald-Rules: <28>, <67>, none, <97> X-Phabricator-Projects: <#bhyve> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk Thread-Topic: PHID-DREV-z4kdgdlru2lrsvfkw4bo X-Phabricator-Mail-ID: 988558 X-Phabricator-Send-Attempt: 6ww5csxw64vpztnx In-Reply-To: References: Thread-Index: OWY4MjZiMDdiMWRiNGU2MWQzZDM0N2I0N2FiIFqgGY0= X-Phabricator-Stamps: actor(@grehan) application(Differential) author(@fabian.freyer_physik.tu-berlin.de) blocking-reviewer(@grehan) herald(H28) herald(H67) herald(H97) monogram(D14473) object-type(DREV) phid(PHID-DREV-z4kdgdlru2lrsvfkw4bo) reviewer(#bhyve) reviewer(@grehan) reviewer(@imp) reviewer(@rgrimes) revision-status(needs-review) subscriber(#contributor_reviews_base) subscriber(@freebsd-virtualization-list) subscriber(@grehan) subscriber(@imp) tag(#bhyve) via(web) MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 16:55:42 -0000 Z3JlaGFuIGFkZGVkIGEgY29tbWVudC4KCgogIFNvIGEgY29tbWVudCBvbiB0aGlzOiBpbiBnZW5l cmFsLCBhcGkncyBhcmUgbm90IGFkZGVkIHRvIEZyZWVCU0QgdGhhdCBkb24ndCBoYXZlIGJhc2Ut c3lzdGVtIGNsaWVudHMgdGhhdCB1c2UgdGhlbSwgb3IgZm9yIGEgZ29vZCByZWFzb24uIEkgdGhp bmsgdGhpcyBmYWxscyBpbnRvIHRoZSBsYXR0ZXIgYnV0IEknZCBsaWtlIHRvIGN1dCBpdCBkb3du IGEgYml0LgogIAogIC0gY2FuIHRoZSBnZXRfdW5yZXN0cmljdGVkX2d1ZXN0KCkgcm91dGluZSBi ZSByZW1vdmVkID8gVGhlcmUgaXMgYW4gZXJyb3IgcmV0dXJuIG9uIHRoZSBzZXQsIHdoaWNoIHNl ZW1zIGxpa2UgaXQgY2FuIGJlIHVzZWQgdG8gZGV0ZXJtaW5lIGlmIHVucmVzdHJpY3RlZCBtb2Rl IGlzIG5vdCBhdmFpbGFibGUgKGUuZy4gdGhhdCdzIGhvdyBiaHl2ZSB1c2VzIHRoZSBpb2N0bCku CiAgLSBpcyB0aGVyZSBhIG5lZWQgZm9yIHZjcHVfcmVzZXQoKSA/IFRoZSBCU1Agc2hvdWxkIGJl IGluaXRpYWxpemVkIHRvIHBvd2VyLW9uIHN0YXRlLi4gVGhhdCBjb3VsZCBiZSBhIGJ1ZyBpbiBi aHl2ZSBhbmQgYmV0dGVyIHRvIGhhdmUgaXQgZml4ZWQgdGhlcmUuCgpSRVZJU0lPTiBERVRBSUwK ICBodHRwczovL3Jldmlld3MuZnJlZWJzZC5vcmcvRDE0NDczCgpFTUFJTCBQUkVGRVJFTkNFUwog IGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9zZXR0aW5ncy9wYW5lbC9lbWFpbHByZWZlcmVu Y2VzLwoKVG86IGZhYmlhbi5mcmV5ZXJfcGh5c2lrLnR1LWJlcmxpbi5kZSwgaW1wLCByZ3JpbWVz LCAjYmh5dmUsIGdyZWhhbgpDYzogZ3JlaGFuLCBpbXAsIGZyZWVic2QtdmlydHVhbGl6YXRpb24t bGlzdCwgI2NvbnRyaWJ1dG9yX3Jldmlld3NfYmFzZQo= From owner-freebsd-virtualization@freebsd.org Wed Mar 7 17:03:14 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5522F3FF7C for ; Wed, 7 Mar 2018 17:03:13 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 839B88100F for ; Wed, 7 Mar 2018 17:03:13 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 7B6B62491C for ; Wed, 7 Mar 2018 17:03:13 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 7A2E514A956; Wed, 7 Mar 2018 17:03:13 +0000 (UTC) Date: Wed, 7 Mar 2018 17:03:13 +0000 To: freebsd-virtualization@freebsd.org From: "fabian.freyer_physik.tu-berlin.de (Fabian Freyer)" Reply-to: D14473+333+002e492985d67ce8@reviews.freebsd.org Subject: [Differential] D14473: userboot: add callbacks to set unrestricted guest mode Message-ID: <2843ee914c79e89f13a257933489caf7@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: X-Herald-Rules: <28>, <67>, none, <97> X-Phabricator-Projects: <#bhyve> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk Thread-Topic: PHID-DREV-z4kdgdlru2lrsvfkw4bo X-Phabricator-Mail-ID: 988580 X-Phabricator-Send-Attempt: n5lxdu435c2hqsnv In-Reply-To: References: Thread-Index: OWY4MjZiMDdiMWRiNGU2MWQzZDM0N2I0N2FiIFqgG1E= X-Phabricator-Stamps: actor(@fabian.freyer_physik.tu-berlin.de) application(Differential) author(@fabian.freyer_physik.tu-berlin.de) blocking-reviewer(@grehan) herald(H28) herald(H67) herald(H97) mention(@grehan) monogram(D14473) object-type(DREV) phid(PHID-DREV-z4kdgdlru2lrsvfkw4bo) reviewer(#bhyve) reviewer(@grehan) reviewer(@imp) reviewer(@rgrimes) revision-status(needs-review) subscriber(#contributor_reviews_base) subscriber(@freebsd-virtualization-list) subscriber(@grehan) subscriber(@imp) tag(#bhyve) via(web) MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 17:03:14 -0000 ZmFiaWFuLmZyZXllcl9waHlzaWsudHUtYmVybGluLmRlIGFkZGVkIGEgY29tbWVudC4KCgogIElu IEQxNDQ3MyMzMDY4MjcgPGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9EMTQ0NzMjMzA2ODI3 PiwgQGdyZWhhbiB3cm90ZToKICAKICA+IFNvIGEgY29tbWVudCBvbiB0aGlzOiBpbiBnZW5lcmFs LCBhcGkncyBhcmUgbm90IGFkZGVkIHRvIEZyZWVCU0QgdGhhdCBkb24ndCBoYXZlIGJhc2Utc3lz dGVtIGNsaWVudHMgdGhhdCB1c2UgdGhlbSwgb3IgZm9yIGEgZ29vZCByZWFzb24uIEkgdGhpbmsg dGhpcyBmYWxscyBpbnRvIHRoZSBsYXR0ZXIgYnV0IEknZCBsaWtlIHRvIGN1dCBpdCBkb3duIGEg Yml0LgogID4KICA+IC0gY2FuIHRoZSBnZXRfdW5yZXN0cmljdGVkX2d1ZXN0KCkgcm91dGluZSBi ZSByZW1vdmVkID8gVGhlcmUgaXMgYW4gZXJyb3IgcmV0dXJuIG9uIHRoZSBzZXQsIHdoaWNoIHNl ZW1zIGxpa2UgaXQgY2FuIGJlIHVzZWQgdG8gZGV0ZXJtaW5lIGlmIHVucmVzdHJpY3RlZCBtb2Rl IGlzIG5vdCBhdmFpbGFibGUgKGUuZy4gdGhhdCdzIGhvdyBiaHl2ZSB1c2VzIHRoZSBpb2N0bCku CiAgCiAgCiAgWWVzLCBpdCBjYW4uCiAgCiAgPiAtIGlzIHRoZXJlIGEgbmVlZCBmb3IgdmNwdV9y ZXNldCgpID8gVGhlIEJTUCBzaG91bGQgYmUgaW5pdGlhbGl6ZWQgdG8gcG93ZXItb24gc3RhdGUu LiBUaGF0IGNvdWxkIGJlIGEgYnVnIGluIGJoeXZlIGFuZCBiZXR0ZXIgdG8gaGF2ZSBpdCBmaXhl ZCB0aGVyZS4KICAKICBOb3QgbmVjZXNzYXJpbHksIGFzIGV2ZXJ5dGhpbmcgaW4gdmNwdV9yZXNl dCgpIGNvdWxkIGFsc28gYmUgYWNjb21wbGlzaGVkIHdpdGggdGhlIG90aGVyIGNhbGxiYWNrcy4g SSBkb24ndCB0aGluayBiaHl2ZXJ1biBzaG91bGQgY2FsbCB2Y3B1X3Jlc2V0KCksIGFzIGJoeXZl bG9hZCBzZXRzIHVwIHJlZ2lzdGVycyBiZWZvcmUuIEkgZ3Vlc3MgdGhpcyBzaG91bGQgaGFwcGVu IGJlZm9yZSBgbG9hZGVyX21haW5gIGlzIGNhbGxlZD8KClJFVklTSU9OIERFVEFJTAogIGh0dHBz Oi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9EMTQ0NzMKCkVNQUlMIFBSRUZFUkVOQ0VTCiAgaHR0cHM6 Ly9yZXZpZXdzLmZyZWVic2Qub3JnL3NldHRpbmdzL3BhbmVsL2VtYWlscHJlZmVyZW5jZXMvCgpU bzogZmFiaWFuLmZyZXllcl9waHlzaWsudHUtYmVybGluLmRlLCBpbXAsIHJncmltZXMsICNiaHl2 ZSwgZ3JlaGFuCkNjOiBncmVoYW4sIGltcCwgZnJlZWJzZC12aXJ0dWFsaXphdGlvbi1saXN0LCAj Y29udHJpYnV0b3JfcmV2aWV3c19iYXNlCg== From owner-freebsd-virtualization@freebsd.org Wed Mar 7 17:14:13 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BD46F4160B for ; Wed, 7 Mar 2018 17:14:13 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6FE681C2F for ; Wed, 7 Mar 2018 17:14:11 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w27HE3O5054614 for ; Wed, 7 Mar 2018 09:14:03 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w27HE31L054613 for freebsd-virtualization@freebsd.org; Wed, 7 Mar 2018 09:14:03 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803071714.w27HE31L054613@pdx.rh.CN85.dnsmgr.net> Subject: Call for testing bhyve cpu topology additions To: freebsd-virtualization@freebsd.org Date: Wed, 7 Mar 2018 09:14:03 -0800 (PST) X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 17:14:13 -0000 There is a new version of the CPU topology review up at http://reviews.freebsd.org/D9930 I would like to ask that if people can test this and provide feedback that they do so. It has some undesired side effects on vm-bhyve and probably other down stream tools, I am in the process of contacting the vm-bhyve author to see what can be done to clean up this output (if you are here please respond to this thread): src-topo # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTOSTART STATE issd30g1 default bhyveload cpus=8,sockets=2,cores=4,threads=1 1024M - No Stopped Notice that due to the new CPU string being much more complicated than the normal int it makes the output format ugly. I have another change to vm-bhyve that I would like to see too, and that is to move the NAME to the end of the line and remove the 16 character limit. I have done that in my local copy already. This string and its parsing are designed to be Qemu compatible. Here is a sample of my local vm-bhyve vm list output (no topology used): /tmp # vm list DATASTORE LOADER CPU MEMORY VNC AUTOSTART STATE NAME default bhyveload 1 128M - No Stopped fb-bld-10-amd64 default bhyveload 1 128M - No Stopped fb-bld-11-amd64 default bhyveload 1 128M - No Stopped fb-bld-11.0-p1-amd64 default bhyveload 1 128M - No Stopped fb-bld-11.0-p1-i386 default bhyveload 4 512M - No Stopped fb-bld-11_1-amd64 default bhyveload 4 512M - No Stopped fb-bld-11_1-i386 default bhyveload 2 256M - No Running (30227) fb-bld-head-amd64 Thoughts are to teach vm-bhyve to parse the string just as bhyve does and only output the vCPU count. Other thoughts I have are to have it parse the string and output either vCPU if cores/threads is 1, or a simple S/C/T string. If you are a downstream maintainer of one of the other vm management packages I am open to input. The implemntation I have done should allow any existing tool that treated -c as a string to use the new topology without changes. This includes the in base vmrun.sh. Also people using the sysctls: hw.vmm.topology.cores_per_package: 1 hw.vmm.topology.threads_per_core: 1 can continue to do so at this time, but there is work in process to deprecate these, that work includes making stable/11 emit a warning message if they are used, and remove them in head/12. If I can get some significant test results back I plan to commit D9930 to ^head and merge it back to stable/11 3 days later. Thanks, -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Wed Mar 7 17:30:53 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDA74F42F24 for ; Wed, 7 Mar 2018 17:30:53 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) by mx1.freebsd.org (Postfix) with ESMTP id 40D1E82DB0 for ; Wed, 7 Mar 2018 17:30:52 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by vito.onthenet.com.au (Postfix) with ESMTP id 3EAFE2092D29 for ; Thu, 8 Mar 2018 03:30:50 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 260D920B4AA6 for ; Thu, 8 Mar 2018 03:30:50 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 2057C281F1F for ; Thu, 8 Mar 2018 03:30:50 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id vSImZzrZnkF0 for ; Thu, 8 Mar 2018 03:30:50 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id F157C280504; Thu, 8 Mar 2018 03:30:47 +1000 (AEST) Subject: Re: Call for testing bhyve cpu topology additions To: "Rodney W. Grimes" References: <201803071714.w27HE31L054613@pdx.rh.CN85.dnsmgr.net> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <2b5b4f82-ee51-dfca-c505-6d15aa9fc95a@freebsd.org> Date: Wed, 7 Mar 2018 09:30:45 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <201803071714.w27HE31L054613@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=dNCIZtRb c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=CfVD2wX5spfmZtrk1A0A:9 a=QEXdDO2ut3YA:10 wl=host:3 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=Mvd8FVSe c=1 sm=1 tr=0 a=mJOSnoNX3k71adV6TmU0eQ==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=CfVD2wX5spfmZtrk1A0A:9 a=QEXdDO2ut3YA:10 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 17:30:54 -0000 > I would like to ask that if people can test this and provide > feedback that they do so. And as mentioned in the review, I'd also like to see your Windows desktop guest test results with this change. > If I can get some significant test results back I plan to commit > D9930 to ^head and merge it back to stable/11 3 days later. Standard MFC time is 3 weeks. later, Peter. From owner-freebsd-virtualization@freebsd.org Wed Mar 7 17:46:44 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58C31F44470 for ; Wed, 7 Mar 2018 17:46:44 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CFB3483D66; Wed, 7 Mar 2018 17:46:43 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w27HkfQx054759; Wed, 7 Mar 2018 09:46:41 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w27HkfU9054758; Wed, 7 Mar 2018 09:46:41 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803071746.w27HkfU9054758@pdx.rh.CN85.dnsmgr.net> Subject: Re: Call for testing bhyve cpu topology additions In-Reply-To: <2b5b4f82-ee51-dfca-c505-6d15aa9fc95a@freebsd.org> To: Peter Grehan Date: Wed, 7 Mar 2018 09:46:41 -0800 (PST) CC: freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 17:46:44 -0000 > > I would like to ask that if people can test this and provide > > feedback that they do so. > > And as mentioned in the review, I'd also like to see your Windows > desktop guest test results with this change. I do not run any windows in bhyve as bhyve can not run the windows I use due to missing/broken ATA support. The person I was helping with bhyve windows regression tests has become unavaliable. > > If I can get some significant test results back I plan to commit > > D9930 to ^head and merge it back to stable/11 3 days later. > > Standard MFC time is 3 weeks. Can you point to this some place? My understanding is that MFC is at the discretion of the committer, and the only thing the big list of rules says: 6. Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. I am making a wide call for testing, above and beyond the normal process already. I am also about to pull this back to my own 11.1 systems to ensure that I spot any merge problems and if need be attach an 11.1 and 11/stable patch to the diffential. There is also a call for testing going out at byhvecon, and there has already been a year to test on this, though perhaps not in final form, at least in functional form. Regards, -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Wed Mar 7 18:51:11 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6AD98F498DB for ; Wed, 7 Mar 2018 18:51:11 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) by mx1.freebsd.org (Postfix) with ESMTP id D8EE787B3E for ; Wed, 7 Mar 2018 18:51:10 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by vito.onthenet.com.au (Postfix) with ESMTP id 14AD42098676 for ; Thu, 8 Mar 2018 04:51:08 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id E1E7120B4A16 for ; Thu, 8 Mar 2018 04:51:07 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id DBF972809DB for ; Thu, 8 Mar 2018 04:51:07 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id vPwCWN050CGy for ; Thu, 8 Mar 2018 04:51:07 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 25F402809BC; Thu, 8 Mar 2018 04:51:05 +1000 (AEST) Subject: Re: Call for testing bhyve cpu topology additions To: "Rodney W. Grimes" Cc: freebsd-virtualization@freebsd.org References: <201803071746.w27HkfU9054758@pdx.rh.CN85.dnsmgr.net> From: Peter Grehan Message-ID: Date: Wed, 7 Mar 2018 10:51:01 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <201803071746.w27HkfU9054758@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=dNCIZtRb c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=gI7P7j3JjJIH98DZ60YA:9 a=QEXdDO2ut3YA:10 wl=host:3 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=Mvd8FVSe c=1 sm=1 tr=0 a=mJOSnoNX3k71adV6TmU0eQ==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=gI7P7j3JjJIH98DZ60YA:9 a=QEXdDO2ut3YA:10 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 18:51:11 -0000 >> And as mentioned in the review, I'd also like to see your Windows >> desktop guest test results with this change. > > I do not run any windows in bhyve as bhyve can not run the > windows I use due to missing/broken ATA support. > The person I was helping with bhyve windows regression tests has > become unavaliable. I'm staggered by this. The *only reason* for having this change is to get past the 1/2 socket limitation when running Windows desktop guests. It has no impact on any other guests, whatsoever. You can easily download an eval of Windows 10 to try this out with. You do not (and have never) required ATA support to run Windows on bhyve. >> Standard MFC time is 3 weeks. > > Can you point to this some place? My understanding is that > MFC is at the discretion of the committer, and the only > thing the big list of rules says: This is project culture, which you should have seen observing MFC times for commits. And why the rush ? I'm yet to understand what the urgency for this work is ? Who is demanding it ? later, Peter. From owner-freebsd-virtualization@freebsd.org Wed Mar 7 19:20:13 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3AD07F4BAF8 for ; Wed, 7 Mar 2018 19:20:13 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B4DE6908D; Wed, 7 Mar 2018 19:20:12 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w27JK9AG055369; Wed, 7 Mar 2018 11:20:09 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w27JK9JQ055368; Wed, 7 Mar 2018 11:20:09 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803071920.w27JK9JQ055368@pdx.rh.CN85.dnsmgr.net> Subject: Re: Call for testing bhyve cpu topology additions In-Reply-To: To: Peter Grehan Date: Wed, 7 Mar 2018 11:20:09 -0800 (PST) CC: freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 19:20:13 -0000 > >> And as mentioned in the review, I'd also like to see your Windows > >> desktop guest test results with this change. > > > > I do not run any windows in bhyve as bhyve can not run the > > windows I use due to missing/broken ATA support. > > The person I was helping with bhyve windows regression tests has > > become unavaliable. > > I'm staggered by this. The *only reason* for having this change is to > get past the 1/2 socket limitation when running Windows desktop guests. > It has no impact on any other guests, whatsoever. I shall iterate again, this change makes no change to what the guests sees if you use the old method sysctl hw.vmm.topology.cores_per_package or hw.vmm.topology.threads_per_core to set these values, it is purely a interface enhancement that makes these tuneables easy to access from userland bhyve(8). A guest can not tell the diffence in what way these are set. If hw.vmm.topology.* needs testing thats not on me, that is an existing problem, and one that has existed for far too long. > > You can easily download an eval of Windows 10 to try this out with. > You do not (and have never) required ATA support to run Windows on bhyve. I have made a call for testing, whats your issue? Are you trying to force me to do that testing? > > >> Standard MFC time is 3 weeks. > > > > Can you point to this some place? My understanding is that > > MFC is at the discretion of the committer, and the only > > thing the big list of rules says: > > This is project culture, which you should have seen observing MFC > times for commits. And I consider this change rather trivial and with near 0 risk, so choose to use a 3 day MFC timer for it. The worse that can happen is the user would have to use the old sysctls which I left in place with full compatibility. > And why the rush ? I'm yet to understand what the urgency for this > work is ? Who is demanding it ? The users have been wanting this for well over a year, it was a frequently requested item when I wrote it. It is long overdue. You seem to be raising a bar higher than any other part of FreeBSD for this rather simple user command enhancement, can I ask what your objection is? If you think the topology seen by the guest is going to be bad, that bug exists today for anyone trying to use the sysctl's, perhaps we can find that there are bugs if we get this in the hands of the user rather than yet again delay a new feature for what appears to be tests of code paths very minimally effected (change of a variable name) by this patch. Regards, -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Wed Mar 7 19:29:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A10DF279AB for ; Wed, 7 Mar 2018 19:29:03 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) by mx1.freebsd.org (Postfix) with ESMTP id 7D9AA6991F for ; Wed, 7 Mar 2018 19:29:02 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by vito.onthenet.com.au (Postfix) with ESMTP id 0A32B209867B for ; Thu, 8 Mar 2018 05:29:00 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id E066B20B4A16 for ; Thu, 8 Mar 2018 05:28:59 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id DA9EF281E00 for ; Thu, 8 Mar 2018 05:28:59 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Y1dUgqQXUJ-O for ; Thu, 8 Mar 2018 05:28:59 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 8C54F2804D6; Thu, 8 Mar 2018 05:28:57 +1000 (AEST) Subject: Re: Call for testing bhyve cpu topology additions To: "Rodney W. Grimes" Cc: freebsd-virtualization@freebsd.org References: <201803071920.w27JK9JQ055368@pdx.rh.CN85.dnsmgr.net> From: Peter Grehan Message-ID: Date: Wed, 7 Mar 2018 11:28:55 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <201803071920.w27JK9JQ055368@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=dNCIZtRb c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=n5e4Vix2roilDiFsZ4gA:9 a=ylQoYIHwv3EluTzj:21 a=GssiOoEsnPXuDgD0:21 a=QEXdDO2ut3YA:10 wl=host:3 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=Mvd8FVSe c=1 sm=1 tr=0 a=mJOSnoNX3k71adV6TmU0eQ==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=IkcTkHD0fZMA:10 a=v2DPQv5-lfwA:10 a=n5e4Vix2roilDiFsZ4gA:9 a=ylQoYIHwv3EluTzj:21 a=GssiOoEsnPXuDgD0:21 a=QEXdDO2ut3YA:10 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2018 19:29:03 -0000 > I shall iterate again, this change makes no change to what the guests > sees if you use the old method sysctl hw.vmm.topology.cores_per_package > or hw.vmm.topology.threads_per_core to set these values, it is > purely a interface enhancement that makes these tuneables easy > to access from userland bhyve(8). Those sysctls were an undocumented workaround with no error checking. You are making this a documented part of bhyve, > A guest can not tell the diffence in what way these are set. > If hw.vmm.topology.* needs testing thats not on me, that is > an existing problem, and one that has existed for far too long. Ah, no, the testing is on you, not the user community. >> You can easily download an eval of Windows 10 to try this out with. >> You do not (and have never) required ATA support to run Windows on bhyve. > > I have made a call for testing, whats your issue? > Are you trying to force me to do that testing? At a minimum, you are expected to test changes that you expect to commit. > And I consider this change rather trivial and with near 0 risk, You've never made a commit to bhyve but somehow feel qualified to make risk assessments. >> And why the rush ? I'm yet to understand what the urgency for this >> work is ? Who is demanding it ? > > The users have been wanting this for well over a year, it was > a frequently requested item when I wrote it. It is long overdue. Right, so 3 weeks for MFC is perfectly acceptable in that case. > You seem to be raising a bar higher than any other part of > FreeBSD for this rather simple user command enhancement, > can I ask what your objection is? The fact that you seem to think testing this is someone else's problem, in a subsystem where rigorous testing is a requirement. later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Mar 9 17:45:34 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A4F1F3D215 for ; Fri, 9 Mar 2018 17:45:34 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 041AA8349C for ; Fri, 9 Mar 2018 17:45:33 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.119.1] (wlan-141-23-171-244.tubit.tu-berlin.de [141.23.171.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 0407C6206E; Fri, 9 Mar 2018 17:45:32 +0000 (UTC) From: "Fabian Freyer" To: rumpkernel-users@freelists.org Cc: freebsd-virtualization@freebsd.org Subject: Re: rumpkernel and bhyve: triple faults Date: Fri, 09 Mar 2018 18:45:28 +0100 X-Mailer: MailMate (1.10r5443) Message-ID: In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_969387E7-81FE-4CE4-B9E3-F313ED190F6E_="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Mar 2018 17:45:34 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_969387E7-81FE-4CE4-B9E3-F313ED190F6E_= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 6 Mar 2018, at 7:45, Fabian Freyer wrote: > Tracking down bios_crtc_base, I find that it=E2=80=99s loaded in > rumprun/platform/hw/arch/amd64/locore.S:70: > > /* save BIOS data area values */ > movw BIOS_COM1_BASE, %bx > movw %bx, bios_com1_base > movw BIOS_CRTC_BASE, %bx > movw %bx, bios_crtc_base > > Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the= bhyve > device node in /dev/vmm with xxd(1), I find the words at these addresse= s to be > Uninitialised: > > 00000400: 0000 .. > 00000483: 0000 .. > > I=E2=80=99m not sure where to go from here. Is this a bug in bhyve(4), = should these > values be initialised somehow, or should I patch rumpkernel(7) to skip = this check > when running on bhyve(4)? I=E2=80=99ve chased this bug down a bit further to what I believe is an i= ssue with the rumprun toolchain I am building on FreeBSD with the misc/rumprun port [1]= =2E objdump -t helloer-rumprun.elf list a number of symbols in the *COM* sect= ion, which holds unallocated C external variables [2]: objdump -t helloer-rumprun.elf | grep \*COM\* 00000001 l O *COM* 00000001 pic1mask 00000004 l O *COM* 00000004 pgalloc_totalkb 00000004 l O *COM* 00000004 pgalloc_usedkb 00001000 l O *COM* 00000020 multiboot_cmdline 00000002 l O *COM* 00000002 bios_crtc_base 00000001 l O *COM* 00000001 pic2mask 00000002 l O *COM* 00000002 bios_com1_base As the pagetable in pagetable.s maps the first page as non-present, acces= sing any of these will result in a fault. I=E2=80=99m pretty sure that these shoul= dn=E2=80=99t be undefined. A build on Linux (which boots fine) shows these not to be uninitialised: 00000000003e3480 g O .bss 0000000000000002 bios_com1_base 00000000003e44a0 g O .bss 0000000000000002 bios_crtc_base Further down the rabbit hole, this goes on in rumprun.o: On Linux, bios_crtc_base is not a local symbol: 0000000000000002 O *COM* 0000000000000002 bios_crtc_base 0000000000000002 O *COM* 0000000000000002 bios_com1_base While on FreeBSD, they are marked as local: 0000000000000002 l O *COM* 0000000000000002 bios_crtc_base 0000000000000002 l O *COM* 0000000000000002 bios_com1_base Fabian [1] https://svnweb.freebsd.org/ports/head/misc/rumprun/Makefile?view=3Dma= rkup&pathrev=3D459195 [2] http://man7.org/linux/man-pages/man5/elf.5.html / SHN_COMMON --=_MailMate_969387E7-81FE-4CE4-B9E3-F313ED190F6E_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJWBAEBCgBAFiEEX6JoxdmEemcFacQZmealkcs85+YFAlqiyDgiHGZhYmlhbi5m cmV5ZXJAcGh5c2lrLnR1LWJlcmxpbi5kZQAKCRCZ5qWRyzzn5ve+EAC3okXKpevy pZecEHqzMRZvT69OtLYYADqwUWxUatoXWv+X9jzyYG+L7XGf2w1EJSEAKDbGpYb9 6pLWXk3Yd38HHL1w0nd5UhOynunw/ru9Ka6vtZYJcQHi39UOVdgqIm7s9x/0HmOv /7f25q5eDaBLRYqPFUpwLegJ46UVtCGJu2eN1EyeTifx/yNw0DTQDo96JPY0SPhE jgoX1eTggerv5Rn1O8hhToVibdNBQLcv+9uM1CET5EgWAOwEDcvZAlgpCq+eu3x8 9yEfy5UgIKMa0wgq7aUzqfGxdhLdnE5A2pHyits3TFcev/HXhzsP471Z0lAGDe9z ap+hG2gcCsEwLjA3UmjtGXD2nBO7yI+7l/me/jfKsSGhoyoA1F+PNa9ElUbmKabz wBE0tylxv1CC47r/V/vWcnvszpmIklaYg+on3AYM9Y0MCmgwKzSjE+uUsgzCkIBt mxx3JukFVGFiUQldTKhnxcorvmeArYtUf3mGh2p/wOp8ZaZ9loUJ5ZmmT046/ssC yOll/2Tbas5uuDPFeZw+iNpfM2POeQZ5Mjm/rJ3ZoO0EYb3WXo6IiDeUrkvJwW0y UDNTrx1v2jLoZi3VR5uqA3VBfng4RRkBWVb4GofM/N0jFfTaqzNbYzA37lypssxa 2e14RvWdO8gU9lWtcDCr/7Ah2NUJsizw1A== =6hrC -----END PGP SIGNATURE----- --=_MailMate_969387E7-81FE-4CE4-B9E3-F313ED190F6E_=-- From owner-freebsd-virtualization@freebsd.org Sat Mar 10 00:32:24 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90F0BF34B50 for ; Sat, 10 Mar 2018 00:32:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E6B175C8F for ; Sat, 10 Mar 2018 00:32:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8CC62154E8 for ; Sat, 10 Mar 2018 00:32:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w2A0WN6V038326 for ; Sat, 10 Mar 2018 00:32:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w2A0WN5G038325 for freebsd-virtualization@FreeBSD.org; Sat, 10 Mar 2018 00:32:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Sat, 10 Mar 2018 00:32:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mainland@apeiron.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Mar 2018 00:32:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 --- Comment #19 from mainland@apeiron.net --- I've attached a patch that fixes this regression in bug #213155. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Mar 10 15:15:32 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35DCBF40A07 for ; Sat, 10 Mar 2018 15:15:32 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lf0-x229.google.com (mail-lf0-x229.google.com [IPv6:2a00:1450:4010:c07::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8E3157794F for ; Sat, 10 Mar 2018 15:15:31 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lf0-x229.google.com with SMTP id t132-v6so750019lfe.2 for ; Sat, 10 Mar 2018 07:15:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=DohFJPVl4bcMpNSNm8k4biLxf7oh0QIT2eF9bwe4yWA=; b=FOzH5DsSyNSv9hBEm7tBn1TT3KYUxxfeFMYmqIOhB5w+lhJ+fv7f0nhV/3+9epmBf3 c93fFpwNp+3FMWu4BVqFl/YwePipc2DayzkIcxArlYFILXTHx7ywMQaMldiAmzN4NMPd Yy/pBr5sTuUeGRa3+yuAWxr474ceyaFhx45CurN7KMUJ91OEZaJGmZFJJ9kSaN6Feqm5 LnHokPdZLCXNBZc2MveOSzEJMVnwqczl9d2NFp9OoB1yyR7ufInIElUC3Phxtj288rKV /+TiVOeiImKc5OQrZkWHCt9q5meyAjfRtLHUcTFeQOxet3Uve78BCU7eLSdiSQZodOpq tmZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=DohFJPVl4bcMpNSNm8k4biLxf7oh0QIT2eF9bwe4yWA=; b=MXJh9FjYITvWxhfhDdRlbCDDvHJ2eBTDV4fCREfecMg1osBmHLoa5fud3+bjARKbaH oI0suFRkS4BLnaYiS7JR+xzFQoobXkvbdy9UosMDnsW/Em+S3ATOxmryO9fEqYoFmNqp stt3gl2LbV6qd7twEOyHRWZflYZW0d+CWazg8xNipr3xUQ8XZ77HzfyXzev97t2Q3Zu/ oq0qRGIzfGy42jpWEDsqOarBvGZtOzsbuQr1HmJ3MT44u9G+/GZLQizFi7eNeqa2kvXU rYD+uEoRESYEbaKjvYoeUo/Z+k6UgYfYsvSNHCq6HCxRXe2OLcRTLjE7dySj1xI7HhQ1 RbeQ== X-Gm-Message-State: AElRT7HshphcaA0Ep7UYwqGjULXAGW9mQkE8rOGd6lo/bfw9uT5ZgvSI WtP3nmmO/Vrp19WF3SYYmYBOwg== X-Google-Smtp-Source: AG47ELvvlk9jINpj1B+D41VOwFZ/yk0UBinLQ3hgorolS1LikvLEO2O36u0QbhSdl4T6YDUkb3eN9w== X-Received: by 2002:a19:a943:: with SMTP id s64-v6mr1495047lfe.80.1520694930102; Sat, 10 Mar 2018 07:15:30 -0800 (PST) Received: from kloomba ([31.29.232.197]) by smtp.gmail.com with ESMTPSA id v83sm798101lje.53.2018.03.10.07.15.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:15:29 -0800 (PST) Sender: Roman Bogorodskiy Date: Sat, 10 Mar 2018 19:15:22 +0400 From: Roman Bogorodskiy To: "Rodney W. Grimes" Cc: freebsd-virtualization@freebsd.org Subject: Re: Call for testing bhyve cpu topology additions Message-ID: <20180310151520.GA19476@kloomba> References: <201803071714.w27HE31L054613@pdx.rh.CN85.dnsmgr.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="X1bOJ3K7DJ5YkBrT" Content-Disposition: inline In-Reply-To: <201803071714.w27HE31L054613@pdx.rh.CN85.dnsmgr.net> User-Agent: Mutt/1.9.3 (2018-01-21) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Mar 2018 15:15:32 -0000 --X1bOJ3K7DJ5YkBrT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Rodney W. Grimes wrote: > There is a new version of the CPU topology review up at > http://reviews.freebsd.org/D9930 >=20 > I would like to ask that if people can test this and provide > feedback that they do so. >=20 > It has some undesired side effects on vm-bhyve and probably > other down stream tools, I am in the process of contacting > the vm-bhyve author to see what can be done to clean up > this output (if you are here please respond to this thread): >=20 > src-topo # vm list > NAME DATASTORE LOADER CPU MEMORY VNC = AUTOSTART STATE > issd30g1 default bhyveload cpus=3D8,sockets=3D2,cores=3D= 4,threads=3D1 1024M - No Stopped >=20 > Notice that due to the new CPU string being much more complicated than > the normal int it makes the output format ugly. I have another change > to vm-bhyve that I would like to see too, and that is to move the NAME > to the end of the line and remove the 16 character limit. I have done > that in my local copy already. This string and its parsing are designed > to be Qemu compatible. >=20 > Here is a sample of my local vm-bhyve vm list output (no topology used): > /tmp # vm list > DATASTORE LOADER CPU MEMORY VNC AUTOSTA= RT STATE NAME > default bhyveload 1 128M - No = Stopped fb-bld-10-amd64 > default bhyveload 1 128M - No = Stopped fb-bld-11-amd64 > default bhyveload 1 128M - No = Stopped fb-bld-11.0-p1-amd64 > default bhyveload 1 128M - No = Stopped fb-bld-11.0-p1-i386 > default bhyveload 4 512M - No = Stopped fb-bld-11_1-amd64 > default bhyveload 4 512M - No = Stopped fb-bld-11_1-i386 > default bhyveload 2 256M - No = Running (30227) fb-bld-head-amd64 >=20 > Thoughts are to teach vm-bhyve to parse the string just as bhyve does > and only output the vCPU count. Other thoughts I have are to have > it parse the string and output either vCPU if cores/threads is 1, > or a simple S/C/T string. >=20 > If you are a downstream maintainer of one of the other vm management pack= ages > I am open to input. The implemntation I have done should allow any exist= ing > tool that treated -c as a string to use the new topology without changes. > This includes the in base vmrun.sh. My general input on adding new features to bhyve is that it would be nice to have a way to detect if the given bhyve version supports this specific feature. In this specific case it looks like this could be checked by running bhyve -c gibberish and checking if the error message contains 'invalid cpu topology' string. But generally it would be good to have some more convenient way of doing that because running bhyve to probe each feature is pretty expensive. By the way, looking at the DR, it seems that the usage output (bhyve -h) wasn't updated, but maybe I'm missing that without context. > Also people using the sysctls: > hw.vmm.topology.cores_per_package: 1 > hw.vmm.topology.threads_per_core: 1 > can continue to do so at this time, but there is work in process to > deprecate these, that work includes making stable/11 emit a warning > message if they are used, and remove them in head/12. >=20 > If I can get some significant test results back I plan to commit > D9930 to ^head and merge it back to stable/11 3 days later. >=20 > Thanks, > --=20 > Rod Grimes rgrimes@freebs= d.org Roman Bogorodskiy --X1bOJ3K7DJ5YkBrT Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJao/aIAAoJEMltX/4IwiJq5joH/A7qOM+9sZm3SgS9rZQRjAVS bjh0H4UGA7wzYmADqfug0TZif+h0nn8Gz5rWhH858lZLeqzHA9TdJfht5D59I9xe 0r834O3slDD9tOtMoJyiQkqZxi8IsJbA3mw5q2Kk0wQ6vLguKgV3rRJlH0ZvexeK LI0NVUxwjtlodbKwF7/EjzqFmGm/qylDx90D5sZGhw9wevziUwJUvTSJXMpWX/uE 1/HHhgJuBkPi6V/6IxQzsC2Td9k47uESwfPtpNK4rTvG4fXeWPHA0IlaLCRdK0J/ mx00BblXss1FlvTyPeKuTktXT8wwakwBItbaJ00U4+FrrSGrc1anqvkZUjiV7Do= =mSPD -----END PGP SIGNATURE----- --X1bOJ3K7DJ5YkBrT-- From owner-freebsd-virtualization@freebsd.org Sat Mar 10 15:38:36 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9280F426B6 for ; Sat, 10 Mar 2018 15:38:36 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 53035783A0; Sat, 10 Mar 2018 15:38:35 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w2AFcVJa070003; Sat, 10 Mar 2018 07:38:31 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w2AFcV9N070002; Sat, 10 Mar 2018 07:38:31 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803101538.w2AFcV9N070002@pdx.rh.CN85.dnsmgr.net> Subject: Re: Call for testing bhyve cpu topology additions In-Reply-To: <20180310151520.GA19476@kloomba> To: Roman Bogorodskiy Date: Sat, 10 Mar 2018 07:38:31 -0800 (PST) CC: freebsd-virtualization@FreeBSD.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Mar 2018 15:38:37 -0000 > Rodney W. Grimes wrote: > > > There is a new version of the CPU topology review up at > > http://reviews.freebsd.org/D9930 > > > > I would like to ask that if people can test this and provide > > feedback that they do so. > > > > It has some undesired side effects on vm-bhyve and probably > > other down stream tools, I am in the process of contacting > > the vm-bhyve author to see what can be done to clean up > > this output (if you are here please respond to this thread): > > > > src-topo # vm list > > NAME DATASTORE LOADER CPU MEMORY VNC AUTOSTART STATE > > issd30g1 default bhyveload cpus=8,sockets=2,cores=4,threads=1 1024M - No Stopped > > > > Notice that due to the new CPU string being much more complicated than > > the normal int it makes the output format ugly. I have another change > > to vm-bhyve that I would like to see too, and that is to move the NAME > > to the end of the line and remove the 16 character limit. I have done > > that in my local copy already. This string and its parsing are designed > > to be Qemu compatible. > > > > Here is a sample of my local vm-bhyve vm list output (no topology used): > > /tmp # vm list > > DATASTORE LOADER CPU MEMORY VNC AUTOSTART STATE NAME > > default bhyveload 1 128M - No Stopped fb-bld-10-amd64 > > default bhyveload 1 128M - No Stopped fb-bld-11-amd64 > > default bhyveload 1 128M - No Stopped fb-bld-11.0-p1-amd64 > > default bhyveload 1 128M - No Stopped fb-bld-11.0-p1-i386 > > default bhyveload 4 512M - No Stopped fb-bld-11_1-amd64 > > default bhyveload 4 512M - No Stopped fb-bld-11_1-i386 > > default bhyveload 2 256M - No Running (30227) fb-bld-head-amd64 > > > > Thoughts are to teach vm-bhyve to parse the string just as bhyve does > > and only output the vCPU count. Other thoughts I have are to have > > it parse the string and output either vCPU if cores/threads is 1, > > or a simple S/C/T string. > > > > If you are a downstream maintainer of one of the other vm management packages > > I am open to input. The implemntation I have done should allow any existing > > tool that treated -c as a string to use the new topology without changes. > > This includes the in base vmrun.sh. > > My general input on adding new features to bhyve is that it would be > nice to have a way to detect if the given bhyve version supports this > specific feature. > > In this specific case it looks like this could be checked by running > > bhyve -c gibberish > > and checking if the error message contains 'invalid cpu topology' > string. > > But generally it would be good to have some more convenient way of doing > that because running bhyve to probe each feature is pretty expensive. I agree that this is a nice thing to have, but I can not think of any other command that implements that feature. In general command line interfaces are for human consumption. This one though is being used extensivly by other scripts and programs. > > By the way, looking at the DR, it seems that the usage output (bhyve -h) > wasn't updated, but maybe I'm missing that without context. Good catch, I infact did miss that. I'll update it and push a new patch to the review. Thank you. > > Also people using the sysctls: > > hw.vmm.topology.cores_per_package: 1 > > hw.vmm.topology.threads_per_core: 1 > > can continue to do so at this time, but there is work in process to > > deprecate these, that work includes making stable/11 emit a warning > > message if they are used, and remove them in head/12. > > > > If I can get some significant test results back I plan to commit > > D9930 to ^head and merge it back to stable/11 3 days later. > > > > Thanks, > > -- > > Rod Grimes rgrimes@freebsd.org > > Roman Bogorodskiy Regards, -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Sat Mar 10 22:55:51 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D178F3C04A for ; Sat, 10 Mar 2018 22:55:51 +0000 (UTC) (envelope-from martin@lucina.net) Received: from smtp.lucina.net (smtp.lucina.net [62.176.169.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E37EA87DD9 for ; Sat, 10 Mar 2018 22:55:50 +0000 (UTC) (envelope-from martin@lucina.net) Received: from nodbug.lucina.net (dynrak234g-65-4-67-105.inwitelecom.net [105.67.4.65]) by smtp.lucina.net (Postfix) with ESMTPSA id B0BC9122804; Sat, 10 Mar 2018 23:46:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lucina.net; s=dkim-201309; t=1520721969; bh=BRUVOrvnGAPexJCTG1fOM0EbqWA8R/IFXQPBn/fRiHw=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=gIP8CV9GABkj0BF2QtE+mMUtrTLEOTpj5G/xjtpN0YBpPCa8nRZBxj9/O/LwuXRr2 Rmb29vpiGZatFU+SwexNA0aoL7ZcmZe7ryWRQsVuGvuuqVVdW1bWnTOKeKJfUNThnT N9KLYLLB7agdOOQU1+btprYoGlE6WMm9jZWSEH4b+Wg+E0puBxtBq9yJtENqEbpM3z MiqXR4s2H1rqXMYcZa4Cs45rwTExnYzr2NRPv2WAwYjR2bcuA7mOkTYtPr3461qyAZ 9C3XmIV9KCPibb0aNs/L6iQC3KFwydhqYZgul+TZhjNMB/sYu1xtGnVFqW7o0pAtoq l6LngMaesuLOA== Received: by nodbug.lucina.net (Postfix, from userid 1000) id 9BD882175E7B; Sat, 10 Mar 2018 23:46:07 +0100 (CET) Date: Sat, 10 Mar 2018 23:46:07 +0100 From: Martin Lucina To: Fabian Freyer Cc: rumpkernel-users@freelists.org, freebsd-virtualization@freebsd.org Subject: Re: rumpkernel and bhyve: triple faults Message-ID: <20180310224607.wscuqebheq5bjxww@nodbug.lucina.net> Mail-Followup-To: Fabian Freyer , rumpkernel-users@freelists.org, freebsd-virtualization@freebsd.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: NeoMutt/20170113 (1.7.2) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Mar 2018 22:55:51 -0000 Hi, On Friday, 09.03.2018 at 18:45, Fabian Freyer wrote: > On 6 Mar 2018, at 7:45, Fabian Freyer wrote: > > Tracking down bios_crtc_base, I find that it’s loaded in > > rumprun/platform/hw/arch/amd64/locore.S:70: > > > > /* save BIOS data area values */ > > movw BIOS_COM1_BASE, %bx > > movw %bx, bios_com1_base > > movw BIOS_CRTC_BASE, %bx > > movw %bx, bios_crtc_base > > > > Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve > > device node in /dev/vmm with xxd(1), I find the words at these addresses to be > > Uninitialised: > > > > 00000400: 0000 .. > > 00000483: 0000 .. > > > > I’m not sure where to go from here. Is this a bug in bhyve(4), should these > > values be initialised somehow, or should I patch rumpkernel(7) to skip this check > > when running on bhyve(4)? You probably want to use a serial console rather than VGA on bhyve in any case, so you'll want to add the appropriate checks to hypervisor.c and cons.c. > I’ve chased this bug down a bit further to what I believe is an issue with the > rumprun toolchain I am building on FreeBSD with the misc/rumprun port [1]. > > objdump -t helloer-rumprun.elf list a number of symbols in the *COM* section, which > holds unallocated C external variables [2]: > > objdump -t helloer-rumprun.elf | grep \*COM\* > 00000001 l O *COM* 00000001 pic1mask > 00000004 l O *COM* 00000004 pgalloc_totalkb > 00000004 l O *COM* 00000004 pgalloc_usedkb > 00001000 l O *COM* 00000020 multiboot_cmdline > 00000002 l O *COM* 00000002 bios_crtc_base > 00000001 l O *COM* 00000001 pic2mask > 00000002 l O *COM* 00000002 bios_com1_base > > As the pagetable in pagetable.s maps the first page as non-present, accessing any > of these will result in a fault. I’m pretty sure that these shouldn’t be undefined. > > A build on Linux (which boots fine) shows these not to be uninitialised: > 00000000003e3480 g O .bss 0000000000000002 bios_com1_base > 00000000003e44a0 g O .bss 0000000000000002 bios_crtc_base When you write "which boots fine", I presume you're referring to booting on bhyve? > Further down the rabbit hole, this goes on in rumprun.o: > > On Linux, bios_crtc_base is not a local symbol: > 0000000000000002 O *COM* 0000000000000002 bios_crtc_base > 0000000000000002 O *COM* 0000000000000002 bios_com1_base > > While on FreeBSD, they are marked as local: > 0000000000000002 l O *COM* 0000000000000002 bios_crtc_base > 0000000000000002 l O *COM* 0000000000000002 bios_com1_base That seems wrong. Can you try and force the toolchain to use the more recent GNU ld from devel/binutils and see if that fixes the problem? -mato