From owner-freebsd-virtualization@freebsd.org Thu Aug 16 20:43:56 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72FE01076611 for ; Thu, 16 Aug 2018 20:43:56 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-b.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D81E278ED4; Thu, 16 Aug 2018 20:43:55 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-b.userve.net (Postfix) with ESMTPS id 928281E7; Thu, 16 Aug 2018 21:43:54 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1534452234; bh=jJiRLLAEn87nToRDzh8nFZSzd6xPWYcOrJ32RXquS4Q=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=qfEvLnmZX3S6rip6zaW66ag8HvS16+KIEAOqd2s1kUH/aG3DpqSmIHhLzUJj5iaEJ E6H4jo4VEJz5iYRPxU4PNxaBzyB03JtU4pFI/TplzO3MGXlhCJYXaDJOtahW4cdTsa pJyhxzQYqD69nikeOlMHkqbfTgw1Jcujb+Z2nNs8= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Thu, 16 Aug 2018 21:43:53 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Thu, 16 Aug 2018 21:43:53 +0100 From: Matt Churchyard To: "araujo@freebsd.org" CC: Allan Jude , "freebsd-virtualization@freebsd.org" , "Rodney W. Grimes" Subject: Re: Checking bhyve supported features (sysctls) Thread-Topic: Checking bhyve supported features (sysctls) Thread-Index: AdQ1Sj/JzUfq4S8/RYq4DRXDNWTckQAIYkuAAANnXRD///SbAIAABAyAgAAHIQCAACHuAIAALy0w Date: Thu, 16 Aug 2018 20:43:53 +0000 Message-ID: <12D64664-2135-4D60-B534-5DACCB839A08@userve.net> References: <201808161628.w7GGS52P054505@pdx.rh.CN85.dnsmgr.net> , In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2018 20:43:56 -0000 On 16 Aug 2018, at 19:55, Marcelo Araujo > wrote: 2018-08-17 0:53 GMT+08:00 Allan Jude >: On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" > wrote: >> >> Text manually wrapped to 80, any broken quoting is my fault - rwg >> >> > > Hello, >> > > >> > > I'm looking for better ways to check for bhyve support / >available >> > > features without trying to scan through dmesg output. >> > >> > >Yes, it would be very good to remove that, as it usually tries >> > >to grep a non-existent file /var/run/dmesg.boot that is not >> > >created until after vm_bhyve has been called from >/usr/local/etc/rc.d >> > >when you have things set to autostartup >in /etc/rc.conf >> > >> > >> > > >> > > I notice that the following 2 sysctl's appear to be set to 1 as >soon >> > > as the vmm module is loaded >> > > >> > > hw.vmm.vmx.initialized: 1 >> > > hw.vmm.vmx.cap.unrestricted_guest: 1 >> > > >> > > Will these be available on both Intel & AMD processors as a way >> > > to determine if the module has loaded successfully and can run >guests? >> > > >> > > I also see the below sysctl related to iommu. >> > > >> > > hw.vmm.iommu.initialized >> > > >> > > Again, will this be set to 1 as soon as the module is loaded if >> > > iommu is supported, or only when it is used? >> > > There also seems to be a vmm.amdvi.enable sysctl. >> > > Would both these need checking or is vmm.iommu enough to >> > > determine support on any processor. >> > >> > >Probalby the safest way for a shell script to decide if bhyve is >> > >up and running is to stat /dev/vmm, if that exists then the >modules >> > >have loaded and initialized and bhyve should be ready to process >guests. >> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests. >> >> I'll investigate that, I was pretty sure that you should get this >> as soon as the vmm.ko module is finished initialzing, but you might >> be right in that it takes a first vm to cause its creation. >> Confirmed, /dev/vmm does not exist until the first vm >> is created. >> >> > >> > >sysctl's mentiond above would be a poor way to make this >determination. >> > >> > It would be nice if sysctls were better documented. >> >> Agreed. >> >> > If vmx.initialized is set once vmm has successfully loaded, I can't >see a better way of checking for bhyve support (assuming it's not Intel >specific). This entry definitely exists and is set to 0 if you load the >module on a non-supported system, and set to 1 as soon as vmm loads on >my Intel test system. >> >> Given its undocumented status you would be relying on an >> undocumented feature that could change in either name or >> behavior, and that is not desirable. >> >> Let me see if I can come up with something else. > >I looked at the code for bhyvectl, bhyveload and >byhve. They do not actually try to decide if vmm >is supported or not, they simply process the error >from a vm_create() or vm_open() call and exit >with an error code if they can not handle it >(some of the code can handle a vm_create failure >if infact we are trying to create a vm that >already exists). > >If you want to maintain full compatibility a similiar >stratergy may be in order. > >Why is it that vm-bhyve specifically needs to know >if the kernel has vmm support or not? >Cant it just be written to handle the errors returned >if the supported functions do not exist? I think the question vm-bhyve wants to answer is: does the CPU have the req= uired features to run a multicore VM. These or similar sysctls do seem to be the correct way to communicate that = support. You are correct! The question in case as I understood was about CPU feature supported, actua= lly vmm(8) knows all this information! Some examples such like CPU with VMX= unrestricted mode support (UG) that is necessary for guest VMs running wit= h multiple vCPU or like VT-d necessary for PCI device passthrough. I have a patch that exposes a sysctl saying what bhyve(8) is capable to run= , however it needs to be polished a bit more to be more informative. I think for third part software like vm-bhyve these information are crucial= as these software can get advantage of these information prior to run a ce= rtain set that will end up in a fail because of a partial CPU support. Best, As mentioned in my first email, it does seem like some of these exist alrea= dy in the way of vmm.vmx.cap.* sysctls. We could look at bhyve output and try to process that, but that seems more = messy if there are sysctls that expose support, especially as the vmm module does seem to = know what features the cpu/hardware supports. vm-bhyve has already forked into the ba= ckground by the time bhyve runs so can't easily provide feedback to the caller other= than through the log file. We do also try and take action in some cases, such as reducing cpu count to= 1 if UG support isn't found, rather than just having bhyve fail. (Of course you could argue= we should just exit with an error and let the user decide if they want to drop the cpu cou= nt to 1. We could just do nothing, let bhyve run and if it falls over people can use= debug mode and see the bhyve output themselves in the log. Just seems useful to be able to= tell users that their hardware doesn't support the features they are trying to use up = front. It still seems to me that vmx.initialised is a reasonable indicator that vm= m has loaded without issue, but it would be useful to have some documented way of checki= ng exactly what virt features the system supports, without just running someth= ing and seeing if it falls over. Matt -- Allan Jude _______________________________________________ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebs= d.org" -- -- Marcelo Araujo (__) araujo@FreeBSD.org \\\'',) http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_)