From owner-freebsd-virtualization@freebsd.org Fri Aug 17 09:14:31 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1ECEF1087C8E for ; Fri, 17 Aug 2018 09:14:31 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EA4F73C00; Fri, 17 Aug 2018 09:14:30 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lj1-x22b.google.com with SMTP id l15-v6so5825903lji.6; Fri, 17 Aug 2018 02:14:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=jOfF9FEfUfcrohxnRAfyR+IcwCjhr0qpJrO3q9GkM+U=; b=exqqkCsbi0tljWk6bHNRMPyumS0l8YGLBdUDdNz4ZX2VsAk21HXwFnMPpista88AwA UPkB68G03IEgVAD+aCyc0AdQZYZKReMhxec3rzYqgkGH38Q7Bxn3qotg1IRF72Iml7lq vbbub/e/DiYT0/sWQgm3aT4/0mPDHMn4dpG2dnbVDnKBsHm5bGKseLnfjreKMp3iRufv g2QZRGT1+Eeo82pGOhC2/N1HtD+S/LSsRo7kW4SWBOufXwzMHR8i6YkfRAtQpvP1jTB+ VdsHaGW6R0W5kt2e74FXH/2SgzIgyeFggCZWw3dwz043nTKsSnFyUEcGU+6FJw8J3arm 6/DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=jOfF9FEfUfcrohxnRAfyR+IcwCjhr0qpJrO3q9GkM+U=; b=sMfjegc21iGDt37OVc5cXDNtzHmOxngP+SplIipx+geJjSwtfsBhgIcO5LzKB/Nfd5 VLZtBBDuWhTnx0XMxThxqwvHSTKlec/22sdaOIreSBpV5FZmP5J8fQCsBPvyn1E/VjDi ZlOrjxUFMYEDqgmesshmYNYgYeo28WRdeQ7ZOnZ1XqpGl+BaBWVKuHU3utCo6bcS42es Mw/KU3rwTKfHN38mFDk3OxQDnOsBawNCinh1gvpH7auh4oQJ6DSQ4mdLp2BpDhjV/G9t UkW0F6ohV/JVm5N5qrNhgFvzIgA5ydxRaS49lm2+RbcRE6GA1Y4Q/nINmEv3act6UWwx Og9Q== X-Gm-Message-State: AOUpUlE6+KOdhlJLlKh/+1TjdBagbTHTlMlS7QeBbw5g/mq58lEBhD34 P5YfVCuqg0RH9H+7No9T3Y4eNnBVrnSUasEbDrqLYA== X-Google-Smtp-Source: AA+uWPyQl50naROas8ThV7G/Q1SFqPPxoaApOWKcKADMK2TCP7v9cClD8vX8Cn/uLL3lfJdkFehYwi+f2z7IQMO2QfU= X-Received: by 2002:a2e:9599:: with SMTP id w25-v6mr1141290ljh.6.1534497268289; Fri, 17 Aug 2018 02:14:28 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:1f4c:0:0:0:0:0 with HTTP; Fri, 17 Aug 2018 02:14:27 -0700 (PDT) Reply-To: araujo@freebsd.org In-Reply-To: <8fad62d974804195bf4518a88b7c53a8@SERVER.ad.usd-group.com> References: <201808161730.w7GHUaWv054788@pdx.rh.CN85.dnsmgr.net> <8fad62d974804195bf4518a88b7c53a8@SERVER.ad.usd-group.com> From: Marcelo Araujo Date: Fri, 17 Aug 2018 17:14:27 +0800 Message-ID: Subject: Re: Checking bhyve supported features (sysctls) To: Matt Churchyard Cc: Allan Jude , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2018 09:14:31 -0000 2018-08-17 16:54 GMT+08:00 Matt Churchyard : > > > > > 2018-08-17 16:25 GMT+08:00 Matt Churchyard : > > -----Original Message----- > From: owner-freebsd-virtualization@freebsd.org < > owner-freebsd-virtualization@freebsd.org> On Behalf Of Rodney W. Grimes > Sent: 16 August 2018 18:31 > To: Allan Jude > Cc: Matt Churchyard ; freebsd-virtualization@ > freebsd.org > Subject: Re: Checking bhyve supported features (sysctls) > > > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" < > freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: > > >> > > >> Text manually wrapped to 80, any broken quoting is my fault - rwg > > >> > > >> > > Hello, > > >> > > > > >> > > I'm looking for better ways to check for bhyve support / > > >available > > >> > > features without trying to scan through dmesg output. > > >> > > > >> > >Yes, it would be very good to remove that, as it usually tries > > >> > >to grep a non-existent file /var/run/dmesg.boot that is not > > >> > >created until after vm_bhyve has been called from > > >/usr/local/etc/rc.d > > >> > >when you have things set to autostartup >in /etc/rc.conf > > >> > > > >> > > > >> > > > > >> > > I notice that the following 2 sysctl's appear to be set to 1 as > > >soon > > >> > > as the vmm module is loaded > > >> > > > > >> > > hw.vmm.vmx.initialized: 1 > > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1 > > >> > > > > >> > > Will these be available on both Intel & AMD processors as a way > > >> > > to determine if the module has loaded successfully and can run > > >guests? > > >> > > > > >> > > I also see the below sysctl related to iommu. > > >> > > > > >> > > hw.vmm.iommu.initialized > > >> > > > > >> > > Again, will this be set to 1 as soon as the module is loaded if > > >> > > iommu is supported, or only when it is used? > > >> > > There also seems to be a vmm.amdvi.enable sysctl. > > >> > > Would both these need checking or is vmm.iommu enough to > > >> > > determine support on any processor. > > >> > > > >> > >Probalby the safest way for a shell script to decide if bhyve is > > >> > >up and running is to stat /dev/vmm, if that exists then the > > >modules > > >> > >have loaded and initialized and bhyve should be ready to process > > >guests. > > >> > > > >> > Hmm, I don't get /dev/vmm unless I actually have running guests. > > >> > > >> I'll investigate that, I was pretty sure that you should get this > > >> as soon as the vmm.ko module is finished initialzing, but you might > > >> be right in that it takes a first vm to cause its creation. > > >> Confirmed, /dev/vmm does not exist until the first vm is created. > > >> > > >> > > > >> > >sysctl's mentiond above would be a poor way to make this > > >determination. > > >> > > > >> > It would be nice if sysctls were better documented. > > >> > > >> Agreed. > > >> > > >> > If vmx.initialized is set once vmm has successfully loaded, I > > >> > can't > > >see a better way of checking for bhyve support (assuming it's not > > >Intel specific). This entry definitely exists and is set to 0 if you > > >load the module on a non-supported system, and set to 1 as soon as > > >vmm loads on my Intel test system. > > >> > > >> Given its undocumented status you would be relying on an > > >> undocumented feature that could change in either name or behavior, > > >> and that is not desirable. > > >> > > >> Let me see if I can come up with something else. > > > > > >I looked at the code for bhyvectl, bhyveload and byhve. They do not > > >actually try to decide if vmm is supported or not, they simply > > >process the error from a vm_create() or vm_open() call and exit with > > >an error code if they can not handle it (some of the code can handle > > >a vm_create failure if infact we are trying to create a vm that > > >already exists). > > > > > >If you want to maintain full compatibility a similiar stratergy may > > >be in order. > > > > > >Why is it that vm-bhyve specifically needs to know if the kernel has > > >vmm support or not? > > >Cant it just be written to handle the errors returned if the > > >supported functions do not exist? > > > > I think the question vm-bhyve wants to answer is: does the CPU have > > the required features to run a multicore VM. > > >Why does it need to know that? If it tries to start a multicore/thread > VM and the system can not support it an error is returned and the bhyve > command fails. > > >Actually determining that specific issue is non-trivial iirc as some vmm > supported CPU's can only run a single VM with a single thread in that VM > (early core cpu's). > > > > > These or similar sysctls do seem to be the correct way to communicate > > that support. > > >I do not believe any of those sysctl's communicate that your on a broken > cpu or to what extent you can run vm's with multiple threads. > > So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is > not a valid way to determine if the host has unrestricted guest support > (required for non-freebsd or multicore freebsd guests, and as you say > missing from some early VT-x capable processors)? > > >I went and looked at why vm-bhyve is groveling around in > /var/run/dmesg.boot and found that it is simply trying to determine if th= e > host CPU is vmm capable, > >specifically: > >util::check_bhyve_support(){ > >... > > >These checks are already built into the kernel. > >This can all go in the bit bucket, if you try to start a VM on an > unsupported system an error is returned, recoding this in shell is just > setting yourself up for "future" bugs. > > The kernel knows what features are supported but does not expose these, s= o > all I can do is similar to libvirt and run bhyve with different options t= o > see what errors pop up. > I think I'll just remove all checking for now and let users discover the > issue for themselves if bhyve won't run. Hopefully the vmx.initialized / > cap.* sysctls will at some point become a defined way of seeing if vmm is > ready / testing for vmm features, as apparently these serve no purpose at > the moment. > > Matt > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > > > > >This is something that is in my long to-do list, I will try to get back > to this in couple weeks. > > >I think the way how you are dealing with it nowadays is the best way to > try to discover the CPU capabilities, better in this way than let the use= rs > blind. > > >Best, > > Thanks Marcelo, > > > > It=E2=80=99s not exactly critical. I just saw another project using > vmx.initialized and it seemed a much cleaner way to determine support by > seeing if vmm loads rather than probing log files. I asked here expecting > someone to just say =E2=80=9Cyes if that=E2=80=99s 1 then the host will r= un guests=E2=80=9D, or =E2=80=9Cno > you can=E2=80=99t tell if guests will run just by that sysctl=E2=80=9D. > > > > Matt > Could you share with me what project is using vmx.initialized? Best, --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_)