From owner-freebsd-virtualization@freebsd.org Thu Sep 10 14:46:34 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D3AE3D960D for ; Thu, 10 Sep 2020 14:46:34 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BnMCY2YqJz460d; Thu, 10 Sep 2020 14:46:32 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.159] (cpe-23-243-161-111.socal.res.rr.com [23.243.161.111]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 2476281e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Sep 2020 14:46:31 +0000 (UTC) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Peter Grehan Cc: FreeBSD virtualization References: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> <5e62480f-14cb-5963-f753-8562aab381e3@nomadlogic.org> From: Pete Wright Message-ID: Date: Thu, 10 Sep 2020 07:46:31 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BnMCY2YqJz460d X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-2.55 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nomadlogic.org]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.018]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.25)[-0.252]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RECEIVED_SPAMHAUS_PBL(0.00)[23.243.161.111:received] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 14:46:34 -0000 u On 9/10/20 12:30 AM, Peter Grehan wrote: > Hi Pete, > >> Sep 10 03:25:01 bells kernel: vmx_init: processor does not support >> desired basic capabilities > >  ^^^  This is what's causing the issues, and is covered in the first > bug report I mentioned > (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203994) > >  Would you be able to dump the MSRs for the FreeBSD guest as mentioned > in the bug report ? > Ah I see - once I realized we had a newer kernel I didn't focus too much on the first bug - here's the MSR's MSR 0x480: 0x00981000 0x11e57ed0 MSR 0x482: 0xfff9fffe 0x0401e172 MSR 0x48b: 0x000003fb 0x00000000 MSR 0x48c: 0x00000f01 0x04114040 >> since you mention this is a possible bug with KVM is there some >> useful information i can relay to my ISP (they are a BSD friendly >> vendor) that they can try on their end as a workaround, or perhaps >> something they ping the linux/kvm community about? > >  As above, this isn't the issue for this particular case: it manifests > as a hypervisor guest immediately exiting when started. > ok perfect this aligns with what i'm seeing as well when i run bhyvload or bhyvectl and it just bails with a "vm_create: Device not configured" error. Cheers, -p -- Pete Wright pete@nomadlogic.org @nomadlogicLA