From owner-freebsd-virtualization@freebsd.org Fri Jul 31 22:38:27 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 31C213A4BB7 for ; Fri, 31 Jul 2020 22:38:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito-vars.onthenet.com.au (vito-vars.OntheNet.com.au [203.13.68.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BJMcy3LT3z4324 for ; Fri, 31 Jul 2020 22:38:26 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito-vars.onthenet.com.au (Postfix) with ESMTPS id 52A6C20C931A for ; Sat, 1 Aug 2020 08:38:15 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 3B22920AE97B for ; Sat, 1 Aug 2020 08:38:15 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 32E98213C9A1 for ; Sat, 1 Aug 2020 08:38:15 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id B_ToleF1egQr for ; Sat, 1 Aug 2020 08:38:15 +1000 (AEST) Received: from MacBook-Air-4.local (CPE-120-29-60-82.dsl.OntheNet.net [120.29.60.82]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 98897213C9A0; Sat, 1 Aug 2020 08:38:14 +1000 (AEST) Subject: rdtscp support (was Re: bhyve guest illegal instruction) To: Chuck Tuffli References: From: Peter Grehan Cc: John Baldwin , freebsd-virtualization@freebsd.org Message-ID: Date: Sat, 1 Aug 2020 08:38:14 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=bM+N3M2Z c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5na4vOg5aJ1be1vsGc0NmA==:17 a=IkcTkHD0fZMA:10 a=y4yBn9ojGxQA:10 a=BCdpW5SkYIIuyLzi6PkA:9 a=QEXdDO2ut3YA:10 a=pHzHmUro8NiASowvMSCR:22 a=Ew2E2A-JSTLzCXPT_086:22 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=LtAJNkVc c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=5na4vOg5aJ1be1vsGc0NmA==:17 a=IkcTkHD0fZMA:10 a=y4yBn9ojGxQA:10 a=BCdpW5SkYIIuyLzi6PkA:9 a=QEXdDO2ut3YA:10 a=pHzHmUro8NiASowvMSCR:22 a=Ew2E2A-JSTLzCXPT_086:22 X-Rspamd-Queue-Id: 4BJMcy3LT3z4324 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/23, country:AU] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2020 22:38:27 -0000 Hi Chuck, > 12 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0time1 =3D __builtin_= ia32_rdtscp(&dummy); rdtscp shouldn't be used without checking that it's available via=20 CPUID first, but as you mentioned the feature is available on the host,=20 just hidden from the guest. > This same program works on the FreeBSD 12-stable machine hosting the VM= =20 > as well as another bare-metal Linux host. Poking around in the vmm code= ,=20 > I found > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 /* > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0* Hide rdtscp/ia32_tsc_aux until we know how > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0* to deal with them. > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0*/ > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 regs[3] &=3D ~AMDID_RDTSCP; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 break; > in sys/amd64/vmm/x86.c which I _think_ is relevant because lscpu doesn'= t=20 > show the rdtscp flag. If this is the root cause, what would need to be=20 > done to implement this? At a quick glance, if the feature is available on the host you'd need t= o - expose it via CPUID - save/restore the TSC_AUX MSR, but using the VMCS MSR h/w=20 save/restore mechanism that will have to be resurrected. (this avoids=20 any preemption issues,even at NMI level). - set the "enable RDTSCP" VM-execution control to one in the VMCS That being said, I've heard anecdotally that rdtscp results in=20 VM-exits on other hypervisors so there may be reason to emulate it=20 rather than allow a pass-thru. More investigation may be required. later, Peter.