From owner-freebsd-virtualization@freebsd.org Mon May 17 09:40:12 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D472641ACB for ; Mon, 17 May 2021 09:40:12 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "CN", Issuer "CN" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FkDd73yXlz3hJT for ; Mon, 17 May 2021 09:40:10 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.15.2/8.15.2) with ESMTP id 14H9e0mL019074; Mon, 17 May 2021 11:40:01 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 11571F61; Mon, 17 May 2021 11:40:00 +0200 (CEST) Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root To: Konstantin Belousov , Wei Hu Cc: Chris Knight , "freebsd-virtualization@freebsd.org" References: From: Harry Schmalzbauer Organization: OmniLAN Message-ID: <22bbe4cd-82ed-b826-9ac0-05ef8aceba0e@omnilan.de> Date: Mon, 17 May 2021 11:39:59 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Greylist: inspected by milter-greylist-4.6.2 (mx0.gentlemail.de [78.138.80.130]); Mon, 17 May 2021 11:40:03 +0200 (CEST) for IP:'78.138.80.135' DOMAIN:'mh0.gentlemail.de' HELO:'mh0.gentlemail.de' FROM:'freebsd@omnilan.de' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx0.gentlemail.de [78.138.80.130]); Mon, 17 May 2021 11:40:03 +0200 (CEST) X-Rspamd-Queue-Id: 4FkDd73yXlz3hJT X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [0.18 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_TO(0.00)[gmail.com,microsoft.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:61157, ipnet:2a00:e10:2800::/38, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:e10:2800::a130:from]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.77)[0.769]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[omnilan.de]; NEURAL_SPAM_MEDIUM(0.71)[0.712]; SPAMHAUS_ZRD(0.00)[2a00:e10:2800::a130:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 May 2021 09:40:12 -0000 Am 17.05.2021 um 08:47 schrieb Konstantin Belousov: > On Mon, May 17, 2021 at 06:33:19AM +0000, Wei Hu wrote: >>> -----Original Message----- >>> From: Konstantin Belousov >>> Sent: Monday, May 17, 2021 1:47 PM >>> To: Wei Hu >>> Cc: Harry Schmalzbauer ; Chris Knight >>> ; freebsd-virtualization@freebsd.org >>> Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root >>> >>> On Mon, May 17, 2021 at 04:12:20AM +0000, Wei Hu wrote: : : : >>>> separately. The workaround for this on amd64 is either build these >>>> Hyper-V modules into the kernel or completely disable mitigation by >>>> setting vm.pmap.pti="0" in /boot/loader.conf in the affect FreeBSD guests. >>>> Just to share some hopefully helpful pre-check's I've done so far: Disabling PTI via vm.pmap.pti="0" in /boot/loader.conf doesn't make any difference for the i386 issue (at least x86/x86/local_apic.c always uses non-pti path; tried nontheless, with identical result). As far as my analysis isn't completely wrong, the problem is about IDT pointer. IDTVEC(rsvd) doesn't match in line 2180 of x86/x86/local_apic.c in native_lapic_ipi_alloc(): (ip->gd_hioffset << 16) | ip->gd_looffset; Workign 11.4 shows these values: 11.4-i386 idx 251 (252-1 because no r363527, IPI_SWI) : hi = (49417) 1100 0001 0000 1001 lo = (5472) 0001 0101 0110 0000 func = 1100 0001 0000 1001 0001 0101 0110 0000 IDTVEC(rsvd) = 1100 0001 0000 1001 0001 0101 0110 0000 == rsvd_pti stable/13: i386 idx 252 Nok: hi = (65472) 1111 1111 1100 0000 lo = (12456) 0011 0000 1010 1000 func = 1111 1111 1100 0000 0011 0000 1010 1000 IDTVEC(rsvd) = 0 1110 0100 1011 1011 1110 1000 == rsvd_pti My knowledge about IDT and it's handling by OS/FreeBSD is by far not enough to get a solution/explanation for the above. I couldn't find the relevant paths in our source code (starting with the lack of basic asm skills). For sure I'm missing the important path here, maybe I'm totaly wrong... if not this hopefully saves you some time. I noticed r332489 too as a candidate for the regression beteween 12(.0) and 11(.2+). I prepared a diff which applies to 11.4 (sources from iso) but recompiling 11.4 fails currently, I would need to rework the i386/i386/pmap.c adoption. : : : >> know the link to the patch or the commit if you already have checked in? I will >> need to test and inform the parties who got affected. > > It was mailed for you after your query, then I pinged you several times. > Anyway, look at and try https://reviews.freebsd.org/D30310 I applied it to stable/13 and this doesn't make any difference, still the (ip->gd_hioffset << 16) | ip->gd_looffset; and (uintptr_t)IDTVEC(rsvd) differ with exactly the same values as shown above. Thanks for your attention, best regards, -harry