From owner-freebsd-virtualization@freebsd.org Wed May 19 11:26:22 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EACDC63EEBB for ; Wed, 19 May 2021 11:26:22 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "CN", Issuer "CN" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FlVtj5zwmz4kBh for ; Wed, 19 May 2021 11:26:21 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800:0:0:0:0:a135]) by mx0.gentlemail.de (8.15.2/8.15.2) with ESMTP id 14JBQAYi036042; Wed, 19 May 2021 13:26:11 +0200 (CEST) (envelope-from freebsd@omnilan.de) X-Authentication-Warning: mx0.gentlemail.de: Host ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800:0:0:0:0:a135] claimed to be mh0.gentlemail.de Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 4CBE249B; Wed, 19 May 2021 13:26:10 +0200 (CEST) Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root From: Harry Schmalzbauer To: Konstantin Belousov , Wei Hu Cc: "freebsd-virtualization@freebsd.org" References: <22bbe4cd-82ed-b826-9ac0-05ef8aceba0e@omnilan.de> Organization: OmniLAN Message-ID: <7986b561-dd26-a134-e76d-530900363deb@omnilan.de> Date: Wed, 19 May 2021 13:26:09 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <22bbe4cd-82ed-b826-9ac0-05ef8aceba0e@omnilan.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Greylist: inspected by milter-greylist-4.6.2 (mx0.gentlemail.de [IPv6:2a00:e10:2800:0:0:0:0:a130]); Wed, 19 May 2021 13:26:13 +0200 (CEST) for IP:'2a00:e10:2800::a135' DOMAIN:'ezra.dcm1.omnilan.net' HELO:'mh0.gentlemail.de' FROM:'freebsd@omnilan.de' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx0.gentlemail.de [IPv6:2a00:e10:2800:0:0:0:0:a130]); Wed, 19 May 2021 13:26:13 +0200 (CEST) X-Rspamd-Queue-Id: 4FlVtj5zwmz4kBh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [-3.30 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_XAW(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com,microsoft.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:61157, ipnet:2a00:e10:2800::/38, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:e10:2800::a130:from]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-0.996]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[omnilan.de]; SPAMHAUS_ZRD(0.00)[2a00:e10:2800::a130:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 May 2021 11:26:23 -0000 Am 17.05.2021 um 11:39 schrieb Harry Schmalzbauer: > Am 17.05.2021 um 08:47 schrieb Konstantin Belousov: >> On Mon, May 17, 2021 at 06:33:19AM +0000, Wei Hu wrote: >>>> -----Original Message----- >>>> From: Konstantin Belousov >>>> Sent: Monday, May 17, 2021 1:47 PM >>>> To: Wei Hu >>>> Cc: Harry Schmalzbauer ; Chris Knight >>>> ; freebsd-virtualization@freebsd.org >>>> Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root >>>> >>>> On Mon, May 17, 2021 at 04:12:20AM +0000, Wei Hu wrote: > : > : > : >>>>> separately. The workaround for this on amd64 is either build these >>>>> Hyper-V modules into the kernel or completely disable mitigation by >>>>> setting vm.pmap.pti="0" in /boot/loader.conf in the affect FreeBSD >>>>> guests. >>>>> > > Just to share some hopefully helpful pre-check's I've done so far: > > Disabling PTI via vm.pmap.pti="0" in /boot/loader.conf doesn't make > any difference for the i386 issue (at least x86/x86/local_apic.c > always uses non-pti path; tried nontheless, with identical result). > > As far as my analysis isn't completely wrong, the problem is about IDT > pointer. > IDTVEC(rsvd) doesn't match in line 2180 of x86/x86/local_apic.c in > native_lapic_ipi_alloc(): > (ip->gd_hioffset << 16) | ip->gd_looffset; > I managed to setup an i386 vm with -current from 3 years ago (not the most straight forward task due to compiler infrastructure changes back then). Up to r332488 vmbus(4) resp. hyperv(4) is working fine. Updating r332488 to r332489 breaks like already described. Since r332489 is a huge patch with lots of x86 art, the only chance I see, how this can be resolved, is with Konstantin's help. I can just assure that I tracked it down to the single r332489 committ which makes vmbus(4) failing on i386. Verification done: - built/installed world&kernel on i386 HyV guest with r332488   reboot fine. - built/installed kernel on the same vm after svnlite up -r 332489   reboot fails with "vmbus0: cannot find free IDT vector" - loaded kernel.old/kernel, did revert 332489, by 'svnlite up -r 332488' and compiled/reinstalled kernel (again r332488)   reboot fine. I tried to revert r332489 on stable/13 but failed due to lots of conflicting changes (including CVE driven) affecting the same area. To summarize: r332489 only breaks i386 native_lapic_ipi_alloc(), amd64 is not affected (but haven't checked if PTI has any influence on amd64). The effect is 100% reproducable. Happy to test any suggestion or debug extension! Thanks, -harry