From nobody Wed Feb 8 19:08:31 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PBqKT6RKCz3npM6 for ; Wed, 8 Feb 2023 19:08:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PBqKT5dmBz46pW; Wed, 8 Feb 2023 19:08:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675883325; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BjsGsTWF0LRyiKZdMANKIK9ch5fSy6aQRSXfKngXPOg=; b=RcZ+w5PLnquG1wjiMpVGcl6UeWcZFLcdk60O9JkDKcagI2FUAgz5Gza6QOk7kEkBHJludZ iUuwKki8PrcIJ3tVXs4NGuCPJTQfX5PPc27ba59E2hkFpktInrX+90BvFxSNAqzxblYQX9 BxEqUdqmWJWx39sLywP9XpRBg8y6ltqyV9iCfUTHbGnYgVhz+5KIfdhtognoPOThwJ7Mg/ +bGpB1wzkeoBQZusl1wpBLl1HToSnX135Fiz1bPsVhptKbYvsWPrmtuk47suBU3DMc3mxp XMJZeXFRYnEC0kANJP3DpU3aiebcI7ehhfDv8i0yi3uYNf8/IfnXUvq1T1OCRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675883325; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BjsGsTWF0LRyiKZdMANKIK9ch5fSy6aQRSXfKngXPOg=; b=tWcgO3TFfbaELup/W9X+Dk/X3u+jRnWJ8n+8dKG3mUl9TWs9KKl0oC+DQ5qHocvuK/0NZf DFII2P4R8ouO47ptxiudGssllZlG2XP5LcRq6CWICs4Fle6/U87cRJ85xfc/S4enMHWRR9 VPw9x66zwNrUNg7dkSAsypBYKsCJtbFHWFVyPeKYsppZDKNrXfuq7C5PB0miSKpN/85Txw wuKQkfOx7UYrl50+YHnY+/MS3wqyzKK0cTlj9CIr+U+4xQJQWNVftcADnpcH4QtEoupuXp ldv2+iNhn6B7H1oNU4dVAJRoxlQ0MpmCq6nbsC80gUaYNGFRFUgr+KmOfu1JmA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675883325; a=rsa-sha256; cv=none; b=m/YKMD9R5peRTYkG+PmoTwOwmVIHtxy3BMKyjl2U8Kp2Pz3mdPoFIDcmQ2M6pgFRPC/0nk zkmBbNGYpBtdLlCgpGbffmKIVGYyuxjvBHOlAJD1dX8exW5tR80SttFy/AddZGsRw4nMMn 77FU8E7Tk6q9OB8MpPQypQ4P+ZgGTfRsK5eQb8h4TOBsq2VOrJGAfYfwFWp0K+a5XFDsa5 2zQPQ7JGloumHj4y5RTmJFp4o6220e2jaPogwP/GUIrAGnnlUoz1z7IpyEqa/dJX8vFx3g hJAzvJ3CXwlYAc09Mi9/1DZoEkJe18d57bF93jQhtJ1KBH81K53OYUcEQuHS6Q== Received: from [10.0.1.4] (ralph.baldwin.cx [66.234.199.215]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PBqKT1tnzz132Q; Wed, 8 Feb 2023 19:08:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: <8aba2bc4-93da-44d7-1d14-8914c4111190@FreeBSD.org> Date: Wed, 8 Feb 2023 11:08:31 -0800 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: bhyve 13.1 compatibility breakage Content-Language: en-US To: Mark Johnston , =?UTF-8?Q?Corvin_K=c3=b6hne?= Cc: freebsd-virtualization@freebsd.org References: <202211230800.2AN80G58068419@gitrepo.freebsd.org> From: John Baldwin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N On 2/8/23 10:05 AM, Mark Johnston wrote: > On Sun, Jan 15, 2023 at 10:46:21AM -0500, Mark Johnston wrote: >> On Wed, Nov 23, 2022 at 08:00:16AM +0000, Corvin Köhne wrote: >>> The branch main has been updated by corvink: >>> >>> URL: https://cgit.FreeBSD.org/src/commit/?id=7c326ab5bb9aced8dcbc2465ac1c9ff8df2ba46b >>> >>> commit 7c326ab5bb9aced8dcbc2465ac1c9ff8df2ba46b >>> Author: Corvin Köhne >>> AuthorDate: 2022-11-21 14:00:04 +0000 >>> Commit: Corvin Köhne >>> CommitDate: 2022-11-23 08:00:04 +0000 >>> >>> vmm: don't lock a mtx in the icr_low write handler >>> >>> x2apic accesses are handled by a wrmsr exit. This handler is called in a >>> critical section. So, we can't lock a mtx in the icr_low handler. >>> >>> Reported by: kp, pho >>> Tested by: kp, pho >>> Approved by: manu (mentor) >>> Fixes: c0f35dbf19c3c8825bd2b321d8efd582807d1940 vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. >>> MFC after: 1 week >>> MFC with: c0f35dbf19c3c8825bd2b321d8efd582807d1940 >>> Sponsored by: Beckhoff Automation GmbH & Co. KG >>> Differential Revision: https://reviews.freebsd.org/D37452 >>> --- >> >> Hi Corvin, >> >> This seems to break AP startup when using bhyve/libvmmapi from FreeBSD >> 13.1 on a kernel built from main. It looks like the commit somehow >> regresses commit 769b884e2e2, but I'm not sure yet exactly how. > > I debugged this further and am not quite sure how to fix the problem, > which isn't specific to this commit after all. I'll try to describe it > here. > > Suppose we're booting a VM with 2 vCPUs. When the BSP raises the INIT > IPI to start AP 1, vlapic_icrlo_write_handler() looks up the destination > vCPU with vlapic_calcdest(), which only returns active vCPUs. However, > old bhyve executables activate APs (i.e., call vm_activate_cpu()) > lazily, only upon receiving a VM_EXITCODE_SPINUP_AP message. Thus, > vm_handle_ipi() simply doesn't doesn't do anything since "dmask" is > empty, so APs don't boot up. > > To further complicate things, new vmm.ko allocates vCPUs lazily. New > bhyve executables call vm_activate_cpu() for all vCPUs before running > the BSP, but as I said above, old bhyve executables do not. So merely > fixing "dmask" in vlapic_icrlo_write_handler() to include > not-yet-activated vCPUs doesn't work, and we can't allocate a new vCPU > in that context. In general it seems that we want an INIT IPI to > trigger allocation of a vcpu structure to preserve compatibility with > old bhyve, but I don't see a good way to implement that. > > I would quite like to fix this since I make heavy use of 13.1-RELEASE > bhyve+jails on a kernel running main. I believe bhyve from stable/13 is > unaffected, but 13.2 is not yet released. Any suggestions would be > appreciated. Hmm, I thought I had fixed this by using the bitmask of started CPUs rather than requiring the vCPU to be allocated. I was definitely testing an old bhyve binary from head against the vCPU branch while working on it and remember hitting this exact case, but I thought I had fixed it. Oh, hmm, my fix was the commit quoted above (769b884e2e2). -- John Baldwin