From owner-freebsd-virtualization@freebsd.org Mon Nov 9 20:24:48 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 51A0146C6B7 for ; Mon, 9 Nov 2020 20:24:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CVMt81gK2z3qTT for ; Mon, 9 Nov 2020 20:24:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 3755446CA82; Mon, 9 Nov 2020 20:24:48 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 35FD646C1E9 for ; Mon, 9 Nov 2020 20:24:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CVMt80gFWz3qpC for ; Mon, 9 Nov 2020 20:24:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 08AB117165 for ; Mon, 9 Nov 2020 20:24:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0A9KOlqZ084664 for ; Mon, 9 Nov 2020 20:24:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0A9KOl6L084662 for virtualization@FreeBSD.org; Mon, 9 Nov 2020 20:24:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 250802] bhyve exited with status 134 when GPU PCI passthrough Date: Mon, 09 Nov 2020 20:24:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 12.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Nov 2020 20:24:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250802 --- Comment #46 from Konstantin Belousov --- (In reply to mr.xanto from comment #45) Ok, so my current understanding after looking at all debug data: 1. The real hardware BAR base address is 0xd000000. 2. We remap the BAR at 0x800000000 in guest. 3. The guest vmexit was due to EPT violation (exit reason 0x30) at GPA=20 0xde6d5000. Indeed, we did not mapped anything in the guest phys address 0xd0000000. I cannot explain it in any other way, then assume that NVIDIA device exposes BAR' bases somewhere beyond config space, and NVIDIA driver knows about tha= t. Then it accesses (remaps in guest VA) something by that side-channel addres= s. So there is no bug in bhyve as is, instead I would say that we have an incompatibility between hardware and virtualization. A possible, but quite work-intensive approach to *try* to fix it is to ensure that pass-through devices get their BARs bases mapped with GPA identical to HPA (host physical address). Doing that requires full rewrite of the BAR resource alloc code in bhyve. Any thought ? --=20 You are receiving this mail because: You are the assignee for the bug.=