From nobody Mon Mar 24 00:09:57 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLYMp4Ldhz5s1Vq for ; Mon, 24 Mar 2025 00:09:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLYMp0Xkkz3mGl for ; Mon, 24 Mar 2025 00:09:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742774998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ANi4L1AYFSC+gXjM/IMnSoOKZtzrm5+qf68dxPvHxhA=; b=vkgOfSVw+A4u6Tf9djoTKeItib+w4A8RZi83yOeFzWi0Izh5fkUTKEPqfs1ec9XV9RnH6c A516jl9xuLpiy4NJhM3662Tz9uUH2dZxrnDrCs07UznkPaFLiHz3qK8cegyMcsjuisxoA3 nwS/BqJ+GXzdHV1wV0EW1MkWV55VdlNy3byzPyTrO9VhAKfru2ZZFpbGRaqqle2Mn/W7so 5ZyT3xAeZ47EMk/1x/gQy3oJ/EiMsgNda2ChUlPl4xjevaKcTpi28ks8XenzmHk1fIk4HT g6dKPdyG31oI/dSkHqeIlBExRs2fKm5a/yV6uLB/ZxMBOwYkT1jqbo6jZgt/8A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742774998; a=rsa-sha256; cv=none; b=kJz1oLcQkMdN0OvvYnq3ULO0w+tLnyn9+YMI0B8XY2GtcsbVEoi4IJC7edmSbJ6/v7hEQ8 ZqpWkU9XgznUjc/LGtCVqX3ZrVQc6XBESYoh9TGL14kQcuz796nVvSOcN692xfTpaNHTVh /SyuqleXcw+2iWwqFoMLgRPvAK87rVBV7DkP2X3p8Klwp9m04dswK4+UbEKfr5vLnJn23D QPLQh3Flw0GRt2QQb4hQKoJOyNsz6MKOl7X87wDmbZgV0EwihHMJjXjuWoSxEo7WZps+8V vJqXdVDDj4PQwBCN6DbRGV0GtfMqIpL/xdwr5svrQ3SV+QZuTT6/2cpz+ea1LQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742774998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ANi4L1AYFSC+gXjM/IMnSoOKZtzrm5+qf68dxPvHxhA=; b=XJDIFJWMn4UtlET9iaY3ooyA+Bl68qqqGffpw4RJU+gNhygLABXj4n3JpFz2I24/LO4lD9 8Nrf2fbhAUpsJpbdBYuqeoRorotm+MDxzBF73WmxazJhgUgJdi4xa929UPic2ZUrJnrHvr LtUUGFDQAZCeR5nQX6gRu85X+d337L76dK4QUP7bKaAuxl8qsFGEK7lFHURnhXTHWr3kbn vBObCG35h1hwXlyfuGBnMfBwhXaou191UQvrojCj2mqJVIa4YITvJZzDVb28IXDPfTyLDe ZSlKuPf5Aqdl6Ez2dBUiJ7Y5vMWOIltjw+f2B7ds8zKqOBBwZNrOYB58NiuXlQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZLYMn6qGMzVQj for ; Mon, 24 Mar 2025 00:09:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52O09v16042096 for ; Mon, 24 Mar 2025 00:09:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52O09vBH042095 for virtualization@FreeBSD.org; Mon, 24 Mar 2025 00:09:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Mon, 24 Mar 2025 00:09:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jhb@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #10 from John Baldwin --- Yeah, x86 does some weird thing where it doesn't truncate ds_len in some ca= ses to the remaining length (one of the folks at Cambridge noticed that arm64 d= oes honor the remaining length of the mapping in a way that x86 does not).=20 Disabling the assertions for arm64 is probably due to that. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Mar 24 13:55:11 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLvh02Ygyz5rx6N for ; Mon, 24 Mar 2025 13:55:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLvh000Kbz45qD for ; Mon, 24 Mar 2025 13:55:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742824512; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CovXY+gb+w4GflrKHgAsnc1pZUyIxswhHfJFLxvBN+s=; b=jl0MuV6vplybSr3ucqZ+0B+4++RBK0cYV7bPzI4ObiEh1O/RGMwExm3m3k/a4dV4i90XtB uzk4b98pfGhUjDo4JIWqfXf4lce+J+bfY8YWtXPntn97zmzNSC0c2fwAiFmFzje+PorFo7 bfI5k34bYZq8YP0n/6/DILHQ5CAaBtst2IiUDuyennRhfDTbwi4h67hOsaq0igjJkSasZ8 orqQtD5Xb7Qy4VPN+M11SFqAsKYEcmFTzR+5m/aol4g7+jTVlsLSr1Td6kMAyJDmnAkwd3 GE6Qjpo8G6cCngViVV4t0wx2XH/v8MfL+6px0v2ZEsIWfCfpkLSNmp/6Y2CwIw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742824512; a=rsa-sha256; cv=none; b=dZmojOfW1+u4O2zaIhLXEN0bYawlJl1luCNaiSwcpdUBCvW6ssFApnJNP9Vtg3PUhpyxn4 qlTmWPah8fBk0yrLDf6Rzh3uJXGAp52C2Ps593jL2MjddQaG+Y9WjTgpZ+cQx83rw8o08q ZTouK7d50Bov/dm/yGJih1ZCnE6x3ng2P9N+8w35JYykjlkil0b52wxgRaV5i/nFTbdHsR 273PV4DMusSFWUpXgPQvxuYPNedoGUZGtaE/BcOTzQvSdaYd0Bc4rnguYevl63qMmA7JV1 +IdCUzut5ki2Kp2eWwHRzIBpolY1xcCCdIPHLA+G5yrRjC23SmK9ehmSDCsKyQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742824512; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CovXY+gb+w4GflrKHgAsnc1pZUyIxswhHfJFLxvBN+s=; b=aK9VhvfzGV3zSwV/y5wE4AeN6O+HuOOZga8ywE6/MqJZ6F43zm/PEXFEatEvAICTz5cm3k darAlwBudlVkEEuVyT8Q5dJMud0sNR47ns+7EPDI4JmzIVyFCDlQrNpsayXZrNytBowlOt 58NGQ8hNyrBqGH2wJJko+97ehqQd+HrWOiGEkg6PdBqjS/MLiZNnzqcXeQkKW36Vd18z/z FCKkTpoD5sQfx8YHRmu0c4cpDLZQb2n4ehbquX+UAQJlAm6iWzLa2LKXjJLB+AL6OPmYGd 3l+xEkSQII3FECtv1zb7HbwQv36CNwkTkEJpJRMJWTq07+F4Gw4Yde/I3jo7xg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZLvgz6PfQzxMT for ; Mon, 24 Mar 2025 13:55:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52ODtB9U054471 for ; Mon, 24 Mar 2025 13:55:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52ODtB0C054470 for virtualization@FreeBSD.org; Mon, 24 Mar 2025 13:55:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Mon, 24 Mar 2025 13:55:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: weh@microsoft.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 Wei Hu changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |weh@microsoft.com --- Comment #11 from Wei Hu --- (In reply to Mark Johnston from comment #9) Storvsc can only send a list of page addresses to host. When the number of total pages is greater than 1, it checks the mapped address inside the first page plus its length ends on the page boundary, so the second page starts r= ight after it without gap. In this panic: panic: invalid 1st page, ofs 0x3985000, len 2048 the mapped address inside the first page plus its length ends in the middle= of the page, leaving 2048 bytes gap before starting the next page. This doesn't seem to be allowed on x86. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Mar 24 14:21:48 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLwGy46GZz5rySt for ; Mon, 24 Mar 2025 14:22:02 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x1133.google.com (mail-yw1-x1133.google.com [IPv6:2607:f8b0:4864:20::1133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLwGx2qnYz3GY0 for ; Mon, 24 Mar 2025 14:22:01 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=SPMHKRoS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shamim.shahriar@gmail.com designates 2607:f8b0:4864:20::1133 as permitted sender) smtp.mailfrom=shamim.shahriar@gmail.com Received: by mail-yw1-x1133.google.com with SMTP id 00721157ae682-6ef9b8b4f13so38585877b3.2 for ; Mon, 24 Mar 2025 07:22:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742826120; x=1743430920; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=IDkkL/yqijTUoWappBqXP6FNcxQEhm+l6R9xn/rku8Y=; b=SPMHKRoSy+wiDZWYVd3rC3UYQJimyc2N/RkTsqKHc8Yid3mklqwuT1TyvCbFIAPScL BBmJHZYYiQrb6qgZe39gt7Vjdmubv0p3OSAdlhwQ36d5ALMztPGgo2x1LCs0hANyjCuX BeiebeNL6p0TwNCfViiR6+8wTGQm97etg8O0YQ/2FJ1zeBJ78rjAbD2xNv3QzITysXlO D1tlqnvneVM9X2wfn9TE2yW/nunkdpDBkgbXzx2YiwYy+JxNSQz6LvJjbkFrK/iUzy8n qYG1uXWPvbzHAxJVJzuML/Rdva36i8nWkDMlsbxFbS0IHNahs0WDdbPIB5b0MkGOB2Bf 45ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742826120; x=1743430920; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IDkkL/yqijTUoWappBqXP6FNcxQEhm+l6R9xn/rku8Y=; b=hg5bsAMeSq6bW0X//S/3WLBaIFbk7MqOBG/gYyFBeLxUjDyr/z+dWBW51l2wECltdm iV6SI39GMuRFmIS48mBQHkT8bM6ZTJd1trcT25g8VJOCtT+8c9zArkKGU6C2mwkpstJV fhuzXfoJkvJ9Vx7LMIibam1wXR0rHc3jN5Lef4Xn1j2TGs94HZJpzLhaZxxJBxFSClc6 eMvFad5zx2yvyohTmVLfPuZM2WsuPRYSf1c053bx4jSkf0jlL/Rsye61/G31/bq/gclz t7lnLT9oicHSKac//G/5L8x/9Grjuh8e9vK9vR2vbfz3sdKBmPt4m4jOEzWRkNlXfihu Ig9g== X-Gm-Message-State: AOJu0YzI6I03cXKynKYx8MCtt7Tb27fSmbarRhIJJ44zWUO6uPkvVHIc X9FC3d/AYZZ7Icj6ZIY3YYoNtzyfmqd4864kPTEcfp3I0rDJBKAqsN+pPxcI/qfUxCfJTBvYanm bZTA1eDNpKIMarQMG6xspjZ6lfep+XOM= X-Gm-Gg: ASbGncuIEY0uMU8HObYUuMIyXqIl8DiREt8eUzLDpwomnsB+B8WtEXjnvuZs1LuU6Y/ OYYWJ6DcSmXMS0WhrIkAeJoDDhEwk9SHPPa0vQWZZpb285BVVb/aG2ZHfSo1fdyC9GIIOhzCl0/ tmwvB4ou0eHSmP8eRYMEM01iw= X-Google-Smtp-Source: AGHT+IHIHI6TItr5ZntpzkEV5I3v//iBE86+M983ON69M5NLE4SIXvVQQRlVtU1KdxyjBc1TvT2DguYSSuS8C4m0heA= X-Received: by 2002:a05:690c:c14:b0:700:a988:59dc with SMTP id 00721157ae682-700bad227e3mr165175657b3.31.1742826119948; Mon, 24 Mar 2025 07:21:59 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 From: Shamim Shahriar Date: Mon, 24 Mar 2025 14:21:48 +0000 X-Gm-Features: AQ5f1Jr8OMSWdJomsXDn9dA-bNtvlGMb6AlCdYBLpZbRoAn60Ulz8dyjDVRI7tw Message-ID: Subject: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c8f694063117538d" X-Spamd-Result: default: False [2.12 / 15.00]; HTTP_TO_IP(1.00)[]; URI_COUNT_ODD(1.00)[1]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; NEURAL_HAM_SHORT(-0.88)[-0.882]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TAGGED_FROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1133:from]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLwGx2qnYz3GY0 X-Spamd-Bar: ++ --000000000000c8f694063117538d Content-Type: text/plain; charset="UTF-8" Good afternoon everyone. I am trying to have VMs with GPU passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/8000 installed already. I have checked the device IDs and put pptdevs in place # cat /boot/loader.conf pptdevs="18/0/0 19/0/0" pptdevs2="72/0/0 73/0/0" this is showing the GPUs as pptdev on the pciconf ppt0@pci0:18:0:0: class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8 vendor = 'NVIDIA Corporation' device = 'TU102GL [Quadro RTX 6000/8000]' class = display subclass = 3D ppt1@pci0:19:0:0: class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8 vendor = 'NVIDIA Corporation' device = 'TU102GL [Quadro RTX 6000/8000]' class = display subclass = 3D as I am using vm-bhyve, I have put the configuration as below # cat /mnt/VMs/jagadish/jagadish.conf loader="uefi" cpu=16 memory=128G xhci_mouse="yes" debug="true" graphics="yes" graphics_listen="127.0.0.1" graphics_port="5920" graphics_res="1024x768" graphics_wait="no" #graphics_vga="io" network0_type="virtio-net" network0_switch="swUNI" network0_mac="58:9c:fc:06:3f:af" disk0_type="nvme" disk0_name="jagadish-disk0.img" #disk0_size="128G" uuid="966e909b-1293-11ef-a9a4-e4434bfe34de" passthru0="19/0/0=6:0" bhyve_options="-A -H -P" #END however, when I start the vm (for OS installation to start with), it shows it is running but I am unable to access the VNC for a while, then when I finally manage to connect via VNC, there is nothing on the screen, just blank dark screen the vm-bhyve.log shows Mar 24 13:59:26: initialising Mar 24 13:59:26: [loader: uefi] Mar 24 13:59:26: [cpu: 16] Mar 24 13:59:26: [memory: 128G] Mar 24 13:59:26: [hostbridge: standard] Mar 24 13:59:26: [com ports: com1] Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] Mar 24 13:59:26: [debug mode: true] Mar 24 13:59:26: [primary disk: jagadish-disk0.img] Mar 24 13:59:26: [primary disk dev: file] Mar 24 13:59:26: initialising network device tap0 Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) Mar 24 13:59:26: booting Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet] Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] Mar 24 13:59:26: [bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] Mar 24 13:59:26: starting bhyve (run 1) based on what I can see, and a little out of desparation, I decided to run the installer in the foreground, and below is what I got # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso Starting jagadish * found guest in /mnt/VMs/jagadish * booting... fbuf frame buffer base: 0x112245400000 [sz 16777216] it stays there for as long as I wait until I poweroff the vm. tried with Debian installer, # vm install -f jagadish debian-12.5.0-amd64-netinst.iso Starting jagadish * found guest in /mnt/VMs/jagadish * booting... fbuf frame buffer base: 0x2747e2400000 [sz 16777216] ideally I need to install Debian for my user base on these GPU based systems. But even that is proving to be impossible since the installer does not move any further from where it is. Any thoughts/ideas/suggestions what I else I can try to make this work? Anything that you have tried that worked? have I missed something? Would appreciate any and all thoughts/suggestions best regards SS --000000000000c8f694063117538d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Good afternoon everyone.

I am trying to have VMs wi= th GPU passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/= 8000 installed already. I have checked the device IDs and put pptdevs in pl= ace

# cat /boot/loader.conf
pptdevs=3D"18/0/0 19/0/0"pptdevs2=3D"72/0/0 73/0/0"

this is showing the GPUs as p= ptdev on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D= 0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D= 0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NV= IDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102G= L [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0= =3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: = =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de= device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 ven= dor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device = =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0= class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D= 3D

as I am using vm-bhyve, I have put the configuration as below
# cat /mnt/VMs/jagadish/jagadish.conf
loader=3D"uefi"
= cpu=3D16
memory=3D128G
xhci_mouse=3D"yes"
debug=3D"= true"

graphics=3D"yes"
graphics_listen=3D"127= .0.0.1"
graphics_port=3D"5920"
graphics_res=3D"10= 24x768"
graphics_wait=3D"no"
#graphics_vga=3D"io&= quot;

network0_type=3D"virtio-net"
network0_switch=3D&q= uot;swUNI"
network0_mac=3D"58:9c:fc:06:3f:af"

disk= 0_type=3D"nvme"
disk0_name=3D"jagadish-disk0.img"#disk0_size=3D"128G"

uuid=3D"966e909b-1293-11ef-a9a4= -e4434bfe34de"

passthru0=3D"19/0/0=3D6:0"

bhyv= e_options=3D"-A -H -P"
#END

however, when I start the v= m (for OS installation to start with), it shows it is running but I am unab= le to access the VNC for a while, then when I finally manage to connect via= VNC, there is nothing on the screen, just blank dark screen

the vm-= bhyve.log shows

Mar 24 13:59:26: initialising
Mar 24 13:59:26: = =C2=A0[loader: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26= : =C2=A0[memory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridge: standard]
= Mar 24 13:59:26: =C2=A0[com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 9= 66e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode:= true]
Mar 24 13:59:26: =C2=A0[primary disk: jagadish-disk0.img]
Mar = 24 13:59:26: =C2=A0[primary disk dev: file]
Mar 24 13:59:26: initialisin= g network device tap0
Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI= addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar= 24 13:59:26: booting
Mar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 12= 8G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -= U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bh= yve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadi= sh-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru= ,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920<= /a>,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve c= onsole: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve iso= device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]=
Mar 24 13:59:26: starting bhyve (run 1)

based on what I can see,= and a little out of desparation, I decided to run the installer in the for= eground, and below is what I got

# vm install -f jagadish FreeBSD-14= .2-RELEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found guest in = /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x11224= 5400000 [sz 16777216]

it stays there for as long as I wait until I p= oweroff the vm.

tried with Debian installer,
# vm install -f jag= adish debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0 * foun= d guest in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer ba= se: 0x2747e2400000 [sz 16777216]


ideally I need to install Debia= n for my user base on these GPU based systems. But even that is proving to = be impossible since the installer does not move any further from where it i= s.

Any thoughts/ideas/suggestions what I else I can try to make this= work? Anything that you have tried that worked? have I missed something?
Would appreciate any and all thoughts/suggestions

best regards=
SS
--000000000000c8f694063117538d-- From nobody Mon Mar 24 14:57:19 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLx4R3Zksz5s1mT for ; Mon, 24 Mar 2025 14:57:59 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLx4Q40Fnz3WDc for ; Mon, 24 Mar 2025 14:57:58 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hQabPplC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::1030 as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-3018e2d042bso5507652a91.2 for ; Mon, 24 Mar 2025 07:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742828277; x=1743433077; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=b3ubtgJDbIbo+oY3JwTrrB3ewyWFAY81utGdzTIBNZM=; b=hQabPplCu/LPOPd7uBOMwgHZ+IlthgaQUqmwTahGc2BpXVm/H1kkXy2P7AVzNlmNBc zaSfEDMZrSkNzxm8+bt8y0YoV3uPDonxg6jYGS6EDCH1Nr4zosPHhdIlw9/2r3DtI15o XXYKWThl9dUfdkKf3cyjvhjYMNi+6DXkkG6J5tTR51N7jCz8HOapbF/n3S+HweyxakPP a9OKOiQrRKKmDli970jMjTaYBjja+RG+6jEXWMtnowEMN4PtViExBm4Iu4dlL3pnGFKb vjZnqFNLErXIThUqGLqPx7KEafWA6guCH/6YZeco6+8gyukPJIOCF9NubAkHWEX/iOXv PW1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742828277; x=1743433077; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=b3ubtgJDbIbo+oY3JwTrrB3ewyWFAY81utGdzTIBNZM=; b=QxqVW1Rv8I7YVd9sQReyde8+Uc6EPxytpBv9o4q9z4Kv1CAJg4+qoRQFdkptrdN57M U39aeIVdr62N3+WsoQVZ86Ch47jnEOX107wbn6T9rHjVQxOhpIyGl/Mf++hKLbQE8gw5 iUFU9FP9Mn0bSRpFl29hT6zb9Rr9oWItRaRLiQFBICvtbVsMwxwheDNG9n+7cDAAcHuu j9wlfydKxU/haNScHwVSerUrO9roWSsKYb1bZD/AZliJxisDM/Lb3jxJFe0dxOlZCTMM fXXosdv9ITTp3xW89hKMPIk41zMsAGMsqxbGcgVoM0Qz7mm9N3LY45gmsOaITFd4u4G9 oBtw== X-Gm-Message-State: AOJu0YyDefkp87Wth+YyuSRGVkeBL/QgckuJgiu5A3hFR0OytQqrwQ5G MJ7GmznEqK7UGCthtOb2csxqcNAhj6uMIWT7kI/B/ykZu+uXsNfKJjsQWF10HMUDO14bH4lQ/11 I5QOb6IzI6Smx1FclgPWtPLo0py5nlKU09ZE= X-Gm-Gg: ASbGncsT8glmyMw2X1TKDQAe1ZrP+PpygzeEyIEu1OXRrt8jme1tJmJ+4gD6LNiQlBc FI2n+bK/TiSbUCo5S/pmSoOrqtmbVUdmcm7VkuoDqL8FIhT8VyjFimxfkAF8nDFSLUR+IIHs0xV gkeKBVqH0UyOty65LRy3Jdf5Y2AM4= X-Google-Smtp-Source: AGHT+IFXW45p4sSR+V7+AqZUhAs/BUkd7/F9CTQPRwul0O9/I9bCBLkHyqGYGSyXAqNcBElskvLxdFGueOQLHx9FkZI= X-Received: by 2002:a17:90a:da85:b0:2fe:8217:2da6 with SMTP id 98e67ed59e1d1-3030fef09a8mr17614005a91.22.1742828276764; Mon, 24 Mar 2025 07:57:56 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 15:57:19 +0100 X-Gm-Features: AQ5f1Jq4gUcTKx1JLpUqj6hBNOkOUoHxZz2lhf_cxbaH4u_Evys23nhoftqwBJ0 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000575a23063117d498" X-Spamd-Result: default: False [2.15 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; NEURAL_SPAM_SHORT(0.15)[0.148]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1030:from]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLx4Q40Fnz3WDc X-Spamd-Bar: ++ --000000000000575a23063117d498 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Usually I use this kind of script to launch a bhyve vm : #!/bin/sh setxkbmap it kldload vmm.ko vms=3D"$(ls /dev/vmm/*)" vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" kldload vmm.ko if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." kldload nvidia-modeset devctl clear driver -f pci0:2:0:0 devctl set driver -f pci0:2:0:0 ppt else echo "rtx 2080ti slot 2/0/0 is already attached to ppt" fi if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." devctl clear driver -f pci0:2:0:1 devctl set driver -f pci0:2:0:1 ppt else echo "rtx 2080ti slot 2/0/1 is already attached to ppt" fi if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." devctl clear driver -f pci0:2:0:2 devctl set driver -f pci0:2:0:2 ppt else echo "rtx 2080ti slot 2/0/2 is already attached to ppt" fi if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." devctl clear driver -f pci0:2:0:3 devctl set driver -f pci0:2:0:3 ppt else echo "rtx 2080ti slot 2/0/3 is already attached to ppt" fi echo "rtx 2080ti is fully attached to ppt" for vm in $vms; do session=3D"${vm##*/}" echo "bhyve session =3D $session" echo "vnc session =3D $vncs" if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then printf 'VNC session not found,destroying ghost vms\n' bhyvectl --vm=3D$session --destroy else printf 'Found VNC session %s\n' "${session},no ghost vms found,not destroying them" fi done vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4= CZ)/ && d{print d}'` echo "Seagate M3 Portable 1.8T ; $vmdisk0" vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2015040204055E)/ && d{print d}'` echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{print d}'` echo "CT1000P1SSD8 ; $vmdisk2" vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{print d}'` echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'` echo "G-DRIVE USB ; $vmdisk4" vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'` echo "Elements 25A3 ; $vmdisk5" vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && d{print d}'` echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'` echo "Corsair Force 3 SSD ; $vmdisk7" vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'` echo "CT500MX500SSD4 ; $vmdisk8" vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}'` echo "SanDisk Cruzer-15GB ; $vmdisk9" vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}'` echo "Kingston DataTraveler 2.0 ; $vmdisk10" vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)/ && d{print d}'` echo "WD 2500BMV External ; $vmdisk11" vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ && d{print d}'` echo "WD 3200BMV External ; $vmdisk12" vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ && d{print d}'` echo "CT500BX500SSD1 ; $vmdisk13" /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H -= A \ -s 0,hostbridge \ -s 1,ahci-hd,/dev/$vmdisk13 \ -s 10,virtio-net,tap6 \ -s 11,virtio-9p,sharename=3D/ \ -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & As you can see,each vm has a unique ID (connected with the vncviewer instance) and the vncviewer window is launched as soon as the bhyve vm is called...and my Geforce RTX 2080 ti is passed through inside the vm. On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto wrote: > First of all,you could use a more flexible technique than using the > pptdevs on /boot/loader.conf. I would use a script like this : > > 1) > > if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then > echo "18/0/0 is not attached to ppt,attaching..." > devctl clear driver -f pci0:18:0:0 > devctl set driver -f pci0:18:0:0 ppt > else > echo "18/0/0 already attached to ppt" > fi > > 2) I would not like to use vm-bhyve. It adds only some confusion... > > On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar > wrote: > >> Good afternoon everyone. >> >> I am trying to have VMs with GPU passthrough. The setup is a Dell server >> with NVidia Quadro RTX 6000/8000 installed already. I have checked the >> device IDs and put pptdevs in place >> >> # cat /boot/loader.conf >> pptdevs=3D"18/0/0 19/0/0" >> pptdevs2=3D"72/0/0 73/0/0" >> >> this is showing the GPUs as pptdev on the pciconf >> >> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D= 0x10de >> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >> vendor =3D 'NVIDIA Corporation' >> device =3D 'TU102GL [Quadro RTX 6000/8000]' >> class =3D display >> subclass =3D 3D >> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D= 0x10de >> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >> vendor =3D 'NVIDIA Corporation' >> device =3D 'TU102GL [Quadro RTX 6000/8000]' >> class =3D display >> subclass =3D 3D >> >> as I am using vm-bhyve, I have put the configuration as below >> >> # cat /mnt/VMs/jagadish/jagadish.conf >> loader=3D"uefi" >> cpu=3D16 >> memory=3D128G >> xhci_mouse=3D"yes" >> debug=3D"true" >> >> graphics=3D"yes" >> graphics_listen=3D"127.0.0.1" >> graphics_port=3D"5920" >> graphics_res=3D"1024x768" >> graphics_wait=3D"no" >> #graphics_vga=3D"io" >> >> network0_type=3D"virtio-net" >> network0_switch=3D"swUNI" >> network0_mac=3D"58:9c:fc:06:3f:af" >> >> disk0_type=3D"nvme" >> disk0_name=3D"jagadish-disk0.img" >> #disk0_size=3D"128G" >> >> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >> >> passthru0=3D"19/0/0=3D6:0" >> >> bhyve_options=3D"-A -H -P" >> #END >> >> however, when I start the vm (for OS installation to start with), it >> shows it is running but I am unable to access the VNC for a while, then >> when I finally manage to connect via VNC, there is nothing on the screen= , >> just blank dark screen >> >> the vm-bhyve.log shows >> >> Mar 24 13:59:26: initialising >> Mar 24 13:59:26: [loader: uefi] >> Mar 24 13:59:26: [cpu: 16] >> Mar 24 13:59:26: [memory: 128G] >> Mar 24 13:59:26: [hostbridge: standard] >> Mar 24 13:59:26: [com ports: com1] >> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >> Mar 24 13:59:26: [debug mode: true] >> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >> Mar 24 13:59:26: [primary disk dev: file] >> Mar 24 13:59:26: initialising network device tap0 >> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >> Mar 24 13:59:26: booting >> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >> Mar 24 13:59:26: [bhyve iso device: -s >> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >> Mar 24 13:59:26: starting bhyve (run 1) >> >> based on what I can see, and a little out of desparation, I decided to >> run the installer in the foreground, and below is what I got >> >> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >> Starting jagadish >> * found guest in /mnt/VMs/jagadish >> * booting... >> fbuf frame buffer base: 0x112245400000 [sz 16777216] >> >> it stays there for as long as I wait until I poweroff the vm. >> >> tried with Debian installer, >> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >> Starting jagadish >> * found guest in /mnt/VMs/jagadish >> * booting... >> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >> >> >> ideally I need to install Debian for my user base on these GPU based >> systems. But even that is proving to be impossible since the installer d= oes >> not move any further from where it is. >> >> Any thoughts/ideas/suggestions what I else I can try to make this work? >> Anything that you have tried that worked? have I missed something? >> >> Would appreciate any and all thoughts/suggestions >> >> best regards >> SS >> > > > -- > Mario. > --=20 Mario. --000000000000575a23063117d498 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Usually I use this kind of script to launch a bhyve v= m :

#!/bin/sh

setxkbmap it
kldload vmm.ko
= vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncv= iewer [0]/{print $6}')"
kldload vmm.ko

if ! pciconf -l p= ci0:2:0:0 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/= 0/0 is not attached to ppt,attaching..."
kldload nvidia-modeset
= devctl clear driver -f pci0:2:0:0
devctl set driver -f pci0:2:0:0 pptelse
echo "rtx 2080ti slot 2/0/0 is already attached to ppt"<= br>fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; thenecho "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."=
devctl clear driver -f pci0:2:0:1
devctl set driver -f pci0:2:0:1 pp= t
else
echo "rtx 2080ti slot 2/0/1 is already attached to ppt&qu= ot;
fi

if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; the= n
echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching...&q= uot;
devctl clear driver -f pci0:2:0:2
devctl set driver -f pci0:2:0:= 2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already attached to pp= t"
fi

if ! pciconf -l pci0:2:0:3 | grep -q "^ppt";= then
echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching.= .."
devctl clear driver -f pci0:2:0:3
devctl set driver -f pci0:= 2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is already attached t= o ppt"
fi

echo "rtx 2080ti is fully attached to ppt&quo= t;

for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve session =3D $sessio= n"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo &qu= ot;vnc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}" | grep "= ${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf &#= 39;VNC session not found,destroying ghost vms\n'
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 printf 'Found VNC session %s\n' "${session},no g= host vms found,not destroying them"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geom disk list | awk = '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ && d{print d}&#= 39;`
echo "Seagate M3 Portable 1.8T ; $vmdisk0"

vmdisk1= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (201504020= 4055E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 = 1.8T ; $vmdisk1"

vmdisk2=3D`geom disk list | awk '/^Geom na= me: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{print d}'`
ec= ho "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`geom disk list | a= wk '/^Geom name: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{= print d}'`
echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3&quo= t;

vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *= ident: (BE0191510218)/ && d{print d}'`
echo "G-DRIVE US= B ; $vmdisk4"

vmdisk5=3D`geom disk list | awk '/^Geom name:= /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'`
e= cho "Elements 25A3 ; $vmdisk5"

vmdisk6=3D`geom disk list |= awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && = d{print d}'`
echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6&qu= ot;

vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (20140108006C)/ && d{print d}'`
echo "Corsair F= orce 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk list | awk '/^G= eom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'`=
echo "CT500MX500SSD4 ; $vmdisk8"

vmdisk9=3D`geom disk = list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ &= ;& d{print d}'`
echo "SanDisk Cruzer-15GB ; $vmdisk9"<= br>
vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *id= ent: (60A44D4138D8F311190A0149)/ && d{print d}'`
echo "= Kingston DataTraveler 2.0 ; $vmdisk10"

vmdisk11=3D`geom disk li= st | awk '/^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)= / && d{print d}'`
echo "WD 2500BMV External ; $vmdisk11= "

vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF}= /^ *ident: (57442E575845323039544433303334)/ && d{print d}'`echo "WD 3200BMV External ; $vmdisk12"

vmdisk13=3D`geom = disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ &am= p;& d{print d}'`
echo "CT500BX500SSD1 ; $vmdisk13"
/usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads= =3D1 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/$vmdisk13 \=
-s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbu= f,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \=
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE= .fd \
vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &<= /div>

As you can see,each vm has a unique ID (connected = with the vncviewer instance) and the vncviewer window is launched as soon a= s the bhyve vm is called...and my Geforce RTX 2080 ti is passed through ins= ide the vm.


On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto= <marietto20= 08@gmail.com> wrote:
First of all,you could use a more flexibl= e technique than using the pptdevs on /boot/loader.conf. I would use a scri= pt like this :

1)

if ! pc= iconf -l pci0:18:0:0 | grep -q "^ppt"; then
echo "18/0/= 0 is not attached to ppt,attaching..."
devctl clear driver -f pci0:= 18:0:0
devctl set driver -f pci0:18:0:0 ppt
else
echo "18/0/0= already attached to ppt"
fi

2) I would not = like to use vm-bhyve. It adds only some confusion...

On Mon, Mar 24, 2= 025 at 3:22=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
=
Go= od afternoon everyone.

I am trying to have VMs with GPU passthrough.= The setup is a Dell server with NVidia Quadro RTX 6000/8000 installed alre= ady. I have checked the device IDs and put pptdevs in place

# cat /b= oot/loader.conf
pptdevs=3D"18/0/0 19/0/0"
pptdevs2=3D"= 72/0/0 73/0/0"

this is showing the GPUs as pptdev on the pcicon= f

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1= hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D= 0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation= 9;
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/= 8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0= =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 cl= ass=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subven= dor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D &= #39;NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D '= TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 = =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D

as I am us= ing vm-bhyve, I have put the configuration as below

# cat /mnt/VMs/j= agadish/jagadish.conf
loader=3D"uefi"
cpu=3D16
memory=3D= 128G
xhci_mouse=3D"yes"
debug=3D"true"

gra= phics=3D"yes"
graphics_listen=3D"127.0.0.1"
graph= ics_port=3D"5920"
graphics_res=3D"1024x768"
graph= ics_wait=3D"no"
#graphics_vga=3D"io"

network0= _type=3D"virtio-net"
network0_switch=3D"swUNI"
ne= twork0_mac=3D"58:9c:fc:06:3f:af"

disk0_type=3D"nvme&q= uot;
disk0_name=3D"jagadish-disk0.img"
#disk0_size=3D"= 128G"

uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"
passthru0=3D"19/0/0=3D6:0"

bhyve_options=3D"-A = -H -P"
#END

however, when I start the vm (for OS installatio= n to start with), it shows it is running but I am unable to access the VNC = for a while, then when I finally manage to connect via VNC, there is nothin= g on the screen, just blank dark screen

the vm-bhyve.log shows
Mar 24 13:59:26: initialising
Mar 24 13:59:26: =C2=A0[loader: uefi]Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[memory: 128G]<= br>Mar 24 13:59:26: =C2=A0[hostbridge: standard]
Mar 24 13:59:26: =C2=A0= [com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-1293-11ef-a9a4-= e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode: true]
Mar 24 13:59:= 26: =C2=A0[primary disk: jagadish-disk0.img]
Mar 24 13:59:26: =C2=A0[pri= mary disk dev: file]
Mar 24 13:59:26: initialising network device tap0Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:= 26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: bootingMar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l bootrom,/us= r/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9= a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices: -s 0,hostb= ridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,vir= tio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp= =3D127.0.0.1:5920,w= =3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve consol= e: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve iso devi= ce: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
M= ar 24 13:59:26: starting bhyve (run 1)

based on what I can see, and = a little out of desparation, I decided to run the installer in the foregrou= nd, and below is what I got

# vm install -f jagadish FreeBSD-14.2-RE= LEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found guest in /mnt/= VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x1122454000= 00 [sz 16777216]

it stays there for as long as I wait until I powero= ff the vm.

tried with Debian installer,
# vm install -f jagadish= debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0 * found gue= st in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0= x2747e2400000 [sz 16777216]


ideally I need to install Debian for= my user base on these GPU based systems. But even that is proving to be im= possible since the installer does not move any further from where it is.
Any thoughts/ideas/suggestions what I else I can try to make this work= ? Anything that you have tried that worked? have I missed something?
Would appreciate any and all thoughts/suggestions

best regards
<= div>SS


--
Ma= rio.


--
Ma= rio.
--000000000000575a23063117d498-- From nobody Mon Mar 24 14:58:49 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLx6B6gZwz5s1v5 for ; Mon, 24 Mar 2025 14:59:30 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLx696b0yz3XDs for ; Mon, 24 Mar 2025 14:59:29 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=h3+KqXWc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::630 as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-227d6b530d8so12390655ad.3 for ; Mon, 24 Mar 2025 07:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742828368; x=1743433168; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/3vwxiqm/CU6etvYPXXXelYSt+UGZmTt+cQ2Bq6SpGY=; b=h3+KqXWcLM6zLXtVfGvZQlODa3PsGe5LPAppHDojf3vBUJraS8rikwBHagNPR1jdKW gE7ET+LhCmO47CIYNc7NH6tBsgJhrHT5Eb1fI1eqICjoQh6a4pbdFILfqobSdOF3YzAY 5K/ghnrJc5vmmwgASKAg8bSpwv2YE8+7PQu2UddoNzZQZLVdC4TNMIafeKS4dLLY9Ll/ AzE3PuFr/F2YFYFX7mu5lZ6SDu2wcCAm11n7o4MTd5Nz4igl4k4sAPuTMXR20TcPysPF JtIXCN6nvSF2ToU5SkQcoxQLhRM6rHPuTovHPSEhDhUpG6HgX4ZDthaBGGUfb8PAG7QC DuFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742828368; x=1743433168; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/3vwxiqm/CU6etvYPXXXelYSt+UGZmTt+cQ2Bq6SpGY=; b=Jjx5+MYtJhPxaHCd/WyfW7RcZl8TO2XhytXPjrzyIfKEco6PHGfj8QMhi5Tv9sRVMw jXuLhO+tshu1b+/x1RIv5GAEg7s9d6ClMVdYx7EG9cHSfDIU8IHuKyS272B6ttSkzTFz ahl4298bPZsbQu1AyiouCmC2q94VsBCvUfOilYmYhXHg77NTU/ZjICOflY6uYJzA81U1 vvozHwhi/djbgC9FPpV61cGJ0Qr8bUSB7Jsx/vzyDiI5fxoaUNNm8r8X4hrC8w6r6wz1 CecWJxJrqJQ9Rbk0WMRgujwFrGzUehjoOBx08YrJtrXGF4noVXITnUCKmunzrUDwlvwb RDrA== X-Gm-Message-State: AOJu0YxvmeL0nS05ynxr6q7gg/ENG6LiLreXCiSOvwflN83SAYLVz/XG SIPjb+4vdwwNjdsrvE+HNxxUctFDzhmhTmAFkqwBW8Sga7nmX4ex9H0fIZuRlw9pCXeCLBCvgdj +DR8DLWiEUWIwup1Xh62QJHw+Ou0= X-Gm-Gg: ASbGnctXPCFXN3bwKjq+twV04tEcoRAieViSppABIrsJT05kPM5V2HKaDN/wNKCoLKd lCZ7TFkLZfGlMF3Lb0Du+ceMQIDbrbLDJMADZeTPMkI//97EVFL3UOIThRWYgDI5qAXTciVh+up QOa551AGmB6BaLW80U3asX7fNhuYA= X-Google-Smtp-Source: AGHT+IH6LmVZ/UIqhoW+JXPfDlCqoYDGERM1xumeLdO/YkEr3TQQB6voRLcVyWySjiOqfZlcQBs0VWjcbi35b+p2X3A= X-Received: by 2002:a05:6a00:4648:b0:730:927c:d451 with SMTP id d2e1a72fcca58-73905a234f3mr22338241b3a.20.1742828367930; Mon, 24 Mar 2025 07:59:27 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 15:58:49 +0100 X-Gm-Features: AQ5f1JonC5p5Mrok0qCItX9CeQnUEBYMdpwoCFwYfanmI-TTlFPxAGbK2qYYatY Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c66db6063117d908" X-Spamd-Result: default: False [2.15 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; NEURAL_SPAM_SHORT(0.15)[0.150]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::630:from]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLx696b0yz3XDs X-Spamd-Bar: ++ --000000000000c66db6063117d908 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Oh sorry. I forgot to add the following bhyve parameters necessary to passthru my GPU :D -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto wrote: > Usually I use this kind of script to launch a bhyve vm : > > #!/bin/sh > > setxkbmap it > kldload vmm.ko > vms=3D"$(ls /dev/vmm/*)" > vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" > kldload vmm.ko > > if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." > kldload nvidia-modeset > devctl clear driver -f pci0:2:0:0 > devctl set driver -f pci0:2:0:0 ppt > else > echo "rtx 2080ti slot 2/0/0 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." > devctl clear driver -f pci0:2:0:1 > devctl set driver -f pci0:2:0:1 ppt > else > echo "rtx 2080ti slot 2/0/1 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." > devctl clear driver -f pci0:2:0:2 > devctl set driver -f pci0:2:0:2 ppt > else > echo "rtx 2080ti slot 2/0/2 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." > devctl clear driver -f pci0:2:0:3 > devctl set driver -f pci0:2:0:3 ppt > else > echo "rtx 2080ti slot 2/0/3 is already attached to ppt" > fi > > echo "rtx 2080ti is fully attached to ppt" > > for vm in $vms; do > session=3D"${vm##*/}" > echo "bhyve session =3D $session" > echo "vnc session =3D $vncs" > if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then > > printf 'VNC session not found,destroying ghost vms\n' > bhyvectl --vm=3D$session --destroy > else > printf 'Found VNC session %s\n' > "${session},no ghost vms found,not destroying them" > fi > done > > vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (AM13N4CZ)/ && d{print d}'` > echo "Seagate M3 Portable 1.8T ; $vmdisk0" > > vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2015040204055E)/ && d{print d}'` > echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" > > vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2027285F1175)/ && d{print d}'` > echo "CT1000P1SSD8 ; $vmdisk2" > > vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (20130508005976F)/ && d{print d}'` > echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" > > vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (BE0191510218)/ && d{print d}'` > echo "G-DRIVE USB ; $vmdisk4" > > vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (38234B4237354B45)/ && d{print d}'` > echo "Elements 25A3 ; $vmdisk5" > > vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (WD-WCAV2X797309)/ && d{print d}'` > echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" > > vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (20140108006C)/ && d{print d}'` > echo "Corsair Force 3 SSD ; $vmdisk7" > > vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (1924E50B2AE5)/ && d{print d}'` > echo "CT500MX500SSD4 ; $vmdisk8" > > vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (0774921DDC4200A6)/ && d{print d}'` > echo "SanDisk Cruzer-15GB ; $vmdisk9" > > vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (60A44D4138D8F311190A0149)/ && d{print d}'` > echo "Kingston DataTraveler 2.0 ; $vmdisk10" > > vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (575845483038524844323238)/ && d{print d}'` > echo "WD 2500BMV External ; $vmdisk11" > > vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (57442E575845323039544433303334)/ && d{print d}'` > echo "WD 3200BMV External ; $vmdisk12" > > vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2414E989076B)/ && d{print d}'` > echo "CT500BX500SSD1 ; $vmdisk13" > > /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H= -A \ > -s 0,hostbridge \ > -s 1,ahci-hd,/dev/$vmdisk13 \ > -s 10,virtio-net,tap6 \ > -s 11,virtio-9p,sharename=3D/ \ > -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ > vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & > > As you can see,each vm has a unique ID (connected with the vncviewer > instance) and the vncviewer window is launched as soon as the bhyve vm is > called...and my Geforce RTX 2080 ti is passed through inside the vm. > > > On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto > wrote: > >> First of all,you could use a more flexible technique than using the >> pptdevs on /boot/loader.conf. I would use a script like this : >> >> 1) >> >> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >> echo "18/0/0 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:18:0:0 >> devctl set driver -f pci0:18:0:0 ppt >> else >> echo "18/0/0 already attached to ppt" >> fi >> >> 2) I would not like to use vm-bhyve. It adds only some confusion... >> >> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >> shamim.shahriar@gmail.com> wrote: >> >>> Good afternoon everyone. >>> >>> I am trying to have VMs with GPU passthrough. The setup is a Dell serve= r >>> with NVidia Quadro RTX 6000/8000 installed already. I have checked the >>> device IDs and put pptdevs in place >>> >>> # cat /boot/loader.conf >>> pptdevs=3D"18/0/0 19/0/0" >>> pptdevs2=3D"72/0/0 73/0/0" >>> >>> this is showing the GPUs as pptdev on the pciconf >>> >>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>> vendor =3D 'NVIDIA Corporation' >>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>> class =3D display >>> subclass =3D 3D >>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>> vendor =3D 'NVIDIA Corporation' >>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>> class =3D display >>> subclass =3D 3D >>> >>> as I am using vm-bhyve, I have put the configuration as below >>> >>> # cat /mnt/VMs/jagadish/jagadish.conf >>> loader=3D"uefi" >>> cpu=3D16 >>> memory=3D128G >>> xhci_mouse=3D"yes" >>> debug=3D"true" >>> >>> graphics=3D"yes" >>> graphics_listen=3D"127.0.0.1" >>> graphics_port=3D"5920" >>> graphics_res=3D"1024x768" >>> graphics_wait=3D"no" >>> #graphics_vga=3D"io" >>> >>> network0_type=3D"virtio-net" >>> network0_switch=3D"swUNI" >>> network0_mac=3D"58:9c:fc:06:3f:af" >>> >>> disk0_type=3D"nvme" >>> disk0_name=3D"jagadish-disk0.img" >>> #disk0_size=3D"128G" >>> >>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>> >>> passthru0=3D"19/0/0=3D6:0" >>> >>> bhyve_options=3D"-A -H -P" >>> #END >>> >>> however, when I start the vm (for OS installation to start with), it >>> shows it is running but I am unable to access the VNC for a while, then >>> when I finally manage to connect via VNC, there is nothing on the scree= n, >>> just blank dark screen >>> >>> the vm-bhyve.log shows >>> >>> Mar 24 13:59:26: initialising >>> Mar 24 13:59:26: [loader: uefi] >>> Mar 24 13:59:26: [cpu: 16] >>> Mar 24 13:59:26: [memory: 128G] >>> Mar 24 13:59:26: [hostbridge: standard] >>> Mar 24 13:59:26: [com ports: com1] >>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>> Mar 24 13:59:26: [debug mode: true] >>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>> Mar 24 13:59:26: [primary disk dev: file] >>> Mar 24 13:59:26: initialising network device tap0 >>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>> Mar 24 13:59:26: booting >>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>> Mar 24 13:59:26: [bhyve iso device: -s >>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>> Mar 24 13:59:26: starting bhyve (run 1) >>> >>> based on what I can see, and a little out of desparation, I decided to >>> run the installer in the foreground, and below is what I got >>> >>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>> Starting jagadish >>> * found guest in /mnt/VMs/jagadish >>> * booting... >>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>> >>> it stays there for as long as I wait until I poweroff the vm. >>> >>> tried with Debian installer, >>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>> Starting jagadish >>> * found guest in /mnt/VMs/jagadish >>> * booting... >>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>> >>> >>> ideally I need to install Debian for my user base on these GPU based >>> systems. But even that is proving to be impossible since the installer = does >>> not move any further from where it is. >>> >>> Any thoughts/ideas/suggestions what I else I can try to make this work? >>> Anything that you have tried that worked? have I missed something? >>> >>> Would appreciate any and all thoughts/suggestions >>> >>> best regards >>> SS >>> >> >> >> -- >> Mario. >> > > > -- > Mario. > --=20 Mario. --000000000000c66db6063117d908 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Oh sorry. I forgot to add the following bhyve paramet= ers necessary to passthru my GPU :D

-s 8:0,passthr= u,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3= ,passthru,2/0/3 \

On Mon, Mar 24, 2025 at 3:57=E2=80= =AFPM Mario Marietto <marietto= 2008@gmail.com> wrote:
Usually I use this kind of script to la= unch a bhyve vm :

#!/bin/sh

setxkbmap it
kldl= oad vmm.ko
vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | = awk '/vncviewer [0]/{print $6}')"
kldload vmm.ko

if = ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
echo "rtx = 2080ti slot 2/0/0 is not attached to ppt,attaching..."
kldload nvid= ia-modeset
devctl clear driver -f pci0:2:0:0
devctl set driver -f pci= 0:2:0:0 ppt
else
echo "rtx 2080ti slot 2/0/0 is already attached= to ppt"
fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt&= quot;; then
echo "rtx 2080ti slot 2/0/1 is not attached to ppt,atta= ching..."
devctl clear driver -f pci0:2:0:1
devctl set driver -f= pci0:2:0:1 ppt
else
echo "rtx 2080ti slot 2/0/1 is already atta= ched to ppt"
fi

if ! pciconf -l pci0:2:0:2 | grep -q "^= ppt"; then
echo "rtx 2080ti slot 2/0/2 is not attached to ppt,= attaching..."
devctl clear driver -f pci0:2:0:2
devctl set drive= r -f pci0:2:0:2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already = attached to ppt"
fi

if ! pciconf -l pci0:2:0:3 | grep -q &qu= ot;^ppt"; then
echo "rtx 2080ti slot 2/0/3 is not attached to = ppt,attaching..."
devctl clear driver -f pci0:2:0:3
devctl set d= river -f pci0:2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is alre= ady attached to ppt"
fi

echo "rtx 2080ti is fully attac= hed to ppt"

for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve sessi= on =3D $session"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "vnc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}&qu= ot; | grep "${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 printf 'VNC session not found,destroying ghost vms\n'=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' &qu= ot;${session},no ghost vms found,not destroying them"
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geo= m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ &= & d{print d}'`
echo "Seagate M3 Portable 1.8T ; $vmdisk0&qu= ot;

vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (2015040204055E)/ && d{print d}'`
echo "TOSHIBA= External USB 3.0 1.8T ; $vmdisk1"

vmdisk2=3D`geom disk list | = awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{pr= int d}'`
echo "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`= geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2013050800597= 6F)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 932= GB ; $vmdisk3"

vmdisk4=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'`
ech= o "G-DRIVE USB ; $vmdisk4"

vmdisk5=3D`geom disk list | awk= '/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{p= rint d}'`
echo "Elements 25A3 ; $vmdisk5"

vmdisk6= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X= 797309)/ && d{print d}'`
echo "WDC WD3200AAJS-00L7A0 ; = 298 GB ; $vmdisk6"

vmdisk7=3D`geom disk list | awk '/^Geom = name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'`
= echo "Corsair Force 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk= list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ &&a= mp; d{print d}'`
echo "CT500MX500SSD4 ; $vmdisk8"

v= mdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (077= 4921DDC4200A6)/ && d{print d}'`
echo "SanDisk Cruzer-15= GB ; $vmdisk9"

vmdisk10=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}&= #39;`
echo "Kingston DataTraveler 2.0 ; $vmdisk10"

vmdi= sk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57584= 5483038524844323238)/ && d{print d}'`
echo "WD 2500BMV = External ; $vmdisk11"

vmdisk12=3D`geom disk list | awk '/^G= eom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ &&= ; d{print d}'`
echo "WD 3200BMV External ; $vmdisk12"
<= br>vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:= (2414E989076B)/ && d{print d}'`
echo "CT500BX500SSD1 ;= $vmdisk13"

/usr/sbin/./bhyve-win -S -c sockets=3D4= ,cores=3D1,threads=3D1 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-h= d,/dev/$vmdisk13 \
-s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename= =3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/loc= al/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:6 < /dev/null & sl= eep 5 && vncviewer 0:6 &

As you can se= e,each vm has a unique ID (connected with the vncviewer instance) and the v= ncviewer window is launched as soon as the bhyve vm is called...and my Gefo= rce RTX 2080 ti is passed through inside the vm.


On Mon, Mar 24, 2= 025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">
Firs= t of all,you could use a more flexible technique than using the pptdevs on = /boot/loader.conf. I would use a script like this :

1)

if ! pciconf -l pci0:18:0:0 | grep -q "^= ppt"; then
echo "18/0/0 is not attached to ppt,attaching...&= quot;
devctl clear driver -f pci0:18:0:0
devctl set driver -f pci0:18= :0:0 ppt
else
echo "18/0/0 already attached to ppt"
fi

2) I would not like to use vm-bhyve. It adds only som= e confusion...

On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar= <shamim.= shahriar@gmail.com> wrote:
Good afternoon everyone.

I am try= ing to have VMs with GPU passthrough. The setup is a Dell server with NVidi= a Quadro RTX 6000/8000 installed already. I have checked the device IDs and= put pptdevs in place

# cat /boot/loader.conf
pptdevs=3D"18/= 0/0 19/0/0"
pptdevs2=3D"72/0/0 73/0/0"

this is sho= wing the GPUs as pptdev on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2= =A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D= 0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0= =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt= 1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
= =C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2= =A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'= ;
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 s= ubclass =C2=A0 =3D 3D

as I am using vm-bhyve, I have put the configu= ration as below

# cat /mnt/VMs/jagadish/jagadish.conf
loader=3D&q= uot;uefi"
cpu=3D16
memory=3D128G
xhci_mouse=3D"yes"=
debug=3D"true"

graphics=3D"yes"
graphics_= listen=3D"127.0.0.1"
graphics_port=3D"5920"
graph= ics_res=3D"1024x768"
graphics_wait=3D"no"
#graphi= cs_vga=3D"io"

network0_type=3D"virtio-net"
ne= twork0_switch=3D"swUNI"
network0_mac=3D"58:9c:fc:06:3f:af= "

disk0_type=3D"nvme"
disk0_name=3D"jagadish-= disk0.img"
#disk0_size=3D"128G"

uuid=3D"966e9= 09b-1293-11ef-a9a4-e4434bfe34de"

passthru0=3D"19/0/0=3D6:0= "

bhyve_options=3D"-A -H -P"
#END

however, = when I start the vm (for OS installation to start with), it shows it is run= ning but I am unable to access the VNC for a while, then when I finally man= age to connect via VNC, there is nothing on the screen, just blank dark scr= een

the vm-bhyve.log shows

Mar 24 13:59:26: initialising
M= ar 24 13:59:26: =C2=A0[loader: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]Mar 24 13:59:26: =C2=A0[memory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridg= e: standard]
Mar 24 13:59:26: =C2=A0[com ports: com1]
Mar 24 13:59:26= : =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: = =C2=A0[debug mode: true]
Mar 24 13:59:26: =C2=A0[primary disk: jagadish-= disk0.img]
Mar 24 13:59:26: =C2=A0[primary disk dev: file]
Mar 24 13:= 59:26: initialising network device tap0
Mar 24 13:59:26: adding tap0 -&g= t; vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (= swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:26: =C2=A0[bhyve op= tions: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_= UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 1= 3:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/V= Ms/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:= af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet= ]
Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev/nmdm-jagadish.1A]<= br>Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/l= ubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26: starting bhyve (ru= n 1)

based on what I can see, and a little out of desparation, I dec= ided to run the installer in the foreground, and below is what I got
# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
Starting = jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 * booting..= .
fbuf frame buffer base: 0x112245400000 [sz 16777216]

it stays t= here for as long as I wait until I poweroff the vm.

tried with Debia= n installer,
# vm install -f jagadish debian-12.5.0-amd64-netinst.isoStarting jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 = * booting...
fbuf frame buffer base: 0x2747e2400000 [sz 16777216]

ideally I need to install Debian for my user base on these GPU based s= ystems. But even that is proving to be impossible since the installer does = not move any further from where it is.

Any thoughts/ideas/suggestion= s what I else I can try to make this work? Anything that you have tried tha= t worked? have I missed something?

Would appreciate any and all thou= ghts/suggestions

best regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000c66db6063117d908-- From nobody Mon Mar 24 15:14:57 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLxSL0ytDz5s2Z8 for ; Mon, 24 Mar 2025 15:15:14 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yw1-x1132.google.com (mail-yw1-x1132.google.com [IPv6:2607:f8b0:4864:20::1132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLxSK4jStz3cDN for ; Mon, 24 Mar 2025 15:15:13 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1132.google.com with SMTP id 00721157ae682-6febbd3b75cso38213307b3.0 for ; Mon, 24 Mar 2025 08:15:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1742829312; x=1743434112; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=A1QGpzQwclM4jL9LhgL1zUXvVGOTcuboFRq9PgBNjLk=; b=QvyoKWw2KZC4R59rwxLMmGo9YBSGWoQgOlcfIYgW1Ja80mKSlFmQV/a/E7aL8KCvqD XSCzzJ2S1U+Xv9XBqAF7ATguLYNtQcEX4Xx5oVZj3jcX+ylVb4+/p8Hqx4mhKMdp77sG mN2IzNxPo0dIbrsv/3wY3EtkvtompH7UvNXBg0EngHYzf5D/zb7i4Jw8mONlqEx53I7D NTkFXvRhxoVPGU7oTJUaRuBIOHLEq6IFIkQg75Nu6O0+c91kCb+5nAoKFWm6Z2fmIXDl zi+FppUaKNeYxPoadSYthE+lbOt1YKoDgtkxVL27kGSgGbJDagBu48CLMpf2UIr18Ow0 52Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742829312; x=1743434112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=A1QGpzQwclM4jL9LhgL1zUXvVGOTcuboFRq9PgBNjLk=; b=CGuPpmgsK1wfZqOzDJ5pXPpRoTDQSFZ1CHWyDFu+g/Hep7k/n4HySPpiU80sipX4S0 +AHGtsTtykOGDZq2Tw+MBhNjsQj/nOM7IhJJ9IH8N+4xwsHBmxwMfL0UaBBX2fMFNLwP lUQhQhKMv96YEqGYwlHpEBtc9yq2vesg72DRSJTuMFI0dFu/2GlCZQ5BAiVj4fX9KY8I yjatQqSQqG03DcpXo5R4sGVqQ/JZdJbShFcJG7sT107hszHSc3OGYLEC+Ua/zCzKGr00 dSsTENh1UshL7rO0DpKn0jJ1BdLJgHzXJvjB6AXHY90oBAk7sqMSz+JVhdtwKe1FnXpK p1NQ== X-Forwarded-Encrypted: i=1; AJvYcCX/108NLRMJJUqxyVHWiFaOiEJn6tIujrai+50iJU5uJG0QXy/ryr2bqHvoBaG8fXQCbHzaTtHcN7g/SRN/YnFltWm4jAtv@freebsd.org X-Gm-Message-State: AOJu0YyoP3vKHfKNa/bRDPS9Brea3s0YBpMG8gcFAlY4TSdGjY0qImpt c317S2U5QQ98pkK+n9ECT3N1IvaclBO2psdt7Do9E6hE1lhshSmbyglM5mNRdvUSglUP6pRHxqU = X-Gm-Gg: ASbGncsgliePSsc5lF59REmbues18PZ4VpT8+OgjnbPZCfBkzWLiOgFqXetbVK9Pbp9 RgI+3UH82ni6vUMiccPi+B3Po8h1SYKd1yQL7L89YnADAsM/On/Bw3ZkVlV+k2uEaiHRJcIfKm0 0/lrBRP4+lFL4Jz5h6rfd1bgVuQTvFRR62/LJf+bixtRIfC1cKHDidQmgYI7Z+XkN/1M5uKDr0J T6i1dpWr3SrMg3Pm4c1m5B7WxOZR1l5ffWv9J8iX3g3tozMc4wiIUDR4mWi3mccPixihXTreshM x9YxWAHeYxYlLUTQPrVIFX93p8fwfcy18T8I0XbbKnHi7VpPccHjyyy9yX9RMrSuKPBio2LQYXp gwYWnLg== X-Google-Smtp-Source: AGHT+IHkdyhzQ21g5YcdHKEgfcNaD+BLnFi8LLDVlJDFPsxqGlcaD+yRfBkwwjhpze71H5bYZu2Szg== X-Received: by 2002:a05:690c:6284:b0:6fb:1f78:d9ee with SMTP id 00721157ae682-700bac5cbfamr190670577b3.15.1742829312394; Mon, 24 Mar 2025 08:15:12 -0700 (PDT) Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com. [209.85.219.174]) by smtp.gmail.com with ESMTPSA id 00721157ae682-700ba76d120sm15735867b3.42.2025.03.24.08.15.11 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 24 Mar 2025 08:15:11 -0700 (PDT) Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-e643f235a34so3077723276.1 for ; Mon, 24 Mar 2025 08:15:11 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCUqvbbdXp9GWWYhmsekprduLVYppT+u8dOS43DF8399OwK21Q6Ry0djGmAKOvyurVJs/NTLUPw6DIqC43x3WZf7BzFzV7Ga@freebsd.org X-Received: by 2002:a05:6902:2609:b0:e60:9db3:af93 with SMTP id 3f1490d57ef6-e66a4db0502mr16840235276.24.1742829311389; Mon, 24 Mar 2025 08:15:11 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Tomek CEDRO Date: Mon, 24 Mar 2025 16:14:57 +0100 X-Gmail-Original-Message-ID: X-Gm-Features: AQ5f1JrvbMl3XeVqDNuH7JoKjFy5E1oqX3AriL9HHNx7rO1mPe8SrvK6g1ezwbg Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Mario Marietto Cc: Shamim Shahriar , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4ZLxSK4jStz3cDN X-Spamd-Bar: ---- One quick question guys, what is the reason to pass GPU to bhyve VM? Is it to get 3D acceleration on a local display or access computational resources of the GPU (i.e. CUDA/CL)? I guess the second option as you connect with VNC anyways and that introduces latencies anyways so its still not really possible to work with CAD for instance? -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Mon Mar 24 15:17:01 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLxVf61Mtz5s2Hq for ; Mon, 24 Mar 2025 15:17:14 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLxVf3k28z3cdj for ; Mon, 24 Mar 2025 15:17:14 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-6ff37565154so40930807b3.3 for ; Mon, 24 Mar 2025 08:17:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742829433; x=1743434233; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=rHB/pCiZf55pZFxH+d3H6XsfUXyPn6ob6xYdQGhZWtI=; b=HFdseBKdyoQlMoBUZI5WCeehNzLT4VhDy1vgsxbfzt4GAtJqsdQOIgQ83/b+ei05ja H0g8fFBlisknytLmnCvDi/HBaC8NNZQneJdM8nL/SZy3v27cDmyZdjjAzShihMvdKh4d g27IWy0jMazzpEUp6/t+ckdUHkuS9gpH6Dy6/AdFa73wgGzpwiBkh5FwQonM6WtVbNKe AzmUJNGXfZGUrJB82g8zvRqdJZW6NsmAMgKtpXxOthmoS+gRqoIgWxfAKefvZF0axqxb kWtyEaZtiZUzUQyGAU4wZKWc/AG6V4MSCP7Txb0UX5KIwR/WmHhL7ul1lFoUi007MmDw cZ5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742829433; x=1743434233; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=rHB/pCiZf55pZFxH+d3H6XsfUXyPn6ob6xYdQGhZWtI=; b=DFiPvx0PL+bk92c4ZFabhRn0qEP+KxYhMk5jtDitlv7QC3fx7S7gPasSH5NQOpGnXV yS+U2YNvQY0F8Ws+cHwfPg2sgUmCzybXJBypCoUjy/CsIXt7h0QgJuJX2Gzt4c+AV6G4 C/mtnXqHrS5eJ++3wriagQ1cPg6IkWuLi4lz0w6+f1QX2Oa4qO8XVVEpLNcS92nrqNxI WBJai224pL7XOY3qtKQvy2Xv70HTl2Y23Sbogm6p8in9ILsCzPe+VT7XBeBvabywFnPA YJhNLeA1nB8Tx6ySZYPXjSuk8vJRYNapnUGGhffIJ2WkhwrW83G/9Pv4W5qXNjEj+y5a hBkQ== X-Forwarded-Encrypted: i=1; AJvYcCXa9SmYD1fPP4FKiYU0vxIZmWxhEGLs1I10zowBw78lvj3GyJCoDFrdOvGJk7vboRAKMqfpO/utGt35RBqs45HwLzHPBdts@freebsd.org X-Gm-Message-State: AOJu0YxmZ8dFw2//hudnGeqfs+7CgHJypg5TrFz8Jeg+Y5x3vc1X24WC Bter+uo0gv/E8L2jtc0QlPpYccg+6VNTnjmjHEpCfvhuFbWw8o+U1wiWwxJzyfuXO2YPWf3TFS4 ka6ebHuo/XmhnssiCB6Pi8AQCPMdW X-Gm-Gg: ASbGncssO6o/I3Sd/1wPYNxMLVaV7B8tWg/wtVhauJGhqgYLx4mjgAAZ3ypBNpvFzzb RVOrdziR6Tx40Al7oGRSO7EUptUtPJ8+OB03g8YaNqf4RxQvc5OdYBgr4zwP8XN5YjqSh9I6dBy wRx38QK/yvXQy6doPGOghlH40= X-Google-Smtp-Source: AGHT+IEsK1e3e6K1q9v0TdwCh7hY/a+NrPF1VzZcNMChqP7PB6Bn86IcdFMeTw0/W0SjTrfkP5njLs+HdUGR7QdEvPw= X-Received: by 2002:a05:690c:7349:b0:6fb:b524:48bc with SMTP id 00721157ae682-700bad5da5cmr185478907b3.35.1742829433442; Mon, 24 Mar 2025 08:17:13 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Shamim Shahriar Date: Mon, 24 Mar 2025 15:17:01 +0000 X-Gm-Features: AQ5f1JrT7sQGiwnTuQmL0cKd-Fvubgm9HQ1kBwfp1UmAvMDwOc9Kew3DzYAAgPo Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Tomek CEDRO Cc: Mario Marietto , freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="00000000000048e051063118194c" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4ZLxVf3k28z3cdj X-Spamd-Bar: ---- --00000000000048e051063118194c Content-Type: text/plain; charset="UTF-8" Hi Tomek The reason in my case is to allow the colleagues to use CUDA/CL for their research. They want/need Debian based system, and that is what I intend to provide if possible. best regards SS On Mon, 24 Mar 2025 at 15:15, Tomek CEDRO wrote: > One quick question guys, what is the reason to pass GPU to bhyve VM? > Is it to get 3D acceleration on a local display or access > computational resources of the GPU (i.e. CUDA/CL)? I guess the second > option as you connect with VNC anyways and that introduces latencies > anyways so its still not really possible to work with CAD for > instance? > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > --00000000000048e051063118194c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Tomek

The reason in my case is to al= low the colleagues to use CUDA/CL for their research. They want/need Debian= based system, and that is what I intend to provide if possible.
=
best regards
SS

On Mon,= 24 Mar 2025 at 15:15, Tomek CEDRO <= tomek@cedro.info> wrote:
One quick question guys, what is the reason to pass GPU to = bhyve VM?
Is it to get 3D acceleration on a local display or access
computational resources of the GPU (i.e. CUDA/CL)? I guess the second
option as you connect with VNC anyways and that introduces latencies
anyways so its still not really possible to work with CAD for
instance?

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
--00000000000048e051063118194c-- From nobody Mon Mar 24 15:29:45 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLxnL4ywhz5s3Xk for ; Mon, 24 Mar 2025 15:29:58 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLxnL2tcbz3hs2 for ; Mon, 24 Mar 2025 15:29:58 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-700b17551cdso41146627b3.0 for ; Mon, 24 Mar 2025 08:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742830197; x=1743434997; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=b+nrJ8y5G8HwazmiK6gZkUF6xv2OuMkMw1IxKbfvbTI=; b=Wu0WKVB81X7EB40cXdudUFTS8f0RLLZyvd6+jImYt2/ATdlhxSk3or+PWuYEpXY9fx VRer0PByIMyUEfRh7tYb82IQB8LlZepggT4UvREehwuV+WUvnUpvCrTOb2eInxkIlYCZ A3E9BCWovIgt+Xce3/6zsEEuB7yrDfOpUXoEwG7F9BHetugLR/kF+Bxy1IkhAtdHh5Hu Tpg+VGOeHWrY9psCUIqJgIq1B64a9zP4fM80Hd/89ocDt/VKEulxfdPxm+sN4ilwphLl DvqRL5FNh/iLWAchov94gzaZUT+YRvMYVUkc12GS8S2E1OM/tvM22XvdXMN56Bzdm5KD kEeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742830197; x=1743434997; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=b+nrJ8y5G8HwazmiK6gZkUF6xv2OuMkMw1IxKbfvbTI=; b=jRoJ84N1P9qplS12FwNU4UcQ0Kz8BufMhJ0DxSeJ00+6BgDTylYspVvrLGxACbxNUe 24byj2upoTFhAoL6bzPkYK8U5aelb8fPrGgY5fhnBNERKotZuCSE9oMXhHGLOQhNgQ8G SJkbEs44/O8pO0tDQvIUNH3qrMLizhehbz8SZCRhMmlacTJu7+0jIk2oMcehZbX4RMXd sE2JRGIJkMqGzdnqeRBGF04B2xjqwAZSFLpui8IcXUiEU/r7o6hiswGKGhJWFg1S+4R0 lAUYvfToPp/7uEZ21LA9Odggw0/I87PLmLuKga1WobzCFQ1TloT2rXvlZzBCHn4SYdi0 9UYQ== X-Gm-Message-State: AOJu0Yz2Q6fOqVG7IapL53kw0zxpTFOnOkg5pJDj3V3YH42qMFj67gdP 6H5EcImWfZagExYnAjQM+Qci1R9F6hE3CiyyYZ0wVTkPxTOxbxCgVwSYlLEh+bLholY9zwJH6Ke ozWn+8vAIYUmwvhmPu+guj47HBSf2 X-Gm-Gg: ASbGncurr0DkffFYPAb74tFBrzLxjzIgmqn6sLou1xV1C6M/OkXSaF1sKvM1dVtHkr+ 7d4ShPDQwDrsCEjBZ919LT/b1uT9BZetyxhLaaFjHkontO3zFJxVRKaqUf4xGHfVjOWjwUiyfKR 4PvGvXjnzswz1SD3lYQow31Ck= X-Google-Smtp-Source: AGHT+IH5dOdXOtyElKV1Ck9vZy/oyI7b9aQf3+8uawzR3bMeMVMLVTZ72gG4IJekbbwxy6wArIqYy7CE4jrviSQJchg= X-Received: by 2002:a05:690c:4444:b0:6fe:e79f:bd8f with SMTP id 00721157ae682-700bacd58eemr193187517b3.26.1742830197228; Mon, 24 Mar 2025 08:29:57 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Shamim Shahriar Date: Mon, 24 Mar 2025 15:29:45 +0000 X-Gm-Features: AQ5f1JrgPqhMVT4Ec2V0-vi-yijJ0Ma3B5TrzkMCb3bbxqMi9ooaaETdvjhTXc8 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Mario Marietto Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000cf4c600631184611" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4ZLxnL2tcbz3hs2 X-Spamd-Bar: ---- --000000000000cf4c600631184611 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mario I will try the options you have used later and let you know how it goes. Not sure if this is intentional or typo, but I don't recall any bhyve-win binary in my system. Best regards SS On Mon, 24 Mar 2025 at 14:59, Mario Marietto wrote= : > Oh sorry. I forgot to add the following bhyve parameters necessary to > passthru my GPU :D > > -s 8:0,passthru,2/0/0 \ > -s 8:1,passthru,2/0/1 \ > -s 8:2,passthru,2/0/2 \ > -s 8:3,passthru,2/0/3 \ > > On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto > wrote: > >> Usually I use this kind of script to launch a bhyve vm : >> >> #!/bin/sh >> >> setxkbmap it >> kldload vmm.ko >> vms=3D"$(ls /dev/vmm/*)" >> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >> kldload vmm.ko >> >> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >> kldload nvidia-modeset >> devctl clear driver -f pci0:2:0:0 >> devctl set driver -f pci0:2:0:0 ppt >> else >> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >> fi >> >> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:2:0:1 >> devctl set driver -f pci0:2:0:1 ppt >> else >> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >> fi >> >> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:2:0:2 >> devctl set driver -f pci0:2:0:2 ppt >> else >> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >> fi >> >> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:2:0:3 >> devctl set driver -f pci0:2:0:3 ppt >> else >> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >> fi >> >> echo "rtx 2080ti is fully attached to ppt" >> >> for vm in $vms; do >> session=3D"${vm##*/}" >> echo "bhyve session =3D $session" >> echo "vnc session =3D $vncs" >> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; the= n >> >> printf 'VNC session not found,destroying ghost vms\n' >> bhyvectl --vm=3D$session --destroy >> else >> printf 'Found VNC session %s\n' >> "${session},no ghost vms found,not destroying them" >> fi >> done >> >> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (AM13N4CZ)/ && d{print d}'` >> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >> >> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (2015040204055E)/ && d{print d}'` >> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >> >> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (2027285F1175)/ && d{print d}'` >> echo "CT1000P1SSD8 ; $vmdisk2" >> >> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (20130508005976F)/ && d{print d}'` >> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >> >> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (BE0191510218)/ && d{print d}'` >> echo "G-DRIVE USB ; $vmdisk4" >> >> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (38234B4237354B45)/ && d{print d}'` >> echo "Elements 25A3 ; $vmdisk5" >> >> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (WD-WCAV2X797309)/ && d{print d}'` >> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >> >> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (20140108006C)/ && d{print d}'` >> echo "Corsair Force 3 SSD ; $vmdisk7" >> >> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (1924E50B2AE5)/ && d{print d}'` >> echo "CT500MX500SSD4 ; $vmdisk8" >> >> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (0774921DDC4200A6)/ && d{print d}'` >> echo "SanDisk Cruzer-15GB ; $vmdisk9" >> >> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (60A44D4138D8F311190A0149)/ && d{print d}'` >> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >> >> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (575845483038524844323238)/ && d{print d}'` >> echo "WD 2500BMV External ; $vmdisk11" >> >> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (57442E575845323039544433303334)/ && d{print d}'` >> echo "WD 3200BMV External ; $vmdisk12" >> >> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (2414E989076B)/ && d{print d}'` >> echo "CT500BX500SSD1 ; $vmdisk13" >> >> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -= H -A \ >> -s 0,hostbridge \ >> -s 1,ahci-hd,/dev/$vmdisk13 \ >> -s 10,virtio-net,tap6 \ >> -s 11,virtio-9p,sharename=3D/ \ >> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >> -s 31,lpc \ >> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >> >> As you can see,each vm has a unique ID (connected with the vncviewer >> instance) and the vncviewer window is launched as soon as the bhyve vm i= s >> called...and my Geforce RTX 2080 ti is passed through inside the vm. >> >> >> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >> wrote: >> >>> First of all,you could use a more flexible technique than using the >>> pptdevs on /boot/loader.conf. I would use a script like this : >>> >>> 1) >>> >>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>> echo "18/0/0 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:18:0:0 >>> devctl set driver -f pci0:18:0:0 ppt >>> else >>> echo "18/0/0 already attached to ppt" >>> fi >>> >>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>> >>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>> shamim.shahriar@gmail.com> wrote: >>> >>>> Good afternoon everyone. >>>> >>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>> server with NVidia Quadro RTX 6000/8000 installed already. I have chec= ked >>>> the device IDs and put pptdevs in place >>>> >>>> # cat /boot/loader.conf >>>> pptdevs=3D"18/0/0 19/0/0" >>>> pptdevs2=3D"72/0/0 73/0/0" >>>> >>>> this is showing the GPUs as pptdev on the pciconf >>>> >>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>> vendor =3D 'NVIDIA Corporation' >>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>> class =3D display >>>> subclass =3D 3D >>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>> vendor =3D 'NVIDIA Corporation' >>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>> class =3D display >>>> subclass =3D 3D >>>> >>>> as I am using vm-bhyve, I have put the configuration as below >>>> >>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>> loader=3D"uefi" >>>> cpu=3D16 >>>> memory=3D128G >>>> xhci_mouse=3D"yes" >>>> debug=3D"true" >>>> >>>> graphics=3D"yes" >>>> graphics_listen=3D"127.0.0.1" >>>> graphics_port=3D"5920" >>>> graphics_res=3D"1024x768" >>>> graphics_wait=3D"no" >>>> #graphics_vga=3D"io" >>>> >>>> network0_type=3D"virtio-net" >>>> network0_switch=3D"swUNI" >>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>> >>>> disk0_type=3D"nvme" >>>> disk0_name=3D"jagadish-disk0.img" >>>> #disk0_size=3D"128G" >>>> >>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>> >>>> passthru0=3D"19/0/0=3D6:0" >>>> >>>> bhyve_options=3D"-A -H -P" >>>> #END >>>> >>>> however, when I start the vm (for OS installation to start with), it >>>> shows it is running but I am unable to access the VNC for a while, the= n >>>> when I finally manage to connect via VNC, there is nothing on the scre= en, >>>> just blank dark screen >>>> >>>> the vm-bhyve.log shows >>>> >>>> Mar 24 13:59:26: initialising >>>> Mar 24 13:59:26: [loader: uefi] >>>> Mar 24 13:59:26: [cpu: 16] >>>> Mar 24 13:59:26: [memory: 128G] >>>> Mar 24 13:59:26: [hostbridge: standard] >>>> Mar 24 13:59:26: [com ports: com1] >>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>> Mar 24 13:59:26: [debug mode: true] >>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>> Mar 24 13:59:26: [primary disk dev: file] >>>> Mar 24 13:59:26: initialising network device tap0 >>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>> Mar 24 13:59:26: booting >>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>> Mar 24 13:59:26: [bhyve iso device: -s >>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>> Mar 24 13:59:26: starting bhyve (run 1) >>>> >>>> based on what I can see, and a little out of desparation, I decided to >>>> run the installer in the foreground, and below is what I got >>>> >>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>> Starting jagadish >>>> * found guest in /mnt/VMs/jagadish >>>> * booting... >>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>> >>>> it stays there for as long as I wait until I poweroff the vm. >>>> >>>> tried with Debian installer, >>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>> Starting jagadish >>>> * found guest in /mnt/VMs/jagadish >>>> * booting... >>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>> >>>> >>>> ideally I need to install Debian for my user base on these GPU based >>>> systems. But even that is proving to be impossible since the installer= does >>>> not move any further from where it is. >>>> >>>> Any thoughts/ideas/suggestions what I else I can try to make this work= ? >>>> Anything that you have tried that worked? have I missed something? >>>> >>>> Would appreciate any and all thoughts/suggestions >>>> >>>> best regards >>>> SS >>>> >>> >>> >>> -- >>> Mario. >>> >> >> >> -- >> Mario. >> > > > -- > Mario. > --000000000000cf4c600631184611 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mario

I will try the options you hav= e used later and let you know how it goes. Not sure if this is intentional = or typo, but I don't recall any bhyve-win binary in my system.

Best regards
SS

On Mo= n, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wrote:
Oh sorry. I forgot t= o add the following bhyve parameters necessary to passthru my GPU :D
<= div>
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
= -s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \

On Mon, Mar 24, 2025 at 3= :57=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
Usually I us= e this kind of script to launch a bhyve vm :

#!/bin/sh<= br>
setxkbmap it
kldload vmm.ko
vms=3D"$(ls /dev/vmm/*)"=
vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"kldload vmm.ko

if ! pciconf -l pci0:2:0:0 | grep -q "^ppt&quo= t;; then
echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attachi= ng..."
kldload nvidia-modeset
devctl clear driver -f pci0:2:0:0<= br>devctl set driver -f pci0:2:0:0 ppt
else
echo "rtx 2080ti slo= t 2/0/0 is already attached to ppt"
fi

if ! pciconf -l pci0:= 2:0:1 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/1 = is not attached to ppt,attaching..."
devctl clear driver -f pci0:2:= 0:1
devctl set driver -f pci0:2:0:1 ppt
else
echo "rtx 2080ti= slot 2/0/1 is already attached to ppt"
fi

if ! pciconf -l p= ci0:2:0:2 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/= 0/2 is not attached to ppt,attaching..."
devctl clear driver -f pci= 0:2:0:2
devctl set driver -f pci0:2:0:2 ppt
else
echo "rtx 20= 80ti slot 2/0/2 is already attached to ppt"
fi

if ! pciconf = -l pci0:2:0:3 | grep -q "^ppt"; then
echo "rtx 2080ti slo= t 2/0/3 is not attached to ppt,attaching..."
devctl clear driver -f= pci0:2:0:3
devctl set driver -f pci0:2:0:3 ppt
else
echo "rt= x 2080ti slot 2/0/3 is already attached to ppt"
fi

echo &quo= t;rtx 2080ti is fully attached to ppt"

for vm in $vms; do
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${v= m##*/}" =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "bhyve session =3D $session"
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "vnc session =3D $vncs"=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '= %s\n' "${vncs}" | grep "${session#vm}"; then =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'VNC session not found,destro= ying ghost vms\n'
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --v= m=3D$session --destroy
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC = session %s\n' "${session},no ghost vms found,not destroying them&q= uot;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
done<= br>
vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ide= nt: (AM13N4CZ)/ && d{print d}'`
echo "Seagate M3 Portab= le 1.8T ; $vmdisk0"

vmdisk1=3D`geom disk list | awk '/^Geom= name: /{d=3D$NF} /^ *ident: (2015040204055E)/ && d{print d}'`<= br>echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"

vmdisk2= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1= 175)/ && d{print d}'`
echo "CT1000P1SSD8 ; $vmdisk2&quo= t;

vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *= ident: (20130508005976F)/ && d{print d}'`
echo "TOSHIBA= External USB 3.0 932 GB ; $vmdisk3"

vmdisk4=3D`geom disk list = | awk '/^Geom name: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{= print d}'`
echo "G-DRIVE USB ; $vmdisk4"

vmdisk5=3D= `geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (38234B423735= 4B45)/ && d{print d}'`
echo "Elements 25A3 ; $vmdisk5&q= uot;

vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^= *ident: (WD-WCAV2X797309)/ && d{print d}'`
echo "WDC W= D3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"

vmdisk7=3D`geom disk list= | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d= {print d}'`
echo "Corsair Force 3 SSD ; $vmdisk7"

v= mdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (192= 4E50B2AE5)/ && d{print d}'`
echo "CT500MX500SSD4 ; $vmd= isk8"

vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$= NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}'`
echo &quo= t;SanDisk Cruzer-15GB ; $vmdisk9"

vmdisk10=3D`geom disk list | = awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ &am= p;& d{print d}'`
echo "Kingston DataTraveler 2.0 ; $vmdisk1= 0"

vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF= } /^ *ident: (575845483038524844323238)/ && d{print d}'`
ech= o "WD 2500BMV External ; $vmdisk11"

vmdisk12=3D`geom disk = list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57442E57584532303954443= 3303334)/ && d{print d}'`
echo "WD 3200BMV External ; $= vmdisk12"

vmdisk13=3D`geom disk list | awk '/^Geom name: /{= d=3D$NF} /^ *ident: (2414E989076B)/ && d{print d}'`
echo &qu= ot;CT500BX500SSD1 ; $vmdisk13"

/usr/sbin/./bhyve-wi= n -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H -A \
-s 0,hostbrid= ge \
-s 1,ahci-hd,/dev/$vmdisk13 \
-s 10,virtio-net,tap6 \
-s 11,v= irtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
-s 31,lpc \
-= l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:6 < = /dev/null & sleep 5 && vncviewer 0:6 &

=
As you can see,each vm has a unique ID (connected with the vncviewer i= nstance) and the vncviewer window is launched as soon as the bhyve vm is ca= lled...and my Geforce RTX 2080 ti is passed through inside the vm.

On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmail.com>= wrote:
First of all,you could use a more flexible technique than usi= ng the pptdevs on /boot/loader.conf. I would use a script like this :
=

1)

if ! pciconf -l pci0:18:0:0= | grep -q "^ppt"; then
echo "18/0/0 is not attached to= ppt,attaching..."
devctl clear driver -f pci0:18:0:0
devctl set= driver -f pci0:18:0:0 ppt
else
echo "18/0/0 already attached to= ppt"
fi

2) I would not like to use vm-bhyve= . It adds only some confusion...

=
On Mon, Mar 24, 2025 at 3:22=E2=80=AF= PM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Good afternoon everyo= ne.

I am trying to have VMs with GPU passthrough. The setup is a Del= l server with NVidia Quadro RTX 6000/8000 installed already. I have checked= the device IDs and put pptdevs in place

# cat /boot/loader.conf
= pptdevs=3D"18/0/0 19/0/0"
pptdevs2=3D"72/0/0 73/0/0"=

this is showing the GPUs as pptdev on the pciconf

ppt0@pci0:= 18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 = =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2= =A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
= =C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subcla= ss =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200= rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de s= ubdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Cor= poration'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadr= o RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D displa= y
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D

as I am using vm-bhyve, I = have put the configuration as below

# cat /mnt/VMs/jagadish/jagadish= .conf
loader=3D"uefi"
cpu=3D16
memory=3D128G
xhci_mou= se=3D"yes"
debug=3D"true"

graphics=3D"ye= s"
graphics_listen=3D"127.0.0.1"
graphics_port=3D"= ;5920"
graphics_res=3D"1024x768"
graphics_wait=3D"= ;no"
#graphics_vga=3D"io"

network0_type=3D"vi= rtio-net"
network0_switch=3D"swUNI"
network0_mac=3D&qu= ot;58:9c:fc:06:3f:af"

disk0_type=3D"nvme"
disk0_na= me=3D"jagadish-disk0.img"
#disk0_size=3D"128G"
uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"

passthru0= =3D"19/0/0=3D6:0"

bhyve_options=3D"-A -H -P"
= #END

however, when I start the vm (for OS installation to start with= ), it shows it is running but I am unable to access the VNC for a while, th= en when I finally manage to connect via VNC, there is nothing on the screen= , just blank dark screen

the vm-bhyve.log shows

Mar 24 13:59:= 26: initialising
Mar 24 13:59:26: =C2=A0[loader: uefi]
Mar 24 13:59:2= 6: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[memory: 128G]
Mar 24 13:59= :26: =C2=A0[hostbridge: standard]
Mar 24 13:59:26: =C2=A0[com ports: com= 1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]Mar 24 13:59:26: =C2=A0[debug mode: true]
Mar 24 13:59:26: =C2=A0[prim= ary disk: jagadish-disk0.img]
Mar 24 13:59:26: =C2=A0[primary disk dev: = file]
Mar 24 13:59:26: initialising network device tap0
Mar 24 13:59:= 26: adding tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up ta= p0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:= 26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/u= efi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de= -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc= -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,ma= c=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768= -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev= /nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci= -cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26:= starting bhyve (run 1)

based on what I can see, and a little out of= desparation, I decided to run the installer in the foreground, and below i= s what I got

# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-dis= c1.iso
Starting jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish=C2=A0 * booting...
fbuf frame buffer base: 0x112245400000 [sz 16777216= ]

it stays there for as long as I wait until I poweroff the vm.
<= br>tried with Debian installer,
# vm install -f jagadish debian-12.5.0-= amd64-netinst.iso
Starting jagadish
=C2=A0 * found guest in /mnt/VMs/= jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x2747e2400000 [= sz 16777216]


ideally I need to install Debian for my user base o= n these GPU based systems. But even that is proving to be impossible since = the installer does not move any further from where it is.

Any though= ts/ideas/suggestions what I else I can try to make this work? Anything that= you have tried that worked? have I missed something?

Would apprecia= te any and all thoughts/suggestions

best regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000cf4c600631184611-- From nobody Mon Mar 24 15:38:52 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLy0Q6F13z5s3yt for ; Mon, 24 Mar 2025 15:39:34 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLy0P3dDHz3mF2 for ; Mon, 24 Mar 2025 15:39:33 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=XcYAzo0k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::102e as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x102e.google.com with SMTP id 98e67ed59e1d1-2ff6a98c638so9554063a91.0 for ; Mon, 24 Mar 2025 08:39:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742830771; x=1743435571; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+MCLZMN2fg5vUJVZKn41goHH+CnaYCjuVW1V5rWEFzQ=; b=XcYAzo0k+b6AJpOI27maG9LVYDhzYzllxJ7SgVIR7xSaHuWXDvXGVlCGsvoA6iJWI9 bsBO0MKJ7uA3vu0wVxdaBPD8JzS4GF4Wd6eMAFQgquNg8cuqPOiNtriRklR37d/UMX7N IRn/IzKkVbR5/JaZJfdDZQCGFvWedFBk3sBHXspuPOzPghE+2wtmdFuK9ztXs/gLLMyo WMIs0SFCT7VSuAQAgpySywJJfsDV4n+CKuyBSuVOuDNf1wV52k2930uThps7zUrHwLXQ v0TQMTIcoGMetBi+AKRl7o+bXBdvROXJsBToex7Udn2RC/VRIqZUDi123EuSneRFJVZl jAJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742830771; x=1743435571; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+MCLZMN2fg5vUJVZKn41goHH+CnaYCjuVW1V5rWEFzQ=; b=V2/dplHA0foPhmLOByI+EVGZomZx7aIevD7/SZAumY47yat/9r1qntKttnfoBE8A94 oVuA2zXyuNZlKofU7RfB7uMaiU6JyFp9lP5A+yHvUz783FEqOMgLssO7DeSxkxYprpak WQRWr7f04wtU4AfBGt3JEbdSGAalPNlnD02gwxsXaurRKDJzG3M0pWOJCA2Q2cFmfIwV bpYGyG4uOasVnD7eaHLiSuxgH91aG7A1BF+ABBs00PGTeXjfhNOrr/wXPpVybWeQOXBX jbxTDhrOmW0dAVcpXNh53yGasU9kV9QsUeHoL7cWad6gSVNgOrUwFk/FqkKYU3V1KDg1 fiyQ== X-Gm-Message-State: AOJu0YzQ85zj44FUp/hNu45I4ilW6Vdmo3XshZfYJQ0B7oNYHwBE6PrG LAtRfO8WzDc4VjgeoZ5sYncASY8hbV+DG4fwL+tSnYMROeUQ2PcQYjz0WSpRx58zqW9PErAT2X4 gMFBZCQLtGsBDwsqgGDTdWDSIjm0thzKr X-Gm-Gg: ASbGnctlW5WBDmzFEPJ2nAIYWaNeDFeKz4hFiv6Hl0P1iHL/1SxtaKt70lOJrIYWua+ WxonEExb4efGNJev0UOUsp4pkiG1g4YSLVPCStoONNV9JUHWSmFGvJ/0K/Ne8MVVrLdDwgQXnoN 7rHpYhCQX+c70tFMsBWUZU4cBd4Fs= X-Google-Smtp-Source: AGHT+IEVuGD6fKZZT69OzSv/dZZ6+G+xr1/nh2s9oTuIIg2MfaGmzSVCcoxNzlzdnwUeqXi2zp15M7DIKfpHwyYTmH4= X-Received: by 2002:a17:90b:53d0:b0:2ee:df70:1ff3 with SMTP id 98e67ed59e1d1-3030fd36f80mr26545538a91.0.1742830770504; Mon, 24 Mar 2025 08:39:30 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 16:38:52 +0100 X-Gm-Features: AQ5f1JpWesxAEyR4YhAda09II1-TG04rYs29DDE-0Cu-0LSoOkuYlEhYpimN32s Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000facef9063118686b" X-Spamd-Result: default: False [1.04 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; NEURAL_HAM_SHORT(-0.96)[-0.959]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102e:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLy0P3dDHz3mF2 X-Spamd-Bar: + --000000000000facef9063118686b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Shamim. You don't have any bhyve-win. You can use bhyve. I use bhyve-win to pass through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are two different executables,because for some reason,my bhyve-lin executable is not able to pass my GPU in a Windows vm. On Mon, Mar 24, 2025 at 4:29=E2=80=AFPM Shamim Shahriar wrote: > Hi Mario > > I will try the options you have used later and let you know how it goes. > Not sure if this is intentional or typo, but I don't recall any bhyve-win > binary in my system. > > Best regards > SS > > On Mon, 24 Mar 2025 at 14:59, Mario Marietto > wrote: > >> Oh sorry. I forgot to add the following bhyve parameters necessary to >> passthru my GPU :D >> >> -s 8:0,passthru,2/0/0 \ >> -s 8:1,passthru,2/0/1 \ >> -s 8:2,passthru,2/0/2 \ >> -s 8:3,passthru,2/0/3 \ >> >> On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto >> wrote: >> >>> Usually I use this kind of script to launch a bhyve vm : >>> >>> #!/bin/sh >>> >>> setxkbmap it >>> kldload vmm.ko >>> vms=3D"$(ls /dev/vmm/*)" >>> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >>> kldload vmm.ko >>> >>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >>> kldload nvidia-modeset >>> devctl clear driver -f pci0:2:0:0 >>> devctl set driver -f pci0:2:0:0 ppt >>> else >>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:1 >>> devctl set driver -f pci0:2:0:1 ppt >>> else >>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:2 >>> devctl set driver -f pci0:2:0:2 ppt >>> else >>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:3 >>> devctl set driver -f pci0:2:0:3 ppt >>> else >>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >>> fi >>> >>> echo "rtx 2080ti is fully attached to ppt" >>> >>> for vm in $vms; do >>> session=3D"${vm##*/}" >>> echo "bhyve session =3D $session" >>> echo "vnc session =3D $vncs" >>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; >>> then >>> printf 'VNC session not found,destroying ghost vms\n' >>> bhyvectl --vm=3D$session --destroy >>> else >>> printf 'Found VNC session %s\n' >>> "${session},no ghost vms found,not destroying them" >>> fi >>> done >>> >>> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (AM13N4CZ)/ && d{print d}'` >>> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >>> >>> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2015040204055E)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >>> >>> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2027285F1175)/ && d{print d}'` >>> echo "CT1000P1SSD8 ; $vmdisk2" >>> >>> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (20130508005976F)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >>> >>> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (BE0191510218)/ && d{print d}'` >>> echo "G-DRIVE USB ; $vmdisk4" >>> >>> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (38234B4237354B45)/ && d{print d}'` >>> echo "Elements 25A3 ; $vmdisk5" >>> >>> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (WD-WCAV2X797309)/ && d{print d}'` >>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >>> >>> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (20140108006C)/ && d{print d}'` >>> echo "Corsair Force 3 SSD ; $vmdisk7" >>> >>> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (1924E50B2AE5)/ && d{print d}'` >>> echo "CT500MX500SSD4 ; $vmdisk8" >>> >>> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (0774921DDC4200A6)/ && d{print d}'` >>> echo "SanDisk Cruzer-15GB ; $vmdisk9" >>> >>> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (60A44D4138D8F311190A0149)/ && d{print d}'` >>> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >>> >>> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (575845483038524844323238)/ && d{print d}'` >>> echo "WD 2500BMV External ; $vmdisk11" >>> >>> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (57442E575845323039544433303334)/ && d{print d}'` >>> echo "WD 3200BMV External ; $vmdisk12" >>> >>> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2414E989076B)/ && d{print d}'` >>> echo "CT500BX500SSD1 ; $vmdisk13" >>> >>> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w = -H -A \ >>> -s 0,hostbridge \ >>> -s 1,ahci-hd,/dev/$vmdisk13 \ >>> -s 10,virtio-net,tap6 \ >>> -s 11,virtio-9p,sharename=3D/ \ >>> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >>> -s 31,lpc \ >>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >>> >>> As you can see,each vm has a unique ID (connected with the vncviewer >>> instance) and the vncviewer window is launched as soon as the bhyve vm = is >>> called...and my Geforce RTX 2080 ti is passed through inside the vm. >>> >>> >>> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >>> wrote: >>> >>>> First of all,you could use a more flexible technique than using the >>>> pptdevs on /boot/loader.conf. I would use a script like this : >>>> >>>> 1) >>>> >>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>>> echo "18/0/0 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:18:0:0 >>>> devctl set driver -f pci0:18:0:0 ppt >>>> else >>>> echo "18/0/0 already attached to ppt" >>>> fi >>>> >>>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>>> >>>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>>> shamim.shahriar@gmail.com> wrote: >>>> >>>>> Good afternoon everyone. >>>>> >>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have che= cked >>>>> the device IDs and put pptdevs in place >>>>> >>>>> # cat /boot/loader.conf >>>>> pptdevs=3D"18/0/0 19/0/0" >>>>> pptdevs2=3D"72/0/0 73/0/0" >>>>> >>>>> this is showing the GPUs as pptdev on the pciconf >>>>> >>>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>>> vendor =3D 'NVIDIA Corporation' >>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>> class =3D display >>>>> subclass =3D 3D >>>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>>> vendor =3D 'NVIDIA Corporation' >>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>> class =3D display >>>>> subclass =3D 3D >>>>> >>>>> as I am using vm-bhyve, I have put the configuration as below >>>>> >>>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>>> loader=3D"uefi" >>>>> cpu=3D16 >>>>> memory=3D128G >>>>> xhci_mouse=3D"yes" >>>>> debug=3D"true" >>>>> >>>>> graphics=3D"yes" >>>>> graphics_listen=3D"127.0.0.1" >>>>> graphics_port=3D"5920" >>>>> graphics_res=3D"1024x768" >>>>> graphics_wait=3D"no" >>>>> #graphics_vga=3D"io" >>>>> >>>>> network0_type=3D"virtio-net" >>>>> network0_switch=3D"swUNI" >>>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>>> >>>>> disk0_type=3D"nvme" >>>>> disk0_name=3D"jagadish-disk0.img" >>>>> #disk0_size=3D"128G" >>>>> >>>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>>> >>>>> passthru0=3D"19/0/0=3D6:0" >>>>> >>>>> bhyve_options=3D"-A -H -P" >>>>> #END >>>>> >>>>> however, when I start the vm (for OS installation to start with), it >>>>> shows it is running but I am unable to access the VNC for a while, th= en >>>>> when I finally manage to connect via VNC, there is nothing on the scr= een, >>>>> just blank dark screen >>>>> >>>>> the vm-bhyve.log shows >>>>> >>>>> Mar 24 13:59:26: initialising >>>>> Mar 24 13:59:26: [loader: uefi] >>>>> Mar 24 13:59:26: [cpu: 16] >>>>> Mar 24 13:59:26: [memory: 128G] >>>>> Mar 24 13:59:26: [hostbridge: standard] >>>>> Mar 24 13:59:26: [com ports: com1] >>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>>> Mar 24 13:59:26: [debug mode: true] >>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>>> Mar 24 13:59:26: [primary disk dev: file] >>>>> Mar 24 13:59:26: initialising network device tap0 >>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: booting >>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>>> Mar 24 13:59:26: [bhyve iso device: -s >>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>>> Mar 24 13:59:26: starting bhyve (run 1) >>>>> >>>>> based on what I can see, and a little out of desparation, I decided t= o >>>>> run the installer in the foreground, and below is what I got >>>>> >>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>>> >>>>> it stays there for as long as I wait until I poweroff the vm. >>>>> >>>>> tried with Debian installer, >>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>>> >>>>> >>>>> ideally I need to install Debian for my user base on these GPU based >>>>> systems. But even that is proving to be impossible since the installe= r does >>>>> not move any further from where it is. >>>>> >>>>> Any thoughts/ideas/suggestions what I else I can try to make this >>>>> work? Anything that you have tried that worked? have I missed somethi= ng? >>>>> >>>>> Would appreciate any and all thoughts/suggestions >>>>> >>>>> best regards >>>>> SS >>>>> >>>> >>>> >>>> -- >>>> Mario. >>>> >>> >>> >>> -- >>> Mario. >>> >> >> >> -- >> Mario. >> > --=20 Mario. --000000000000facef9063118686b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Shamim.

You don't = have any bhyve-win. You can use bhyve. I use bhyve-win to pass through my G= PU in a Windows vm,bhyve-lin for a Linux vm. They are two different executa= bles,because for some reason,my bhyve-lin executable is not able to pass my= GPU in a Windows vm.=C2=A0

On Mon, Mar 24, 2025= at 4:29=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Hi Mario

=
I will try the options you have used later and let you know how it goe= s. Not sure if this is intentional or typo, but I don't recall any bhyv= e-win binary in my system.

Best regards
= SS

On Mon, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wr= ote:
Oh sorry. I forgot to add the following bhyve parameters necessa= ry to passthru my GPU :D

-s 8:0,passthru,2/0/0 \-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2= /0/3 \

On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
Usually I use this kind of script to launch a bhyve vm :=

#!/bin/sh

setxkbmap it
kldload vmm.ko
vms= =3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncview= er [0]/{print $6}')"
kldload vmm.ko

if ! pciconf -l pci0= :2:0:0 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/0= is not attached to ppt,attaching..."
kldload nvidia-modeset
dev= ctl clear driver -f pci0:2:0:0
devctl set driver -f pci0:2:0:0 ppt
el= se
echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
= fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
ec= ho "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."devctl clear driver -f pci0:2:0:1
devctl set driver -f pci0:2:0:1 pptelse
echo "rtx 2080ti slot 2/0/1 is already attached to ppt"=
fi

if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; thenecho "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."= ;
devctl clear driver -f pci0:2:0:2
devctl set driver -f pci0:2:0:2 p= pt
else
echo "rtx 2080ti slot 2/0/2 is already attached to ppt&q= uot;
fi

if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; th= en
echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching...&= quot;
devctl clear driver -f pci0:2:0:3
devctl set driver -f pci0:2:0= :3 ppt
else
echo "rtx 2080ti slot 2/0/3 is already attached to p= pt"
fi

echo "rtx 2080ti is fully attached to ppt"<= br>
for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve session =3D $session&qu= ot;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "v= nc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 if ! printf '%s\n' "${vncs}" | grep "${se= ssion#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf '= VNC session not found,destroying ghost vms\n'
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 printf 'Found VNC session %s\n' "${session},no ghos= t vms found,not destroying them"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geom disk list | awk = 9;/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ && d{print d}'= `
echo "Seagate M3 Portable 1.8T ; $vmdisk0"

vmdisk1=3D= `geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (201504020405= 5E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 1.8= T ; $vmdisk1"

vmdisk2=3D`geom disk list | awk '/^Geom name:= /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{print d}'`
echo = "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`geom disk list | awk = '/^Geom name: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{pri= nt d}'`
echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"<= br>
vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ide= nt: (BE0191510218)/ && d{print d}'`
echo "G-DRIVE USB ;= $vmdisk4"

vmdisk5=3D`geom disk list | awk '/^Geom name: /{= d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'`
echo= "Elements 25A3 ; $vmdisk5"

vmdisk6=3D`geom disk list | aw= k '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && d{p= rint d}'`
echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"=

vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *id= ent: (20140108006C)/ && d{print d}'`
echo "Corsair Forc= e 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk list | awk '/^Geom= name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'`echo "CT500MX500SSD4 ; $vmdisk8"

vmdisk9=3D`geom disk lis= t | awk '/^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ &&a= mp; d{print d}'`
echo "SanDisk Cruzer-15GB ; $vmdisk9"
=
vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident= : (60A44D4138D8F311190A0149)/ && d{print d}'`
echo "Kin= gston DataTraveler 2.0 ; $vmdisk10"

vmdisk11=3D`geom disk list = | awk '/^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)/ &= amp;& d{print d}'`
echo "WD 2500BMV External ; $vmdisk11&qu= ot;

vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^= *ident: (57442E575845323039544433303334)/ && d{print d}'`
e= cho "WD 3200BMV External ; $vmdisk12"

vmdisk13=3D`geom dis= k list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ &&= amp; d{print d}'`
echo "CT500BX500SSD1 ; $vmdisk13"
/usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 = -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/$vmdisk13 \
-= s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp= =3D0.0.0.0:5906,w=3D1= 600,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/B= HYVE_BHF_CODE.fd \
vm0:6 < /dev/null & sleep 5 && vncview= er 0:6 &

As you can see,each vm has a unique I= D (connected with the vncviewer instance) and the vncviewer window is launc= hed as soon as the bhyve vm is called...and my Geforce RTX 2080 ti is passe= d through inside the vm.


On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM M= ario Marietto <marietto2008@gmail.com> wrote:
First of all,you could use a= more flexible technique than using the pptdevs on /boot/loader.conf. I wou= ld use a script like this :

1)

if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
ech= o "18/0/0 is not attached to ppt,attaching..."
devctl clear dr= iver -f pci0:18:0:0
devctl set driver -f pci0:18:0:0 ppt
else
echo= "18/0/0 already attached to ppt"
fi

2)= I would not like to use vm-bhyve. It adds only some confusion...

On M= on, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com&g= t; wrote:
Good afternoon everyone.

I am trying to have VMs with GPU= passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/8000 i= nstalled already. I have checked the device IDs and put pptdevs in place
# cat /boot/loader.conf
pptdevs=3D"18/0/0 19/0/0"
pptd= evs2=3D"72/0/0 73/0/0"

this is showing the GPUs as pptdev = on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x0302= 00 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de= subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA C= orporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Qua= dro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D disp= lay
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2= =A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D= 0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0= =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
as I am using vm-bhyve, I have put the configuration as below

# cat= /mnt/VMs/jagadish/jagadish.conf
loader=3D"uefi"
cpu=3D16memory=3D128G
xhci_mouse=3D"yes"
debug=3D"true"=

graphics=3D"yes"
graphics_listen=3D"127.0.0.1&quo= t;
graphics_port=3D"5920"
graphics_res=3D"1024x768&quo= t;
graphics_wait=3D"no"
#graphics_vga=3D"io"
<= br>network0_type=3D"virtio-net"
network0_switch=3D"swUNI&= quot;
network0_mac=3D"58:9c:fc:06:3f:af"

disk0_type=3D&= quot;nvme"
disk0_name=3D"jagadish-disk0.img"
#disk0_si= ze=3D"128G"

uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe3= 4de"

passthru0=3D"19/0/0=3D6:0"

bhyve_options= =3D"-A -H -P"
#END

however, when I start the vm (for OS= installation to start with), it shows it is running but I am unable to acc= ess the VNC for a while, then when I finally manage to connect via VNC, the= re is nothing on the screen, just blank dark screen

the vm-bhyve.log= shows

Mar 24 13:59:26: initialising
Mar 24 13:59:26: =C2=A0[load= er: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[me= mory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridge: standard]
Mar 24 13:5= 9:26: =C2=A0[com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-129= 3-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode: true]
M= ar 24 13:59:26: =C2=A0[primary disk: jagadish-disk0.img]
Mar 24 13:59:26= : =C2=A0[primary disk dev: file]
Mar 24 13:59:26: initialising network d= evice tap0
Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
M= ar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:2= 6: booting
Mar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l= bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-= 1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices= : -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.im= g -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s = 7:0,fbuf,tcp=3D127.0.0.= 1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[= bhyve console: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bh= yve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.= iso,ro]
Mar 24 13:59:26: starting bhyve (run 1)

based on what I c= an see, and a little out of desparation, I decided to run the installer in = the foreground, and below is what I got

# vm install -f jagadish Fre= eBSD-14.2-RELEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found gu= est in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: = 0x112245400000 [sz 16777216]

it stays there for as long as I wait un= til I poweroff the vm.

tried with Debian installer,
# vm install= -f jagadish debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0= * found guest in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame bu= ffer base: 0x2747e2400000 [sz 16777216]


ideally I need to instal= l Debian for my user base on these GPU based systems. But even that is prov= ing to be impossible since the installer does not move any further from whe= re it is.

Any thoughts/ideas/suggestions what I else I can try to ma= ke this work? Anything that you have tried that worked? have I missed somet= hing?

Would appreciate any and all thoughts/suggestions

best = regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000facef9063118686b-- From nobody Mon Mar 24 15:48:35 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLyCc1cXHz5s4RZ for ; Mon, 24 Mar 2025 15:49:16 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLyCb1pvJz3q5L for ; Mon, 24 Mar 2025 15:49:15 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=TzNVRRj0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::102f as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-2ff799d99dcso7571170a91.1 for ; Mon, 24 Mar 2025 08:49:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742831353; x=1743436153; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=UQxtYh2Btv7uI0Q0CCjPEiMu/aYXx1wCdyierlZ5PMg=; b=TzNVRRj0bK1/s1aEV3IEZKdosFI/h+9fQY1T427S6j6BwQk26f+BpKfTb6CvnnlcDA 6wY3x3R/Rz45eRalYai6UtJjok2tkhzAiPMDso1F7af7YBh/djROwvusxkx5RISjj5sW r8DZZcvwRs0i3a6nZhUo6/u8ECC0QBKVFgBW6F/1UJuBxmcptzPbKj65Q8V/M8HztWLj 5sn/zzMq9ih5D6JyiER68Q2D1Rl8tLeayPPVmcr9v9sSoNIhOD8ixilqS82JLsj42bKV oT8/T6RQbH6VgDS51alKJ3P7qy8jPn5jTZnrCPUdWbQ9tiAr+5xyOJZugipHKRERdTmo /EuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742831353; x=1743436153; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UQxtYh2Btv7uI0Q0CCjPEiMu/aYXx1wCdyierlZ5PMg=; b=k8GURFMdyqlSY9QTdLFBqsXSNTspC6/F0b1jbFJwKFu9XazE14KMTyxz7ZhvSWxNJR u9GYzmYnzulxwS1nm2WEzsVq/xDmchc1xtsW1ZmnLXSxwM9XN4tCtqO7b9sv8JdvIMJ4 u6UYkg/hKX1sVw4feJSEMzXVkwfsygZksAcmIwyoWDLQgtwrPPkSHlSDb69Tamb2T69V i1a49F3j3iZjU43+LPVeH5im4QjmB+3KqpKabnDvkmZHaW+cp4gSQTM/cMSo8+2qXKe2 mdYU5uL5OFrrIArQ8iam/DCZlAn6PJr0GM7E+YYOpoMv+Iqzw1Q9SgwZLsYzJ2wie8pS Ps5A== X-Gm-Message-State: AOJu0YyD3Yz83/bTVQ4ar3ghgid6/rASeVUi9Cxu6uOXEOVoakJhbYZF RJ9uObfyQmk7y1HpKp6EdP27BLS6islKuCVnOtwj3g7XaBBnauzsrgq/jNlmt7OET3Bq1liTZDf pUh0Q+hZKNSjhEBjmnEty7bjvWrLwTjb3 X-Gm-Gg: ASbGncvW7Afo1nvw7wH+BJIUKT24vpGOvFR38VSRB4KgyMdFxouEbMezXOgTSBUMIYQ zlIYsDTZGKZkdAmOnBJJKbhm3fb8FSal6aC1Idgk98flWeNvi86xyJtTlphcji+t9wULKaFRYzT +cwo7YQstp+IPag8hYt4xze03lnJ4= X-Google-Smtp-Source: AGHT+IHRKMNSrYRqmMXa7nuV7zCxdnwkJHYIjGx/RaMu0I0JcfxWOyxR0xdnYC5A2YHHGKzo/TxiRSQDFZ9yhSBL7T0= X-Received: by 2002:a17:90a:d008:b0:2ff:693a:7590 with SMTP id 98e67ed59e1d1-3030ff08de9mr26351330a91.33.1742831353190; Mon, 24 Mar 2025 08:49:13 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 16:48:35 +0100 X-Gm-Features: AQ5f1Jp4wslYlF0ooDdNFX4E0rqtDEM0PIxsiR-u9Wqwr7xagDmZQ1CCMTRmZg8 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000b5df190631188bc5" X-Spamd-Result: default: False [2.99 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; NEURAL_SPAM_SHORT(0.99)[0.993]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102f:from]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLyCb1pvJz3q5L X-Spamd-Bar: ++ --000000000000b5df190631188bc5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ohhh man. I forgot to say that you need to apply the Corvin's patches if you want to pass your gpu to a LINUX vm. And you should apply another patch if you want to pass it to a Windows VM. It seems that the necessary patches have never been uploaded mainstream. On Mon, Mar 24, 2025 at 4:38=E2=80=AFPM Mario Marietto wrote: > Hello Shamim. > > You don't have any bhyve-win. You can use bhyve. I use bhyve-win to pass > through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are two > different executables,because for some reason,my bhyve-lin executable is > not able to pass my GPU in a Windows vm. > > On Mon, Mar 24, 2025 at 4:29=E2=80=AFPM Shamim Shahriar > wrote: > >> Hi Mario >> >> I will try the options you have used later and let you know how it goes. >> Not sure if this is intentional or typo, but I don't recall any bhyve-wi= n >> binary in my system. >> >> Best regards >> SS >> >> On Mon, 24 Mar 2025 at 14:59, Mario Marietto >> wrote: >> >>> Oh sorry. I forgot to add the following bhyve parameters necessary to >>> passthru my GPU :D >>> >>> -s 8:0,passthru,2/0/0 \ >>> -s 8:1,passthru,2/0/1 \ >>> -s 8:2,passthru,2/0/2 \ >>> -s 8:3,passthru,2/0/3 \ >>> >>> On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto >>> wrote: >>> >>>> Usually I use this kind of script to launch a bhyve vm : >>>> >>>> #!/bin/sh >>>> >>>> setxkbmap it >>>> kldload vmm.ko >>>> vms=3D"$(ls /dev/vmm/*)" >>>> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >>>> kldload vmm.ko >>>> >>>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >>>> kldload nvidia-modeset >>>> devctl clear driver -f pci0:2:0:0 >>>> devctl set driver -f pci0:2:0:0 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >>>> fi >>>> >>>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:2:0:1 >>>> devctl set driver -f pci0:2:0:1 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >>>> fi >>>> >>>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:2:0:2 >>>> devctl set driver -f pci0:2:0:2 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >>>> fi >>>> >>>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:2:0:3 >>>> devctl set driver -f pci0:2:0:3 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >>>> fi >>>> >>>> echo "rtx 2080ti is fully attached to ppt" >>>> >>>> for vm in $vms; do >>>> session=3D"${vm##*/}" >>>> echo "bhyve session =3D $session" >>>> echo "vnc session =3D $vncs" >>>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; >>>> then >>>> printf 'VNC session not found,destroying ghost vms\n' >>>> bhyvectl --vm=3D$session --destroy >>>> else >>>> printf 'Found VNC session %s\n' >>>> "${session},no ghost vms found,not destroying them" >>>> fi >>>> done >>>> >>>> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (AM13N4CZ)/ && d{print d}'` >>>> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >>>> >>>> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (2015040204055E)/ && d{print d}'` >>>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >>>> >>>> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (2027285F1175)/ && d{print d}'` >>>> echo "CT1000P1SSD8 ; $vmdisk2" >>>> >>>> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (20130508005976F)/ && d{print d}'` >>>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >>>> >>>> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (BE0191510218)/ && d{print d}'` >>>> echo "G-DRIVE USB ; $vmdisk4" >>>> >>>> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (38234B4237354B45)/ && d{print d}'` >>>> echo "Elements 25A3 ; $vmdisk5" >>>> >>>> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (WD-WCAV2X797309)/ && d{print d}'` >>>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >>>> >>>> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (20140108006C)/ && d{print d}'` >>>> echo "Corsair Force 3 SSD ; $vmdisk7" >>>> >>>> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (1924E50B2AE5)/ && d{print d}'` >>>> echo "CT500MX500SSD4 ; $vmdisk8" >>>> >>>> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (0774921DDC4200A6)/ && d{print d}'` >>>> echo "SanDisk Cruzer-15GB ; $vmdisk9" >>>> >>>> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (60A44D4138D8F311190A0149)/ && d{print d}'` >>>> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >>>> >>>> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (575845483038524844323238)/ && d{print d}'` >>>> echo "WD 2500BMV External ; $vmdisk11" >>>> >>>> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (57442E575845323039544433303334)/ && d{print d}'` >>>> echo "WD 3200BMV External ; $vmdisk12" >>>> >>>> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (2414E989076B)/ && d{print d}'` >>>> echo "CT500BX500SSD1 ; $vmdisk13" >>>> >>>> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w= -H -A \ >>>> -s 0,hostbridge \ >>>> -s 1,ahci-hd,/dev/$vmdisk13 \ >>>> -s 10,virtio-net,tap6 \ >>>> -s 11,virtio-9p,sharename=3D/ \ >>>> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >>>> -s 31,lpc \ >>>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >>>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >>>> >>>> As you can see,each vm has a unique ID (connected with the vncviewer >>>> instance) and the vncviewer window is launched as soon as the bhyve vm= is >>>> called...and my Geforce RTX 2080 ti is passed through inside the vm. >>>> >>>> >>>> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >>>> wrote: >>>> >>>>> First of all,you could use a more flexible technique than using the >>>>> pptdevs on /boot/loader.conf. I would use a script like this : >>>>> >>>>> 1) >>>>> >>>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>>>> echo "18/0/0 is not attached to ppt,attaching..." >>>>> devctl clear driver -f pci0:18:0:0 >>>>> devctl set driver -f pci0:18:0:0 ppt >>>>> else >>>>> echo "18/0/0 already attached to ppt" >>>>> fi >>>>> >>>>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>>>> >>>>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>>>> shamim.shahriar@gmail.com> wrote: >>>>> >>>>>> Good afternoon everyone. >>>>>> >>>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have ch= ecked >>>>>> the device IDs and put pptdevs in place >>>>>> >>>>>> # cat /boot/loader.conf >>>>>> pptdevs=3D"18/0/0 19/0/0" >>>>>> pptdevs2=3D"72/0/0 73/0/0" >>>>>> >>>>>> this is showing the GPUs as pptdev on the pciconf >>>>>> >>>>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d= 8 >>>>>> vendor =3D 'NVIDIA Corporation' >>>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>>> class =3D display >>>>>> subclass =3D 3D >>>>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d= 8 >>>>>> vendor =3D 'NVIDIA Corporation' >>>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>>> class =3D display >>>>>> subclass =3D 3D >>>>>> >>>>>> as I am using vm-bhyve, I have put the configuration as below >>>>>> >>>>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>>>> loader=3D"uefi" >>>>>> cpu=3D16 >>>>>> memory=3D128G >>>>>> xhci_mouse=3D"yes" >>>>>> debug=3D"true" >>>>>> >>>>>> graphics=3D"yes" >>>>>> graphics_listen=3D"127.0.0.1" >>>>>> graphics_port=3D"5920" >>>>>> graphics_res=3D"1024x768" >>>>>> graphics_wait=3D"no" >>>>>> #graphics_vga=3D"io" >>>>>> >>>>>> network0_type=3D"virtio-net" >>>>>> network0_switch=3D"swUNI" >>>>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>>>> >>>>>> disk0_type=3D"nvme" >>>>>> disk0_name=3D"jagadish-disk0.img" >>>>>> #disk0_size=3D"128G" >>>>>> >>>>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>>>> >>>>>> passthru0=3D"19/0/0=3D6:0" >>>>>> >>>>>> bhyve_options=3D"-A -H -P" >>>>>> #END >>>>>> >>>>>> however, when I start the vm (for OS installation to start with), it >>>>>> shows it is running but I am unable to access the VNC for a while, t= hen >>>>>> when I finally manage to connect via VNC, there is nothing on the sc= reen, >>>>>> just blank dark screen >>>>>> >>>>>> the vm-bhyve.log shows >>>>>> >>>>>> Mar 24 13:59:26: initialising >>>>>> Mar 24 13:59:26: [loader: uefi] >>>>>> Mar 24 13:59:26: [cpu: 16] >>>>>> Mar 24 13:59:26: [memory: 128G] >>>>>> Mar 24 13:59:26: [hostbridge: standard] >>>>>> Mar 24 13:59:26: [com ports: com1] >>>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>>>> Mar 24 13:59:26: [debug mode: true] >>>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>>>> Mar 24 13:59:26: [primary disk dev: file] >>>>>> Mar 24 13:59:26: initialising network device tap0 >>>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>>>> Mar 24 13:59:26: booting >>>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -= s >>>>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>>>> Mar 24 13:59:26: [bhyve iso device: -s >>>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>>>> Mar 24 13:59:26: starting bhyve (run 1) >>>>>> >>>>>> based on what I can see, and a little out of desparation, I decided >>>>>> to run the installer in the foreground, and below is what I got >>>>>> >>>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>>>> Starting jagadish >>>>>> * found guest in /mnt/VMs/jagadish >>>>>> * booting... >>>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>>>> >>>>>> it stays there for as long as I wait until I poweroff the vm. >>>>>> >>>>>> tried with Debian installer, >>>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>>>> Starting jagadish >>>>>> * found guest in /mnt/VMs/jagadish >>>>>> * booting... >>>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>>>> >>>>>> >>>>>> ideally I need to install Debian for my user base on these GPU based >>>>>> systems. But even that is proving to be impossible since the install= er does >>>>>> not move any further from where it is. >>>>>> >>>>>> Any thoughts/ideas/suggestions what I else I can try to make this >>>>>> work? Anything that you have tried that worked? have I missed someth= ing? >>>>>> >>>>>> Would appreciate any and all thoughts/suggestions >>>>>> >>>>>> best regards >>>>>> SS >>>>>> >>>>> >>>>> >>>>> -- >>>>> Mario. >>>>> >>>> >>>> >>>> -- >>>> Mario. >>>> >>> >>> >>> -- >>> Mario. >>> >> > > -- > Mario. > --=20 Mario. --000000000000b5df190631188bc5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ohhh man.

I forgot to s= ay that you need to apply the Corvin's patches if you want to pass your= gpu to a LINUX vm. And you should apply another patch if you want to pass = it to a Windows VM. It seems that the necessary patches have never been upl= oaded mainstream.

On Mon, Mar 24, 2025 at 4:38=E2=80=AFPM Mario= Marietto <m= arietto2008@gmail.com> wrote:
Hello Shamim.

You don't have any bhyve-win. You can use bhyve. I use bhyve-win= to pass through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are = two different executables,because for some reason,my bhyve-lin executable i= s not able to pass my GPU in a Windows vm.=C2=A0

On Mon, Mar 24, 2025= at 4:29=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Hi Ma= rio

I will try the options you have used later and let y= ou know how it goes. Not sure if this is intentional or typo, but I don'= ;t recall any bhyve-win binary in my system.

Best = regards
SS

On Mon, 24 Mar 2025 at 14:59, Mario Marietto <= marietto2008@gm= ail.com> wrote:
Oh sorry. I forgot to add the following bhyve = parameters necessary to passthru my GPU :D

-s 8:0,= passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \-s 8:3,passthru,2/0/3 \

On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marie= tto <mariett= o2008@gmail.com> wrote:
Usually I use this kind of script to l= aunch a bhyve vm :

#!/bin/sh

setxkbmap it
kld= load vmm.ko
vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax |= awk '/vncviewer [0]/{print $6}')"
kldload vmm.ko

if= ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
echo "rtx= 2080ti slot 2/0/0 is not attached to ppt,attaching..."
kldload nvi= dia-modeset
devctl clear driver -f pci0:2:0:0
devctl set driver -f pc= i0:2:0:0 ppt
else
echo "rtx 2080ti slot 2/0/0 is already attache= d to ppt"
fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt= "; then
echo "rtx 2080ti slot 2/0/1 is not attached to ppt,att= aching..."
devctl clear driver -f pci0:2:0:1
devctl set driver -= f pci0:2:0:1 ppt
else
echo "rtx 2080ti slot 2/0/1 is already att= ached to ppt"
fi

if ! pciconf -l pci0:2:0:2 | grep -q "= ^ppt"; then
echo "rtx 2080ti slot 2/0/2 is not attached to ppt= ,attaching..."
devctl clear driver -f pci0:2:0:2
devctl set driv= er -f pci0:2:0:2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already= attached to ppt"
fi

if ! pciconf -l pci0:2:0:3 | grep -q &q= uot;^ppt"; then
echo "rtx 2080ti slot 2/0/3 is not attached to= ppt,attaching..."
devctl clear driver -f pci0:2:0:3
devctl set = driver -f pci0:2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is alr= eady attached to ppt"
fi

echo "rtx 2080ti is fully atta= ched to ppt"

for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve sessi= on =3D $session"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "vnc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}&qu= ot; | grep "${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 printf 'VNC session not found,destroying ghost vms\n'=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' &qu= ot;${session},no ghost vms found,not destroying them"
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geo= m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ &= & d{print d}'`
echo "Seagate M3 Portable 1.8T ; $vmdisk0&qu= ot;

vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (2015040204055E)/ && d{print d}'`
echo "TOSHIBA= External USB 3.0 1.8T ; $vmdisk1"

vmdisk2=3D`geom disk list | = awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{pr= int d}'`
echo "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`= geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2013050800597= 6F)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 932= GB ; $vmdisk3"

vmdisk4=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'`
ech= o "G-DRIVE USB ; $vmdisk4"

vmdisk5=3D`geom disk list | awk= '/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{p= rint d}'`
echo "Elements 25A3 ; $vmdisk5"

vmdisk6= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X= 797309)/ && d{print d}'`
echo "WDC WD3200AAJS-00L7A0 ; = 298 GB ; $vmdisk6"

vmdisk7=3D`geom disk list | awk '/^Geom = name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'`
= echo "Corsair Force 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk= list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ &&a= mp; d{print d}'`
echo "CT500MX500SSD4 ; $vmdisk8"

v= mdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (077= 4921DDC4200A6)/ && d{print d}'`
echo "SanDisk Cruzer-15= GB ; $vmdisk9"

vmdisk10=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}&= #39;`
echo "Kingston DataTraveler 2.0 ; $vmdisk10"

vmdi= sk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57584= 5483038524844323238)/ && d{print d}'`
echo "WD 2500BMV = External ; $vmdisk11"

vmdisk12=3D`geom disk list | awk '/^G= eom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ &&= ; d{print d}'`
echo "WD 3200BMV External ; $vmdisk12"
<= br>vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:= (2414E989076B)/ && d{print d}'`
echo "CT500BX500SSD1 ;= $vmdisk13"

/usr/sbin/./bhyve-win -S -c sockets=3D4= ,cores=3D1,threads=3D1 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-h= d,/dev/$vmdisk13 \
-s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename= =3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/loc= al/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:6 < /dev/null & sl= eep 5 && vncviewer 0:6 &

As you can se= e,each vm has a unique ID (connected with the vncviewer instance) and the v= ncviewer window is launched as soon as the bhyve vm is called...and my Gefo= rce RTX 2080 ti is passed through inside the vm.


On Mon, Mar 24, 2= 025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">
Firs= t of all,you could use a more flexible technique than using the pptdevs on = /boot/loader.conf. I would use a script like this :

1)

if ! pciconf -l pci0:18:0:0 | grep -q "^= ppt"; then
echo "18/0/0 is not attached to ppt,attaching...&= quot;
devctl clear driver -f pci0:18:0:0
devctl set driver -f pci0:18= :0:0 ppt
else
echo "18/0/0 already attached to ppt"
fi

2) I would not like to use vm-bhyve. It adds only som= e confusion...

On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar= <shamim.= shahriar@gmail.com> wrote:
Good afternoon everyone.

I am try= ing to have VMs with GPU passthrough. The setup is a Dell server with NVidi= a Quadro RTX 6000/8000 installed already. I have checked the device IDs and= put pptdevs in place

# cat /boot/loader.conf
pptdevs=3D"18/= 0/0 19/0/0"
pptdevs2=3D"72/0/0 73/0/0"

this is sho= wing the GPUs as pptdev on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2= =A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D= 0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0= =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt= 1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
= =C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2= =A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'= ;
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 s= ubclass =C2=A0 =3D 3D

as I am using vm-bhyve, I have put the configu= ration as below

# cat /mnt/VMs/jagadish/jagadish.conf
loader=3D&q= uot;uefi"
cpu=3D16
memory=3D128G
xhci_mouse=3D"yes"=
debug=3D"true"

graphics=3D"yes"
graphics_= listen=3D"127.0.0.1"
graphics_port=3D"5920"
graph= ics_res=3D"1024x768"
graphics_wait=3D"no"
#graphi= cs_vga=3D"io"

network0_type=3D"virtio-net"
ne= twork0_switch=3D"swUNI"
network0_mac=3D"58:9c:fc:06:3f:af= "

disk0_type=3D"nvme"
disk0_name=3D"jagadish-= disk0.img"
#disk0_size=3D"128G"

uuid=3D"966e9= 09b-1293-11ef-a9a4-e4434bfe34de"

passthru0=3D"19/0/0=3D6:0= "

bhyve_options=3D"-A -H -P"
#END

however, = when I start the vm (for OS installation to start with), it shows it is run= ning but I am unable to access the VNC for a while, then when I finally man= age to connect via VNC, there is nothing on the screen, just blank dark scr= een

the vm-bhyve.log shows

Mar 24 13:59:26: initialising
M= ar 24 13:59:26: =C2=A0[loader: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]Mar 24 13:59:26: =C2=A0[memory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridg= e: standard]
Mar 24 13:59:26: =C2=A0[com ports: com1]
Mar 24 13:59:26= : =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: = =C2=A0[debug mode: true]
Mar 24 13:59:26: =C2=A0[primary disk: jagadish-= disk0.img]
Mar 24 13:59:26: =C2=A0[primary disk dev: file]
Mar 24 13:= 59:26: initialising network device tap0
Mar 24 13:59:26: adding tap0 -&g= t; vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (= swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:26: =C2=A0[bhyve op= tions: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_= UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 1= 3:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/V= Ms/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:= af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet= ]
Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev/nmdm-jagadish.1A]<= br>Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/l= ubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26: starting bhyve (ru= n 1)

based on what I can see, and a little out of desparation, I dec= ided to run the installer in the foreground, and below is what I got
# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
Starting = jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 * booting..= .
fbuf frame buffer base: 0x112245400000 [sz 16777216]

it stays t= here for as long as I wait until I poweroff the vm.

tried with Debia= n installer,
# vm install -f jagadish debian-12.5.0-amd64-netinst.isoStarting jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 = * booting...
fbuf frame buffer base: 0x2747e2400000 [sz 16777216]

ideally I need to install Debian for my user base on these GPU based s= ystems. But even that is proving to be impossible since the installer does = not move any further from where it is.

Any thoughts/ideas/suggestion= s what I else I can try to make this work? Anything that you have tried tha= t worked? have I missed something?

Would appreciate any and all thou= ghts/suggestions

best regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000b5df190631188bc5-- From nobody Mon Mar 24 17:34:58 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZM0Ys0XqWz5rBy4 for ; Mon, 24 Mar 2025 17:35:13 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x1133.google.com (mail-yw1-x1133.google.com [IPv6:2607:f8b0:4864:20::1133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZM0Yq2Yk4z3KmD for ; Mon, 24 Mar 2025 17:35:11 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=GxPpjdGB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shamim.shahriar@gmail.com designates 2607:f8b0:4864:20::1133 as permitted sender) smtp.mailfrom=shamim.shahriar@gmail.com Received: by mail-yw1-x1133.google.com with SMTP id 00721157ae682-6fedf6aaed2so39060447b3.0 for ; Mon, 24 Mar 2025 10:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742837710; x=1743442510; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kB9qwAfFa1F4NP5c3sUFIS5IH+C4IqcENg8kydR6NVo=; b=GxPpjdGBgVTtg5f2kTLtdfckpVcAdNktV80C4sv38NiijKuq/Wgk9rdFxLB81hsmqb Nvt3tAguYlonYKXZz1MYNozi9r8PdnDLAgCpbyxiHl7oOJPdCcD7xQl3J0GWTOdnfY5w IBslrE8WP02l0r9NT4dnPlF20vDNkHpfIV1o4F6W5cH7EuQ4k0/N0qf99ltr9Wd0+4/W 41rgHByhl3dtgDlKR95wEhrs1h/zxtvLw2ZxctkZZkjrpIE9uiJMOMvj9efdwXWXLB9P TTxYjNjR5AIwXATyLtIv0+L4RsoWMy/N5ozBZQOWuXexOx01hxpoCEqZpaYT72lv619a 8yVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742837710; x=1743442510; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kB9qwAfFa1F4NP5c3sUFIS5IH+C4IqcENg8kydR6NVo=; b=c1sFAUJ6q2myhyhxd7ggEM9x3cjfbtK/WIGFF5LjTZPCdEfMmi6yT61Woo9tmjAaK/ QiQdUCOxzHUKKgVe4denjhgmDa2d5Uxo67eBCiJxObICiA6HpJ9Pyghu3BBGSMzdu8kD LD/frP2ghzVe+ZVtUNMPXSVLvzkYFAtyRDymv1ZLW+uza+jYrAFHY4jJgYXOOMLzqwiz m3GhtbxNDnK/4sSx5i4icJ4LO8uerVBLErNyHxNgBK200liTdgEVOfzdpIDTLjsMe12k Khr+6AkYQl+I1Pkjbnx8XOtr3nIHiHXfEaMIKJm1i/cn0emE6il6egnGjqdg4gYYsILn niaA== X-Gm-Message-State: AOJu0YzdhaPFocrfiZixudFDcyGIWvHq02HlbxFd1CXlIJEHqtHZopkL B5jQI+X1fYgBKUTwVJg3nmTvHF6mx4/TqUm3UNZDn+T4Uya094lRbCh8Qjcxa9LrqtIo122Cgfb EN+QAukyNUkFv7GJB+kcQIlhQoQ== X-Gm-Gg: ASbGncv58UzFWEhbkGYX5AZ8GVIrhSXq7By9U5eth2FBiox83cRkzACGtTjaRr60CV/ PrvrhfOzPumhhg5/+m/4XPoRwV7o7+Dy9sR1YcfzuvCOJhzU48+yBgW+Mnj4o2zzpIwnoiDaCQj 7fBndzrKiTBtuYU8LeatVSNmgx X-Google-Smtp-Source: AGHT+IFdQ/MB6wc84K0a+RXIQnXkJ/LOjYSFBpxNHd1m38JbzWmiRTk+JJW73ZHigV6Ak5L9+Ze3aCEoT1K9wO4MVbQ= X-Received: by 2002:a0d:d2c7:0:b0:6f9:c8d9:50f with SMTP id 00721157ae682-700ab22d20bmr195867137b3.2.1742837710269; Mon, 24 Mar 2025 10:35:10 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Shamim Shahriar Date: Mon, 24 Mar 2025 17:34:58 +0000 X-Gm-Features: AQ5f1Jo2YsRxsTO-fvXVXzqMyAb9akUfv12ZcrBAD4nN0ECVguvbw-rf6CtCK_4 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Mario Marietto Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="0000000000009f3bc906311a0606" X-Spamd-Result: default: False [2.83 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; NEURAL_SPAM_LONG(0.95)[0.951]; NEURAL_SPAM_SHORT(0.88)[0.883]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; TAGGED_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1133:from]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZM0Yq2Yk4z3KmD X-Spamd-Bar: ++ --0000000000009f3bc906311a0606 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mario, tried options similar to what you suggested, no difference to the outcome. Stuck where it was before. Best regards SS On Mon, 24 Mar 2025 at 15:29, Shamim Shahriar wrote: > Hi Mario > > I will try the options you have used later and let you know how it goes. > Not sure if this is intentional or typo, but I don't recall any bhyve-win > binary in my system. > > Best regards > SS > > On Mon, 24 Mar 2025 at 14:59, Mario Marietto > wrote: > >> Oh sorry. I forgot to add the following bhyve parameters necessary to >> passthru my GPU :D >> >> -s 8:0,passthru,2/0/0 \ >> -s 8:1,passthru,2/0/1 \ >> -s 8:2,passthru,2/0/2 \ >> -s 8:3,passthru,2/0/3 \ >> >> On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto >> wrote: >> >>> Usually I use this kind of script to launch a bhyve vm : >>> >>> #!/bin/sh >>> >>> setxkbmap it >>> kldload vmm.ko >>> vms=3D"$(ls /dev/vmm/*)" >>> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >>> kldload vmm.ko >>> >>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >>> kldload nvidia-modeset >>> devctl clear driver -f pci0:2:0:0 >>> devctl set driver -f pci0:2:0:0 ppt >>> else >>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:1 >>> devctl set driver -f pci0:2:0:1 ppt >>> else >>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:2 >>> devctl set driver -f pci0:2:0:2 ppt >>> else >>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:3 >>> devctl set driver -f pci0:2:0:3 ppt >>> else >>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >>> fi >>> >>> echo "rtx 2080ti is fully attached to ppt" >>> >>> for vm in $vms; do >>> session=3D"${vm##*/}" >>> echo "bhyve session =3D $session" >>> echo "vnc session =3D $vncs" >>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; >>> then >>> printf 'VNC session not found,destroying ghost vms\n' >>> bhyvectl --vm=3D$session --destroy >>> else >>> printf 'Found VNC session %s\n' >>> "${session},no ghost vms found,not destroying them" >>> fi >>> done >>> >>> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (AM13N4CZ)/ && d{print d}'` >>> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >>> >>> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2015040204055E)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >>> >>> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2027285F1175)/ && d{print d}'` >>> echo "CT1000P1SSD8 ; $vmdisk2" >>> >>> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (20130508005976F)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >>> >>> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (BE0191510218)/ && d{print d}'` >>> echo "G-DRIVE USB ; $vmdisk4" >>> >>> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (38234B4237354B45)/ && d{print d}'` >>> echo "Elements 25A3 ; $vmdisk5" >>> >>> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (WD-WCAV2X797309)/ && d{print d}'` >>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >>> >>> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (20140108006C)/ && d{print d}'` >>> echo "Corsair Force 3 SSD ; $vmdisk7" >>> >>> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (1924E50B2AE5)/ && d{print d}'` >>> echo "CT500MX500SSD4 ; $vmdisk8" >>> >>> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (0774921DDC4200A6)/ && d{print d}'` >>> echo "SanDisk Cruzer-15GB ; $vmdisk9" >>> >>> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (60A44D4138D8F311190A0149)/ && d{print d}'` >>> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >>> >>> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (575845483038524844323238)/ && d{print d}'` >>> echo "WD 2500BMV External ; $vmdisk11" >>> >>> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (57442E575845323039544433303334)/ && d{print d}'` >>> echo "WD 3200BMV External ; $vmdisk12" >>> >>> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2414E989076B)/ && d{print d}'` >>> echo "CT500BX500SSD1 ; $vmdisk13" >>> >>> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w = -H -A \ >>> -s 0,hostbridge \ >>> -s 1,ahci-hd,/dev/$vmdisk13 \ >>> -s 10,virtio-net,tap6 \ >>> -s 11,virtio-9p,sharename=3D/ \ >>> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >>> -s 31,lpc \ >>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >>> >>> As you can see,each vm has a unique ID (connected with the vncviewer >>> instance) and the vncviewer window is launched as soon as the bhyve vm = is >>> called...and my Geforce RTX 2080 ti is passed through inside the vm. >>> >>> >>> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >>> wrote: >>> >>>> First of all,you could use a more flexible technique than using the >>>> pptdevs on /boot/loader.conf. I would use a script like this : >>>> >>>> 1) >>>> >>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>>> echo "18/0/0 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:18:0:0 >>>> devctl set driver -f pci0:18:0:0 ppt >>>> else >>>> echo "18/0/0 already attached to ppt" >>>> fi >>>> >>>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>>> >>>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>>> shamim.shahriar@gmail.com> wrote: >>>> >>>>> Good afternoon everyone. >>>>> >>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have che= cked >>>>> the device IDs and put pptdevs in place >>>>> >>>>> # cat /boot/loader.conf >>>>> pptdevs=3D"18/0/0 19/0/0" >>>>> pptdevs2=3D"72/0/0 73/0/0" >>>>> >>>>> this is showing the GPUs as pptdev on the pciconf >>>>> >>>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>>> vendor =3D 'NVIDIA Corporation' >>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>> class =3D display >>>>> subclass =3D 3D >>>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>>> vendor =3D 'NVIDIA Corporation' >>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>> class =3D display >>>>> subclass =3D 3D >>>>> >>>>> as I am using vm-bhyve, I have put the configuration as below >>>>> >>>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>>> loader=3D"uefi" >>>>> cpu=3D16 >>>>> memory=3D128G >>>>> xhci_mouse=3D"yes" >>>>> debug=3D"true" >>>>> >>>>> graphics=3D"yes" >>>>> graphics_listen=3D"127.0.0.1" >>>>> graphics_port=3D"5920" >>>>> graphics_res=3D"1024x768" >>>>> graphics_wait=3D"no" >>>>> #graphics_vga=3D"io" >>>>> >>>>> network0_type=3D"virtio-net" >>>>> network0_switch=3D"swUNI" >>>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>>> >>>>> disk0_type=3D"nvme" >>>>> disk0_name=3D"jagadish-disk0.img" >>>>> #disk0_size=3D"128G" >>>>> >>>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>>> >>>>> passthru0=3D"19/0/0=3D6:0" >>>>> >>>>> bhyve_options=3D"-A -H -P" >>>>> #END >>>>> >>>>> however, when I start the vm (for OS installation to start with), it >>>>> shows it is running but I am unable to access the VNC for a while, th= en >>>>> when I finally manage to connect via VNC, there is nothing on the scr= een, >>>>> just blank dark screen >>>>> >>>>> the vm-bhyve.log shows >>>>> >>>>> Mar 24 13:59:26: initialising >>>>> Mar 24 13:59:26: [loader: uefi] >>>>> Mar 24 13:59:26: [cpu: 16] >>>>> Mar 24 13:59:26: [memory: 128G] >>>>> Mar 24 13:59:26: [hostbridge: standard] >>>>> Mar 24 13:59:26: [com ports: com1] >>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>>> Mar 24 13:59:26: [debug mode: true] >>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>>> Mar 24 13:59:26: [primary disk dev: file] >>>>> Mar 24 13:59:26: initialising network device tap0 >>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: booting >>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>>> Mar 24 13:59:26: [bhyve iso device: -s >>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>>> Mar 24 13:59:26: starting bhyve (run 1) >>>>> >>>>> based on what I can see, and a little out of desparation, I decided t= o >>>>> run the installer in the foreground, and below is what I got >>>>> >>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>>> >>>>> it stays there for as long as I wait until I poweroff the vm. >>>>> >>>>> tried with Debian installer, >>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>>> >>>>> >>>>> ideally I need to install Debian for my user base on these GPU based >>>>> systems. But even that is proving to be impossible since the installe= r does >>>>> not move any further from where it is. >>>>> >>>>> Any thoughts/ideas/suggestions what I else I can try to make this >>>>> work? Anything that you have tried that worked? have I missed somethi= ng? >>>>> >>>>> Would appreciate any and all thoughts/suggestions >>>>> >>>>> best regards >>>>> SS >>>>> >>>> >>>> >>>> -- >>>> Mario. >>>> >>> >>> >>> -- >>> Mario. >>> >> >> >> -- >> Mario. >> > --0000000000009f3bc906311a0606 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mario, tried options similar to what you suggested, no = difference to the outcome. Stuck where it was before.

Be= st regards
SS

On Mon, 24 Mar 2025 at 1= 5:29, Shamim Shahriar <sham= im.shahriar@gmail.com> wrote:
Hi Mario

I will tr= y the options you have used later and let you know how it goes. Not sure if= this is intentional or typo, but I don't recall any bhyve-win binary i= n my system.

Best regards
SS
=
On Mon= , 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wrote:
=
Oh = sorry. I forgot to add the following bhyve parameters necessary to passthru= my GPU :D

-s 8:0,passthru,2/0/0 \
-s 8:1,pas= sthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
On Mon, = Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
=
Usually I use this kind of script to launch a bhyve vm :
#!/bin/sh

setxkbmap it
kldload vmm.ko
vms=3D"$(ls = /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncviewer [0]/{print = $6}')"
kldload vmm.ko

if ! pciconf -l pci0:2:0:0 | grep = -q "^ppt"; then
echo "rtx 2080ti slot 2/0/0 is not attach= ed to ppt,attaching..."
kldload nvidia-modeset
devctl clear driv= er -f pci0:2:0:0
devctl set driver -f pci0:2:0:0 ppt
else
echo &qu= ot;rtx 2080ti slot 2/0/0 is already attached to ppt"
fi

if != pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
echo "rtx 2= 080ti slot 2/0/1 is not attached to ppt,attaching..."
devctl clear = driver -f pci0:2:0:1
devctl set driver -f pci0:2:0:1 ppt
else
echo= "rtx 2080ti slot 2/0/1 is already attached to ppt"
fi

= if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
echo "r= tx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
devctl cl= ear driver -f pci0:2:0:2
devctl set driver -f pci0:2:0:2 ppt
else
= echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
fi
=
if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
echo &qu= ot;rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
devct= l clear driver -f pci0:2:0:3
devctl set driver -f pci0:2:0:3 ppt
else=
echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
fi=

echo "rtx 2080ti is fully attached to ppt"

for vm = in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sess= ion=3D"${vm##*/}" =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 echo "bhyve session =3D $session"
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "vnc session =3D= $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if = ! printf '%s\n' "${vncs}" | grep "${session#vm}"= ;; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'VNC session no= t found,destroying ghost vms\n'
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 bhyvectl --vm=3D$session --destroy
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 print= f 'Found VNC session %s\n' "${session},no ghost vms found,not = destroying them"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 fi
done

vmdisk0=3D`geom disk list | awk '/^Geom name: = /{d=3D$NF} /^ *ident: (AM13N4CZ)/ && d{print d}'`
echo "= ;Seagate M3 Portable 1.8T ; $vmdisk0"

vmdisk1=3D`geom disk list= | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2015040204055E)/ &&= d{print d}'`
echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1&qu= ot;

vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (2027285F1175)/ && d{print d}'`
echo "CT1000P1S= SD8 ; $vmdisk2"

vmdisk3=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{print d}'`
= echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"

vmdisk4= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (BE0191510= 218)/ && d{print d}'`
echo "G-DRIVE USB ; $vmdisk4"= ;

vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *i= dent: (38234B4237354B45)/ && d{print d}'`
echo "Element= s 25A3 ; $vmdisk5"

vmdisk6=3D`geom disk list | awk '/^Geom = name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && d{print d}'`<= br>echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"

vmdisk= 7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20140108= 006C)/ && d{print d}'`
echo "Corsair Force 3 SSD ; $vmd= isk7"

vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$= NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'`
echo "CT= 500MX500SSD4 ; $vmdisk8"

vmdisk9=3D`geom disk list | awk '/= ^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}= '`
echo "SanDisk Cruzer-15GB ; $vmdisk9"

vmdisk10= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44D413= 8D8F311190A0149)/ && d{print d}'`
echo "Kingston DataTr= aveler 2.0 ; $vmdisk10"

vmdisk11=3D`geom disk list | awk '/= ^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)/ && d{= print d}'`
echo "WD 2500BMV External ; $vmdisk11"

v= mdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57= 442E575845323039544433303334)/ && d{print d}'`
echo "WD= 3200BMV External ; $vmdisk12"

vmdisk13=3D`geom disk list | awk= '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ && d{print= d}'`
echo "CT500BX500SSD1 ; $vmdisk13"

/usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H = -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/$vmdisk13 \
-s 10,virtio-= net,tap6 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D95= 0 \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_C= ODE.fd \
vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &am= p;

As you can see,each vm has a unique ID (connect= ed with the vncviewer instance) and the vncviewer window is launched as soo= n as the bhyve vm is called...and my Geforce RTX 2080 ti is passed through = inside the vm.


On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marie= tto <mariett= o2008@gmail.com> wrote:
First of all,you could use a more flex= ible technique than using the pptdevs on /boot/loader.conf. I would use a s= cript like this :

1)

if != pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
echo "18= /0/0 is not attached to ppt,attaching..."
devctl clear driver -f pc= i0:18:0:0
devctl set driver -f pci0:18:0:0 ppt
else
echo "18/= 0/0 already attached to ppt"
fi

2) I would n= ot like to use vm-bhyve. It adds only some confusion...

On Mon, Mar 24= , 2025 at 3:22=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:<= br>
Good afternoon everyone.

I am trying to have VMs with GPU passthrou= gh. The setup is a Dell server with NVidia Quadro RTX 6000/8000 installed a= lready. I have checked the device IDs and put pptdevs in place

# cat= /boot/loader.conf
pptdevs=3D"18/0/0 19/0/0"
pptdevs2=3D&qu= ot;72/0/0 73/0/0"

this is showing the GPUs as pptdev on the pci= conf

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0= xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice= =3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation= '
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 60= 00/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2= =A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0= class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 sub= vendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 = =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D = 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2= =A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D

as I a= m using vm-bhyve, I have put the configuration as below

# cat /mnt/V= Ms/jagadish/jagadish.conf
loader=3D"uefi"
cpu=3D16
memor= y=3D128G
xhci_mouse=3D"yes"
debug=3D"true"
graphics=3D"yes"
graphics_listen=3D"127.0.0.1"
g= raphics_port=3D"5920"
graphics_res=3D"1024x768"
g= raphics_wait=3D"no"
#graphics_vga=3D"io"

netw= ork0_type=3D"virtio-net"
network0_switch=3D"swUNI"network0_mac=3D"58:9c:fc:06:3f:af"

disk0_type=3D"nv= me"
disk0_name=3D"jagadish-disk0.img"
#disk0_size=3D&q= uot;128G"

uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de&quo= t;

passthru0=3D"19/0/0=3D6:0"

bhyve_options=3D"= ;-A -H -P"
#END

however, when I start the vm (for OS install= ation to start with), it shows it is running but I am unable to access the = VNC for a while, then when I finally manage to connect via VNC, there is no= thing on the screen, just blank dark screen

the vm-bhyve.log shows
Mar 24 13:59:26: initialising
Mar 24 13:59:26: =C2=A0[loader: uefi= ]
Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[memory: 12= 8G]
Mar 24 13:59:26: =C2=A0[hostbridge: standard]
Mar 24 13:59:26: = =C2=A0[com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-1293-11ef= -a9a4-e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode: true]
Mar 24 = 13:59:26: =C2=A0[primary disk: jagadish-disk0.img]
Mar 24 13:59:26: =C2= =A0[primary disk dev: file]
Mar 24 13:59:26: initialising network device= tap0
Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
Mar 24= 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: bo= oting
Mar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l boot= rom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-= 11ef-a9a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices: -s = 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s = 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,f= buf,tcp=3D127.0.0.1:592= 0,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve= console: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve i= so device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,r= o]
Mar 24 13:59:26: starting bhyve (run 1)

based on what I can se= e, and a little out of desparation, I decided to run the installer in the f= oreground, and below is what I got

# vm install -f jagadish FreeBSD-= 14.2-RELEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found guest i= n /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x112= 245400000 [sz 16777216]

it stays there for as long as I wait until I= poweroff the vm.

tried with Debian installer,
# vm install -f j= agadish debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0 * fo= und guest in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer = base: 0x2747e2400000 [sz 16777216]


ideally I need to install Deb= ian for my user base on these GPU based systems. But even that is proving t= o be impossible since the installer does not move any further from where it= is.

Any thoughts/ideas/suggestions what I else I can try to make th= is work? Anything that you have tried that worked? have I missed something?=

Would appreciate any and all thoughts/suggestions

best regar= ds
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--0000000000009f3bc906311a0606-- From nobody Mon Mar 24 19:13:42 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZM2lW054dz5rKCD for ; Mon, 24 Mar 2025 19:13:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZM2lV3hSZz3CCk for ; Mon, 24 Mar 2025 19:13:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742843622; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JMYYVF3oKkl4P8FgFJMsha3qVhd0CSqlVn6YWMv6vHc=; b=x4eEUbYV2YDRVBoLxcyBKWLT2JuLKjsHICSBeb11BWU4GhMFfu6hTDS0KCq4JG/XnqnnEw iKbrlB44coNl63h5sOqjB5fWBl1PIa6iOmSmqWD8nDIyqlkpi/e9t7u3vgdW4tfvt1ueaN bmA0whaZdNJuaZ2VYtAGhYppULD8mWx5LLMIyQ/OTRGTTqUd46mGRUudE5cTxXmLj76l/q CtOcI+SV9MZsKLBCXN2mWvAF70KIJUYPZMlG2rKnRox3V7MwBmJt1NQT+bQIki4M3+nNcY /CKjlRY8qbO9yOp1x09/PbyhAtI5tsQxDzo9nUv/9ZAcm8nC3NqsrL2RMSeuZQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742843622; a=rsa-sha256; cv=none; b=iOtd1GpIvNzY1lIO9nVq7e0nqrb5K3JGJIRRIn4Hqsul3oe4+dlgWq5vPelByUdSHllpi/ 6SGTEJ8oJ+p5LzfMoK5kahpovQNHv4JSiYzua8kF1PtHz2V3S/T47EoPIdkSo28JCU/HpV DefoLqqN61eq+P2RWLop58tdUjSOu24VcKnOEBlKfdWLd3takQq02s0MsDL1VfVGoFW6BY +Y3sJR3/Bj59qXI4QN1PxLo76RP40R7JK4O9mgcScNCHsQlrrdYg6NqOCeBqoGTNvWRSTY N8PwaqPdtilAHWaiUx+r4P2Oopeu62nc+FEd6luqeGCHKgjGf4QWlP9ohnQY0Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742843622; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JMYYVF3oKkl4P8FgFJMsha3qVhd0CSqlVn6YWMv6vHc=; b=kD3qhFJJWkx18DIdodgPgz5sehBFoZddQ/V5GP7jnHHgNjnTPQdM9ZgPqJE1NqufqBkzHf Whjhz6Dg//8RoUjoerkP3O/3YbCJNlP473E7NhWcyAYNIhpokh9scsiDwvuxisjhx7fcT5 ciV/knBosZclaIWkyDtEBVrhjMc/VizYPtuAqs51lNxRlgPwjZ/MdsApLcTo2LYDcFCt1z kRWMXV4WZPAlZ24Y5GANs4CgzN4fkY5TqFH8WLqEGrcXdW42ghTyvSf4egDmkf88nunfT2 fq/TkqAJJkNNCAmkK8OSTlDAjhLWUomCL0Zqgh5LnLpuAeJOTGPXr48XFbxyoA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZM2lV32mqz16Sr for ; Mon, 24 Mar 2025 19:13:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52OJDgtl050735 for ; Mon, 24 Mar 2025 19:13:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52OJDgX1050734 for virtualization@FreeBSD.org; Mon, 24 Mar 2025 19:13:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Mon, 24 Mar 2025 19:13:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #12 from Dimitry Andric --- (In reply to Wei Hu from comment #11) The older version I'm running successfully on this Hyper-V host isstable/14-n270742-5c97b7c296ac, which is a relatively recent stable/14 snapshot. For some reason this kernel has no issues with storvsc... :) Here's some storvsc related sysctl output from a running system, maybe it is useful: storvsc0: on vmbus0 storvsc1: on vmbus0 storvsc2: hw.storvsc.chan_cnt: 0 hw.storvsc.max_io: 512 hw.storvsc.ringbuffer_size: 262144 hw.storvsc.use_pim_unmapped: 1 hw.storvsc.use_win8ext_flags: 1 dev.storvsc.2.data_sg_cnt: 0 dev.storvsc.2.data_vaddr_cnt: 6 dev.storvsc.2.data_bio_cnt: 0 dev.storvsc.2.channel.24.send_req: 0 dev.storvsc.2.channel.16.send_req: 6 dev.storvsc.2.channel.16.sub.1.br.tx.state: intrcnt:0 rindex:0 windex:0 ima= sk:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.2.channel.16.sub.1.br.rx.state: intrcnt:0 rindex:0 windex:0 ima= sk:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.2.channel.16.sub.1.mnf: 0 dev.storvsc.2.channel.16.sub.1.cpu: 3 dev.storvsc.2.channel.16.sub.1.chanid: 24 dev.storvsc.2.channel.16.br.tx.state: intrcnt:8 rindex:1112 windex:1112 ima= sk:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.2.channel.16.br.rx.state: intrcnt:0 rindex:968 windex:968 imask= :0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.2.channel.16.mnf: 0 dev.storvsc.2.channel.16.cpu: 2 dev.storvsc.2.%iommu: dev.storvsc.2.%parent: vmbus0 dev.storvsc.2.%pnpinfo: classid=3Dba6163d9-04a1-4d29-b605-72e2ffb1dc7f deviceid=3D00958b68-9b0f-4115-bea0-24f4872073d3 dev.storvsc.2.%location: dev.storvsc.2.%driver: storvsc dev.storvsc.2.%desc: Hyper-V SCSI dev.storvsc.1.data_sg_cnt: 0 dev.storvsc.1.data_vaddr_cnt: 119 dev.storvsc.1.data_bio_cnt: 1 dev.storvsc.1.channel.3.send_req: 121 dev.storvsc.1.channel.3.br.tx.state: intrcnt:115 rindex:16648 windex:16648 imask:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.1.channel.3.br.rx.state: intrcnt:0 rindex:11000 windex:11000 imask:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.1.channel.3.mnf: 0 dev.storvsc.1.channel.3.cpu: 1 dev.storvsc.1.%iommu: dev.storvsc.1.%parent: vmbus0 dev.storvsc.1.%pnpinfo: classid=3D32412632-86cb-44a2-9b5c-50d1417354f5 deviceid=3D00000000-0001-8899-0000-000000000000 dev.storvsc.1.%location: dev.storvsc.1.%driver: storvsc dev.storvsc.1.%desc: Hyper-V IDE dev.storvsc.0.data_sg_cnt: 0 dev.storvsc.0.data_vaddr_cnt: 1916 dev.storvsc.0.data_bio_cnt: 3948 dev.storvsc.0.channel.2.send_req: 5899 dev.storvsc.0.channel.2.br.tx.state: intrcnt:4960 rindex:189600 windex:1896= 00 imask:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.0.channel.2.br.rx.state: intrcnt:0 rindex:3336 windex:3336 imas= k:0 psndsz:0 fvalue:1 ravail:0 wavail:258048 dev.storvsc.0.channel.2.mnf: 0 dev.storvsc.0.channel.2.cpu: 0 dev.storvsc.0.%iommu: dev.storvsc.0.%parent: vmbus0 dev.storvsc.0.%pnpinfo: classid=3D32412632-86cb-44a2-9b5c-50d1417354f5 deviceid=3D00000000-0000-8899-0000-000000000000 dev.storvsc.0.%location: dev.storvsc.0.%driver: storvsc dev.storvsc.0.%desc: Hyper-V IDE dev.storvsc.%parent: --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Mar 24 22:20:31 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZM6v42sWyz5rXvh for ; Mon, 24 Mar 2025 22:20:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZM6v41ptdz3KL2 for ; Mon, 24 Mar 2025 22:20:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742854832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NrPzA3HD5uKrykNxOEfUTkOHH6tuI+8Te+QArRqevIk=; b=ucEt3dmDhC1Sgwd2b3CAP6Jw7JvuX8eIFSnFqoCq7l8PZHZZBl9e48RY1tL51fqQuz6jcw N56RFnu1MUmRE+CClIyV4GqSHXKsGxwd6eiokP8nerBn91wjxNVZOIrEearaMNaK0Ho5gy AGqbQM5tcDCcZM5nrTg81n6vYRTfB3DzTdxo4FY3bKxgSpwosQGw5TdP6Y8mU6N55PYMV1 232lJ8j+olJex9XkgniIYHwuNDZB8YvlORfli7rv2wB4EGkVbRniNA+hQUcPCTmxtNuasj jPhaAnnE5/CsSXSncGkT1DlBXVMMDUveoZ5x/Zaj8Tagg+T9PfXIG/sf2CkBoA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742854832; a=rsa-sha256; cv=none; b=ITaF55vOw7FLgdF1lPcnvsUQwJO7FEtxDD073kU2UJzhaaDt2gHbvXV7iyvkaxrZJf7CJN zhLm3fZlUFlKuQeUIbBl4H9IVeWALHjDgDe6xJiZ31P6ayiO/ot5VQNWKFJ8S5WUVlc/tP WdbzA5m59qWqai8ghkggp5v5ha1yqlP5i+v8Hn83d2HEGcn8haeq/QQ++VaCkdRNY1Cao8 BPjf1vnQ4UdcHCvnqqIN3PkzClNJtdJUjINgjLP+YNQg7NWh7RmQB73eBtcf9w4VfrIka7 c/tkFfp0bdG/xBSjGNM3AFzDMw8TrXpJlZY1DQ3XrpUK+o6sKsnq8k5Tq/vYGg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742854832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NrPzA3HD5uKrykNxOEfUTkOHH6tuI+8Te+QArRqevIk=; b=rafJi70xK/RjeJb1g7zFeDEsjvgg/JQ7tphFajZq5lpzB22XfrXO752zX22ZViqznpwnVT tgMEM/ftCTqAViqeOSfxDfc7Cj/LCQsETT3DUpoupbh05Im7F5H4897vjTvbVPN/NFTKjb Pl1cjik6qM9JE634ik8ULuMvzzdls9fw8Z9qEjfnZZS7KqYCY93Vf9zPjGQtvwcv07w3ND uDikb0MVoQSUx8xdfAc4jkuV9eBypny75wWemJ2+ibJg6CTZt5fbeVGsatv63ZXUZY3bG+ phcQAZL+8oFVHpDHKobIdAuJ8sX7y+47RNZ8RSXmEKC1XcPB4KcrPQFC7up47w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZM6v419tdz1BxK for ; Mon, 24 Mar 2025 22:20:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52OMKWox097145 for ; Mon, 24 Mar 2025 22:20:32 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52OMKWgS097143 for virtualization@FreeBSD.org; Mon, 24 Mar 2025 22:20:32 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Mon, 24 Mar 2025 22:20:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #13 from Mark Johnston --- (In reply to Dimitry Andric from comment #12) > For some reason this kernel has no issues with storvsc... :) Well, if it's stable/14 GENERIC, it won't have assertions enabled, so won't panic in the same way. Does a GENERIC-NODEBUG kernel built from main with = my suggested change boot successfully? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Mon Mar 24 22:38:15 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZM7HW6mpWz5rYk2 for ; Mon, 24 Mar 2025 22:38:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZM7HW6CZnz3XPt for ; Mon, 24 Mar 2025 22:38:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742855895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=411OAPIk9cC2hH5vHBzO4i6eNKiFqmQaeKxNIYdinkQ=; b=gFvEbAHRyMsMzFZDaXxawfayQauH4L5xidevHkmY9PrZBMS4tDG0rzntjp87QDAgfAs7k5 ogl+/kWAgFtTA+SHTpRETnFh/o4eYjpRWEWXfCt49kn1aA9h9kfjp5f5zvrCY1CV1QltLO NRUPX2Ra39ACOTFMx4CI++xHwVlnDgwhtso7rjOQ1VaZNSPhQ+tbI2jxsHPkVT8kPdDzhW vNKnWfd1phho1z7QgKBIvSqs6dFPJgz9UjO0VARHFp9NpqVRzFfePeq+ULD6bvTeEc8dBg dWgR+Sk6/+4++Z6jFL0T2k5QlKfCyeMCCsuMoLneqIqPxyVpOG6iL6pYJ7jzuw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742855895; a=rsa-sha256; cv=none; b=Bb7ib0WHgxu2SpaL6Mj1fgDynZEeI9FBd8flz49Hcsbx4AjLh5Y9k6Qu72ZOxHsfm9HOIW yL+Az3KzbF2C1ptAEUxgZsS1Dky13UqQXg32JS7E06k4622qnEoQ9N00y3J7G1u+63O2cL Ng8/HnFI5K50BA+2dxZ2sr3FurVNNRqg6DealE0uHjaUZj+5z++bPhzwEGrzTyJeGBBk8x GUiMrbMOus0s3Be/uEJuawCHsy5UteuC9xF7XSMEQ32dIPq+8RX+wlvWk+TtI7xg2cfB+p DBXAxFeEBgYV9Nuhkd/d0Wn+xPJsnVycbO90gnwm3lnlja1W84+3gogN7x57Uw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742855895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=411OAPIk9cC2hH5vHBzO4i6eNKiFqmQaeKxNIYdinkQ=; b=eqZs7nfYWaHjrIKJCakLKdkd64wKGFr5sA4qdysM5D9YBGOPePFOMEjqeJlDqoDbTzWeM6 M08sDRUDSqnQhGjuoueSCuB6I7LpQHzJl3MDlSaH6Q/ZFobnLrSoKtF7B9q68SSIoy3fZg LqKlbLqcA2Tm2Fbzszt3gZA7ohIJNnmJjUoPTZntQhz6tCqXvqNHhMRIptvHhStTq7k4kY P1xGne4OEi8LfgHQBx1k9MwP3bI/MEN53bpT9PSghoYAK2LZyBn9xpdJ7BfN+yUbrac5my RW4PqxQwC2vnMuuuJSHfVrq0rNOzaJryr/W5VoE2hj1QL0rMPOICfGkajG8xPA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZM7HW5lxpz1Ck5 for ; Mon, 24 Mar 2025 22:38:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52OMcFGL030139 for ; Mon, 24 Mar 2025 22:38:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52OMcFgH030138 for virtualization@FreeBSD.org; Mon, 24 Mar 2025 22:38:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Mon, 24 Mar 2025 22:38:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #14 from Dimitry Andric --- (In reply to Mark Johnston from comment #13) Ah yes, you're right of course. GENERIC-NODEBUG boots fine, in that case, a= nd goes to multiuser. I guess the assertions added in base 6f7b1310b6fe3 and base 0af5a0cd2788e should be refined? But I assume Wei won't have added them for fun. :) In any case, the file system seems to behave normally, as far as I can see. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 25 03:28:57 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZMFky4xp1z5rtS1 for ; Tue, 25 Mar 2025 03:28:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZMFky2n6Gz4Lqj for ; Tue, 25 Mar 2025 03:28:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742873338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2BkRHG7YMqCz96S/JHNArmbbzFKD50FfvoCsgnHScAA=; b=K7wXQ86DQU9TQ+Q2K/POdAVaD1fRZZ9tevp78VZvHJZbs8QXtTyV2CZVSczXM9LdZ6A0mn SLXhDQeq4QnkJ4aIPxyJ0G7aHgkEg00yw+GzSB3ejD+j6r1dgsjyBTlIB+s4cawGMlLk8A 4geGYCJOJRMM0KH6mgcLoauGhkY5q/hpxGFGi3lzcA9wbIqjLRYo+qSXGWmrc+aGYfZstr ez/zFV48G4+wfVGbFrmHPUODc+o8PJ1AQUljmLXmvDwC2dPk33EjolJYwq7JK6ldEs3OSC OtbmtlVxYu2cAu0yFYOykYzRrUA4SisstY1KXZzTu5tgXg1OeoMYReOjEQhCCw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742873338; a=rsa-sha256; cv=none; b=SJ9jI99b69jCzzxSa7cnSG51gmyIfHyKdURf5SMvNq3VfgwXPIM03yg/WJS9aAseSlj0qD c5nVRZVaIgZJ4qfJ42cM6MQ2BZXdRnXXUW1BNIVVIzXkCLc0zd/LS4FZrLnNhUGJ2ERxa6 LCViGl+SFA+r9RQeE57QfhI9MgzSpqNsBEKNehNJnn0UOxIYZFhAxIKIkUwOYTqppCtv+x YrH4gKecJhWHIl8a/ZaT2xQx4WncNRQDx5wUuq7Dr/QcEXAZpFZMAygBUfJeB1VG5PjC0e pdM8ojNLS2hBqGdaV5ep3xCVG+S+r/jwYQRIOiwJ3hHrEkruC63W0wNTbW3Axw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742873338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2BkRHG7YMqCz96S/JHNArmbbzFKD50FfvoCsgnHScAA=; b=xiv2lrURWO07Z5z0s0vlCDlihQafVhJiuEvqGwqYH80ZNP7UDSRfQYgPgXjYLu39Pgzojd 2j3Zxb5Qcci6MUMZSmj0/iqX7ifLOz/NR1ybHdGGABg3uvDLJDb2uFMgElsXocEaU5Zhdo ZxhFjSPOYEyZQ3A/2jRdr+b+hg4RF/JVw8gAhv4CTpC/bHq3zSR/8AUXg/go8+ygVtZChU 6bScV77dJ+L3a95l43uKA5RVexHYR6LjAY/FyCyAsO2ZOA5UAZEDTuGqtpsPcWjfIuL+b/ edyE3nFQhM5+QaYeqafwx/rQZJCWLnY40UUwmqPohPnxzeP0M8oyArrEJ4APvw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZMFky294Sz7jm for ; Tue, 25 Mar 2025 03:28:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52P3Sw2S070330 for ; Tue, 25 Mar 2025 03:28:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52P3Swxi070329 for virtualization@FreeBSD.org; Tue, 25 Mar 2025 03:28:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Tue, 25 Mar 2025 03:28:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: weh@microsoft.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #15 from Wei Hu --- (In reply to Dimitry Andric from comment #14) Those asserts have been there since the beginning of storvsc driver. I didn= 't add them. They were removed in those two commits for aarch64. It is fine to remove them for aarch64 probably because it uses bounce buffer in busdma. X= 86 doesn't seem to use bounce buffer. A gap between first and second addresses seems directly inherent from the original requestor on x86 and alarming. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 25 13:26:17 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZMW0L0x6rz5rHgK for ; Tue, 25 Mar 2025 13:26:26 +0000 (UTC) (envelope-from 9vlc@proton.me) Received: from mail-106101.protonmail.ch (mail-106101.protonmail.ch [79.135.106.101]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZMW0H682fz3KF5 for ; Tue, 25 Mar 2025 13:26:23 +0000 (UTC) (envelope-from 9vlc@proton.me) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=proton.me header.s=protonmail header.b=eL18J1Rt; dmarc=pass (policy=quarantine) header.from=proton.me; spf=pass (mx1.freebsd.org: domain of 9vlc@proton.me designates 79.135.106.101 as permitted sender) smtp.mailfrom=9vlc@proton.me DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=protonmail; t=1742909180; x=1743168380; bh=xchB8yKj79mO2OyRXI+FuBQoTQF86Mba5CrEYj6BJXc=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector: List-Unsubscribe:List-Unsubscribe-Post; b=eL18J1RtDmFcBiyRf3UTveCrmOimO7S2Oar0LhCM0kH0emKq8sxOdEvs0dXq1QrBK bYhxa2Hts2i0Q1WEgEq/mBww0Qo+WfuKGL4QXydPpIzBZUvwnxm1lHUu5yqpuqDSkH 07dLSumY1Dbxbxrk6WijAzqplo5SolvilH/kIamIBZmmJSj2JFz+jkruSDOBR77myM WtnYq12ecR6fcSf6f7fPAmejtpInOrGvgTeA/zgJGHzq54lqaakPDkAlIw/BrIDs/E qYViv7RjiAQzR6rn0Jc0Xs5YlQtK13FC0F1w8oOtiSE67/pWUEpxKbpkzCy9U41Lyw XdcplrGJMA8OQ== Date: Tue, 25 Mar 2025 13:26:17 +0000 To: "freebsd-virtualization@FreeBSD.org" From: Alex <9vlc@proton.me> Subject: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" Message-ID: Feedback-ID: 117452339:user:proton X-Pm-Message-ID: 57f29f9129675e010c012cc4685eeacefb83536b List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.32 / 15.00]; RBL_SENDERSCORE_REPUT_9(-1.00)[79.135.106.101:from]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; NEURAL_SPAM_LONG(0.77)[0.766]; DMARC_POLICY_ALLOW(-0.50)[proton.me,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; R_DKIM_ALLOW(-0.20)[proton.me:s=protonmail]; RWL_MAILSPIKE_GOOD(-0.10)[79.135.106.101:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@FreeBSD.org]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[proton.me:+] X-Rspamd-Queue-Id: 4ZMW0H682fz3KF5 X-Spamd-Bar: --- This is a hefty help email for Shamim Shahriar in regards to passing throug= h Nvidia GPU to a bhyve VM. Hi. I have recently gotten this mailing list message forwarded to me by Michael= Dexter https://lists.freebsd.org/archives/freebsd-virtualization/2025-Marc= h/002818.html and it alone made me sign up here. So, here's a big mail containing a bunch of useful GPU passthrough info! To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, l= et me help you out make it work correctly! Parts and short description: 1. Pass through the GPU audio and other devices (if exist) =09Modern GPUs usually create two PCI devices that you can see in pciconf -= VGA and audio. They come on the same PCI function (i think it is called th= at), for example GPU 19/0/0 will also have 19/0/1 for audio and may have 19= /0/2 for usb. 2. Give the GPU a VGA BIOS as PCI Option ROM =09GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a trimmed VG= A BIOS since the PCI option rom that is used by the VM is nested inside the= ROM file. =09(from now on i will refer to the vga bios as vbios) 3. Possibly, re-BAR in BIOS? =09Some UEFI firmware have a feature called re-BAR and "above 4G decoding /= mining", it makes some GPUs in some configurations not work in passthru. 5. Possibly, Try newer EDK2 =09The edk2-bhyve package (provides UEFI firmware for bhyve) on the repos i= s still using the 2023 release. You can compile a 2025 version with my patc= h: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285554 Full parts and HOWTOs: !!!! MISC GPU DEVICES As decribed before, pass through all the devices that your GPU creates in p= ciconf right after the GPU device on the same PCI function. All audio, USB, etc. devices belonging to the GPU should be passed through = with it since they are a part of it, else the passthru will NOT work correc= tly. If that is not the case and there is only a single GPU VGA device with noth= ing else Nvidia to be found, skip this step. !!!! VBIOS You first need to get the pure vBIOS file. To do that you can either: 1. Use a Linux system and dump it with https://github.com/ilya-zlobintsev/L= ACT 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz 3. Download one that someone has dumped from their GPU using GPU-z from htt= ps://www.techpowerup.com/vgabios If you don't want to install a different OS on the PC you're currently runn= ing, we can just find and download a correct vBIOS from Techpowerup. But we stumble across an issue! You specified "NVidia Quadro RTX 6000/8000"= , so which GPU is it really? Those are two different GPUs with different sp= ecifications! Thankfully, we have your pciconf output which says "subdevice=3D0x13d8" and= if we refer to Nvidias documentation, on page 7 over here: https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/qua= dro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-121= 9.pdf we can see that it is a RTX 8000! Here is an unverified but most likely working vBIOS: https://www.techpoweru= p.com/vgabios/256271/256271 Now after we've gotten the vBIOS, we need to trim the start of the file to = get to the embedded PCI Option ROM. For that we will use my tool "trim_nvidia_vbios" from https://github.com/9v= lc/ptnotes located in progs/trim_nvidia_vbios Compile and run the resulting executable with the first argument being the = downloaded vBIOS and the second argument being an output PCI OpROM. Example= : $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c $ ./trim.elf techpowerup.rom trimmed-vbios.rom And we come upon a road block: vm-bhyve does not support passing a PCI Opti= on ROMs in "passthru=3D", so we're going to use "bhyve_options=3D" instead = to define the GPU VGA device: bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D" Other devices that do not require a PCI Option ROM can be added with "passt= hru=3D". These devices include the rest of the GPU pci functions like audio= and usb. !!!! RE-BAR If after everything you do get the GPU to work in the VM at least in the ed= k2 firmware, Go to your regular UEFI BIOS setup utility and search for options titled "B= AR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for Mini= ng". If you do see re-BAR but cannot disable it, you could try using https://git= hub.com/DavidS95/Smokeless_UMAF to modify locked UEFI settings. Be warned t= hough, smokeless is a heck of an unstable MESS. !!!! UPDATED EDK2 As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some ra= re cases, that might cause issues. I have submitted a package update bug report for sysutils/edk2 to FreeBSD b= ugzilla over here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2855= 54 You can download the patch, apply it to your ports tree, then compile and i= nstall the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UEFI firmwar= e. ~~~~ Hope I didn't forget anything! I wish you success in the GPU passthrough journey!! - Alexey From nobody Wed Mar 26 12:01:35 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZN54020xXz5rqVy for ; Wed, 26 Mar 2025 12:01:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZN53z590qz3rDW for ; Wed, 26 Mar 2025 12:01:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742990495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ETbP26HnwUXzVoipUvc8L98oHmKk2cWoPTjacjF1+JQ=; b=RaF9c758ydFc4LCa6KStCSpiXcGUXkTivuGXEujvxUZy8jvz7mYO2naDpJ3azDkVBnlLuS y8Pue6tjqXYPdtSgK4wgtF6+5Xh7u18FYLJu/Zf2byQEDvTP1IWuYakuV9z3BioBFSf4v8 aUiasBkaF7MRW66M13YgXEH/Fcn/G1MBJS16t+E4NJ25NassScDpmE72J6TxGzqPJR1sru 9yhcOL1gWNS4yo4rkj81Zla1hs1Dolx5uZXo5bDZGmeU4DEHn5uB3iHlSOaR8bM1AYazGW kkv8ZP0OhJJ1A2Vh/b7zWLnEjxtdR83YNndXavi65PtQysIaCBf4pQ4pG4ASVA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742990495; a=rsa-sha256; cv=none; b=rAqS0vQEuW5vJdKJQr/8gLcKTMmWs8NnZjPxCKFQ4bL+EjuRMDwRcVEqG/y32YvcPHtxlM IgzMvFNlW9qJQirs82szPcRBz0wymD2SNCJm8uTRCG0hGIFJh7ZARwSDNNm6NSnUJRm30F EsUmF0evZmVz1JdXyLpM2uLrnvektaNgWjdZ+aPHaeRt2OJ4OV+bjDA4PZIx0mf2vnwd4h aJDHflrY4d7v/F2wh+xVFbB/VukM103r6JdXFGBt5ZTeOx42pWvL6v24Zow/Yqd/8twCVp 70LqpmDzC5GcZFd+ryjDVDppRS/G/iC2sYmDn3yji4pJXt895LQUNw3SrApQig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742990495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ETbP26HnwUXzVoipUvc8L98oHmKk2cWoPTjacjF1+JQ=; b=OjKoUKd+OgbNMNDTDgocgrBs7RhQ6QyGLmuU/ihz2ZpRF2JR2CoQlW2M6VgxxrCv9C1+6w W9l6QDXAIArzMXW4P2fSpRtIE0fSo5NDvLofEdq64Iw+BFJh3bMR1PJusTN5vGEMnaAE03 puFJCzEX/82vZAXXLPtFWTlB9NirVshk89SGjSOuU9YyR7umha95gcKoUKZ0fZN9ea309X mdLRcryhOj2a6nP3lSpP97fTuW6rrIfNLCdWdnKxoEAJO1z3J3UZzOwubUpZMn5qCMZc4w as9cs4nJpBLn6KAXH+uO/IzLQQ0MSsdYm38NmmGYcIMsKXWSwqU+aLcGtdrLxg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZN53z4jwDz1PGw for ; Wed, 26 Mar 2025 12:01:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52QC1ZSk062253 for ; Wed, 26 Mar 2025 12:01:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52QC1ZgN062252 for virtualization@FreeBSD.org; Wed, 26 Mar 2025 12:01:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Wed, 26 Mar 2025 12:01:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #16 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3De173855c9f6b3b87c975418d0acf1c22a= c4f1626 commit e173855c9f6b3b87c975418d0acf1c22ac4f1626 Author: Mark Johnston AuthorDate: 2025-03-26 09:34:53 +0000 Commit: Mark Johnston CommitDate: 2025-03-26 09:54:39 +0000 i386: Avoid registering overlapping vm_phys_seg entries The entry added in pmap_bootstrap() is guaranteed to be covered by that added in pmap_cold(). This apparently went unnoticed until commit 8a14ddcc1d8e ("vm_phys: Check for overlap when adding a segment"). PR: 285415 Tested by: dim MFC after: 2 weeks sys/i386/i386/pmap.c | 9 --------- 1 file changed, 9 deletions(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 26 12:16:20 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZN5P11R67z5rrV9 for ; Wed, 26 Mar 2025 12:16:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZN5P05vvNz3y8P for ; Wed, 26 Mar 2025 12:16:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742991380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qhmFY+93gCv8m4IaAAPfTSCfa/TPYx8QcSDllPzC1oE=; b=r68neyMbVrp76DfyKVgGQbTomYJeldXVhS3X1Tfh/duyNLIEkauanS8txUmmr5TB4bZZ9J 05pnP/uS7tgepp1TLmzYFxpVriVtETjd2cW6PC2wSkhDYE+akJc64k48HM1kQC9xUIE/3L HT8tIh8WbSvnNz6PFyjGZ/L+vOBzKg9BBN+/CjnKsNNqbkib5haIbKK3Wm3fgDyNJZRTnt cdHNkH1+4+4vDvC9iDgcG1zgISCPDfi0b3kM9pH82gg/rzP6vvhQxwNM2GZHseh768o+Et /Y2r7YiFnV+u6wd35lAUR9nwXhA4HOSMCkXxyxHIJd9GAQ9k7rz6HV6f/VgARA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1742991380; a=rsa-sha256; cv=none; b=Uk7vtHAiqRGKIG2X7C5KKsnVNo6gvW8zXLfXv83QkJEkDWfP4uee7LyT9ZKdcDYGO5vd1T ihfAg6DKM03wXPJ/4ub50h4UtIsIitynsW+8GMDhs80HNM9M3HOzvs9u4SadZ3Ooz05Smh W8/V4izbi+SsXicVP1w7FeWj75l/uHjqWxHuluzchSLDohgSQKiNN6lkrT/lmBa+TBqlo8 VeveWoZJUAgm1V89TgKrD+vkUDSRQFOkY6uoo6dnrWnWq49ov6SNWc7Oany+00x/77gnO+ /NiYQHOdbaKVp1UaTok7r9y6ypORm38oBBsFZ83RNjrtuLhws172i6N6zHqrsA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1742991380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qhmFY+93gCv8m4IaAAPfTSCfa/TPYx8QcSDllPzC1oE=; b=aMn5nNrk1vORG78D5LuIsGSEy1VP2FkMe1Ryg30OY8+uu67ZzrkPiPFkbFJLDGwo6H+UYP ++l8v7j6vu+5Oywzwj+PIBXM76RihZEW7KafmnLPXKrZcSWSF2jP61x4n26R2orf2MnsPc Q4UKKQhz/ZalAc+/Z0GjMw2LXnxMnl4sCKJ6y8vwzjoljbzteywIfM9FZlrEhD6NqcFWXA G0WI1J6cu/sPbseWwfsHcz/UxCAO1ImiNWCjhmWbOxaxp7pCmKXn7XxMtROvq4iHDP5h16 frJACjLEYyi5ouwdZZsekNB7aXdyLiHT901kc/YfpKe7EIM2X9UI9GY2n7iTRA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZN5P05GHSz1PLQ for ; Wed, 26 Mar 2025 12:16:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52QCGKbF093446 for ; Wed, 26 Mar 2025 12:16:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52QCGKbN093445 for virtualization@FreeBSD.org; Wed, 26 Mar 2025 12:16:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Wed, 26 Mar 2025 12:16:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #17 from Mark Johnston --- (In reply to Dimitry Andric from comment #14) So, the assertions here are only in effect if there is more than one segmen= t of DMA memory. In your case, the first segment is page-aligned but only 2048 bytes long, which is kind of weird. I wonder if you could add some code to print the full segs[] array before panicking? Something like https://people.freebsd.org/~markj/patches/storvsc_debug.diff (Maybe this should be taken to a new PR, since it's somewhat unrelated?) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 26 21:52:16 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNL9b3Psgz5rFpd for ; Wed, 26 Mar 2025 21:52:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNL9b1nCCz3Q23 for ; Wed, 26 Mar 2025 21:52:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743025939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bwOltmlt/mQ4+K5XhoUusVR0ieb3ROcX7KY1sP4TYkg=; b=Gvk2klwRlPF54a+ocNU0qeHF29ifwhtL5q7fJgLEReWrfSvaQEpBYq0glciVESxqgX7AGO 7jKbX2o4+GmGWuVPQrtNEuKayGHqgfW3+QvF7bxN6Pwyk90wpW5rrC3mtSUKWV3/h9Ilmf NtHEhNpHdwRFiZe26vER+st/lROZ4AuF1t0O59/M7OmyuGuFVGNNUK5Ff47muq/FzrjbuR B6H9DmKg5LNuzJ+meibqD+QsqghdEjEWaSe8GyOEYsklV35S1qlnRbhCu3Ezx+3PfwKhFa shUnNDvHiEjSxKotErjxmt1nkDXLD1XGYwrUjWJvQlRnIsCQHCH/wOtvedGNVQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743025939; a=rsa-sha256; cv=none; b=c4wTV1qoqrz+ibrikjH7XhB0rOkAAzhVDO9ESrWoFzNWeIw6UitIn8RlHw10h7rxTVD8Cz zMaJ45+NDl2wY5vxnTN0aNs4tmNEbT5V1VecNJcOh+NitTrxgx2Uy/zdwk/TxIcQGx4hxs UUWPwqSEFo/5h9zLsl+PypjFQrHG1T6dZHRq/SwxA+IUQWQGznNf1wjs2vKjYzCn7uGFf7 +IiZW5lKr04J4lO50a1adsTq36e8RCsEPeOv896mXwJE56STVCunCNv31jivOGQFWIC3A9 mhlraWH071bwpQDUlYVXwoN3/kN5JxXtWf49NbEqvULgSrqpIqaxitVaZSaJrA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743025939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bwOltmlt/mQ4+K5XhoUusVR0ieb3ROcX7KY1sP4TYkg=; b=wKF8Cx5k79sk5BItsH7QMEdsFKgG9oJtNrjgd5UHDGv3KzZHtgBCYbkHXW3L9sUEtR7XbT SkKC5fMql2NiwuAxUuD1elafwKAVnUbNUoR2NEA3D+Zj70MJ3ah9qJSCW1st6J7/FlCICh C4TiGOEcq4kMcOv8k/sw6uXyBUMH1WQ0xYWne5u6hneB/yWOdgZpdtHIehAO4QajXwEDph 9dzcEs9LTR7R63CVbiPry8LqoevvSY9oSS3I32WA+QK6bgm4/MAwXFDN/uQ80O2IK+8Uux dQstzyiyWjc07H8CU9qL1egc6pCUl5omRlX+Xcsui297a6meunzax/jGDj/Urg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZNL9b1B1JzD8G for ; Wed, 26 Mar 2025 21:52:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52QLqJLu061212 for ; Wed, 26 Mar 2025 21:52:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52QLqJfp061211 for virtualization@FreeBSD.org; Wed, 26 Mar 2025 21:52:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Wed, 26 Mar 2025 21:52:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 --- Comment #18 from Dimitry Andric --- (In reply to Mark Johnston from comment #17) Yes, continuing in bug 285681 with that one. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 26 22:03:19 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNLQJ2ZmLz5rGk4 for ; Wed, 26 Mar 2025 22:03:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNLQH6DhDz3Y3w for ; Wed, 26 Mar 2025 22:03:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743026599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ADbOy8AaPeNnFOIDX8BCrZgh449GJg+FOGABUmw5Jak=; b=ThDARbZTIoUydCgjNLnfCs/Ti/XyJj5xuBf1U4k5zzgZ0v+pk6ecTgexveqdgz/iYQXDIy BIAUqtf8k+9PTfhjiJuauy9L9nQ4Po3+TrXHs1iEcZV8zHjLme4CP6u0wYttbQvgCZ/2XP cUrVjXre0KI/JgLm8W6V220p78irUixcpab5oDN6Rix/sRhsrgUL6Nl+0tldvsiVcc7dFV /0qo8NgrPHeySOxKi3TbNkc2OVfMgUn6KYE/VoPNPU6fxFh2RcSNAHzZHeIIkr5Q2yzrsV 1hwPhi7PaQAiIMkKnh0XI9kazF+4Tgdc+dkvU+kqMfP62k3BXD7kFNQzoKihYw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743026599; a=rsa-sha256; cv=none; b=bSQ5byvQKDnQjjG3TgYEBB2jf39nhA0uwlV1wmQC7h5RzZrf8cS9H7ZNQUadMZ63/xEUrq PtvdvbeUDQt3rcOyNICz7ntEnI4kOViGuvzO6oNA2LyRlLtLql1e7RWuLQjMX7PrFWjgXc beIWRh5giXuN4qMarc0Alzbwdan/EN734flAojPAPKhNYHBrLiwEwftRL3EndpSNgiNrMn KlVeYh/1iYOiruR2QgywDm+uE2IVSweh8xllayVK/zaXfxcUJtG3rujCBtNdIs3wlb/Zz1 M2bT60v7lH2VjOrodaETJP0uA4rnoUgvYY7C3xW9os4KFqEUlhN+UjMM2xsaJw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743026599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ADbOy8AaPeNnFOIDX8BCrZgh449GJg+FOGABUmw5Jak=; b=Xi2f3OrdUzMwo1Ebf5IH1Gt9ByjadyJgJp7vXMW2G1fK+satQn/cWyMJqmJnK48rH6dPxx lxY6o/JsDHhAHekdP/Xf8djz8Ap/Yd134ePVN5OCR9wSAGv5dvIEu71EZk3Xw6v56TthgK srG1Ntd9t536FodvwUtX/U1tAHbbAI8W4Qf0j95XNGWtzrYLAA7hkUXMy1fH3ZoKJe+DQY GB6OWRfZ9LYBwXKrw+zAk7NkV8/D7jB5uH0NbOLsv0OoXryI3d8tWHrn7Pin2n4YIDp0kH qaNQ+DkowOxxLK3M7xaCU20GpCF6ltm+0krQvtfWG4QERGuh97bnS5OhWiAVkQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZNLQH4gxczDCM for ; Wed, 26 Mar 2025 22:03:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52QM3JNS081648 for ; Wed, 26 Mar 2025 22:03:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52QM3J5Y081647 for virtualization@FreeBSD.org; Wed, 26 Mar 2025 22:03:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Wed, 26 Mar 2025 22:03:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |crash Assignee|bugs@FreeBSD.org |virtualization@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 27 08:19:39 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNc5S0fZnz5s06l for ; Thu, 27 Mar 2025 08:19:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNc5S02y7z3hXw for ; Thu, 27 Mar 2025 08:19:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743063580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F2m9wPwwJr5LG0a7u8yVz94jABb0fxbpR1quGeje6Ik=; b=ruNFToFoDeW0WG5hBQzk8vhBK245I9paJdspynchBt9AXHIJkhXiT84WYg5mqss1IdDrRS cmQX69p0ghHR12JCb6ZrCZIhYfCewWGnkwtM5v9s+eHy7bcEo+MvBvJNErrhdYbdUXXGfC oPfgNRODYiSGR2zzhCkpNrA6oh/ATMY5qg1eiB9FkaO+jp4zYeO8f72TRHN7ervbqV9yxJ kDKfr0f928RUE6ffCkO8ccfBEJv+P/ZwQfgqYWXaFTSVBPA1XpuM3CkoDHA/5JrMevni+R zIIZ9WLIcnJKC6GofnwWqhFY1Z2UTeJRYsgZ7/FDgrdyLUh9kzgzXSGfiw5+nQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743063580; a=rsa-sha256; cv=none; b=kaR5WaGiq9AR6tR7dF4YI+zaOk+bnPiQHD1OlJV2IramiKF/6kdrL8TD4QOqne2cjx3Qxp XbrlFWebuxNebRA9jxVmLU+U3ysN+5CzyiHSRyenGCJO/Vit6mBwsaxCHC7DQrfx0Jw5Ub MdmpcBHEAdWJoLcuVdF9ela8u5BetjDC3Dl7Ma/HftuG6TPNtkdttY98G5hrxzi4izzpKH X5PyYZs5AuwjFeeoHq1KTmPbe5EeTVV0C14a7PtSTUx92coskhjslbk9DgkEdcv9Q7Od2E qfUnsmqjTz0ugqawF6ecfuElMuX0gq+thvsTcfVdiJUYhEuf+RqYkgtrhzoKTw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743063580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F2m9wPwwJr5LG0a7u8yVz94jABb0fxbpR1quGeje6Ik=; b=jNV/fIwOXR6JHg88BQzV0CvXmxIQCs9KNcdX/kwQKZm0La/oT/plYDgk82jHhieoF6Bumf 2qZQk9zu+2MnYnvQ3nHYrd7TTQTfhV/rfRwKaxiLo84cUP2ZMNpZcpmvRqCrVYdejtAtgX 3Q2MC/rHc1WFNlL9ZGbN0Nx7ovWyFUI3I3GE/zlGGMLt9PmmTm7R1xKzCMDr/lsxlmdBaV LWkhKPFIZNH5sz25W24QDRRgANj6YPoF7erJaH3TdwJB+46KlFNQGR1n1vsDw33wNc127e VScxAuggocqs7X73vFO3LuzftOaduVKG/duS7IpqXJSsXjz/m80B/VsGdldfEA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZNc5R6Xr2zprB for ; Thu, 27 Mar 2025 08:19:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52R8Jdn6031914 for ; Thu, 27 Mar 2025 08:19:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52R8JdeI031913 for virtualization@FreeBSD.org; Thu, 27 Mar 2025 08:19:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285415] [Hyper-V] i386 panic during vm_phys_early_startup() Date: Thu, 27 Mar 2025 08:19:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285415 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Open |Closed --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 27 08:53:47 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNcrq4KQhz5s2PB for ; Thu, 27 Mar 2025 08:53:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNcrq3jzNz3vJ5 for ; Thu, 27 Mar 2025 08:53:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743065627; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SKu9jVSamQeFcprkryRtESQ7JODpu2+1vMNxU4Gk5YQ=; b=HK89TXkTg6sQ6rzbHl2BZQg0p0LmbrTPQNhJC5HgPNK6L8OfDGzt1oPxnFgRvvDIF6EAH/ k+T3Ko3hWX8lFm98mkVKNjXhBOFdPtKbkYUeBbD5lAZNNgdnnFc6ssBUoGLiwZRr/tJBiB RofPVJRBrA1o6c7bFw3m97GgLvBWcwFSeOvYWO69GO6ffDygaqfhPscHDNr8RcTbCLioPp +Am9tfS9hfixvwddraSqvh0ieZ20ALH8WWGPp59FK+I9ClpqXQw+A+AcHgIEBt/Do4R9ic Yql8aHiwjCU7B1hn320xeBAQXWSA6GV+FXid61E97V6HHqE6CTAa2AcJIDH5SQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743065627; a=rsa-sha256; cv=none; b=GC8GkY3NcKhq0zEy83Nps04ootcCFQEACyVhtKuz61GxHDS6RK3YLvgIpUV/H3qF9sj6CX YzC2UuKrKXAFvOT6dcclBOh9LM5OGu91Yq26IhNlw87Jak/sOgsuarm5FDonoMIk8L7erw E/k9LhZa4JXRrSmcgPFpszPL2K+irXTZqLUxLY5rT+A604eFSDV/lPB869E8kEcV8V8+ey 5Xb5/X655NeqGzq0VmdwKk6qTvZjWk0p5m1YnovQFpiGOUvC9zHJjXypnkDb0qjk8438X7 Bij2RiniihykMRarvIaXAXp/78+Z3oAUdnf/slTbYI1ZGWmuaXHvK+zVGM+gcQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743065627; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SKu9jVSamQeFcprkryRtESQ7JODpu2+1vMNxU4Gk5YQ=; b=mKnGkuEIepSJqlSK4cVRgHezEoLxwkipav2Pchzfzk+MJjvyy4rkv9lwQzBE3FOifOn8fL FYg7jcgz4HEdyk1jOtYlPZSuIBsXBJzm6tulG8SN0UfT2ZzK3Z7lGlu1uC/Cybn8LCvqc5 htWQfujsA788itUNGOMJgb5rdJkQJ0SR3BpA98kCWB8fAxd/yaH7xpTAIk7DqBr8jEbkLT +EuUJyuXlb3JZO81oSIRpPQip3ywu/ic06WpYjeeFFGRbBWHpTI29b4bwIbDH5rjLAPPXc hkPLQcgdX4Ls1vdQQe/dqCMoXFU7mo7iLaRQXmS8eSmdNy2q9HjbLaCQcvwrBQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZNcrq3HNjzq5L for ; Thu, 27 Mar 2025 08:53:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52R8rl2d098646 for ; Thu, 27 Mar 2025 08:53:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52R8rlwm098645 for virtualization@FreeBSD.org; Thu, 27 Mar 2025 08:53:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Thu, 27 Mar 2025 08:53:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #3 from Mark Johnston --- Created attachment 259079 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D259079&action= =3Dedit possible patch How much RAM does the instance have? Does the attached patch help? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 27 10:22:18 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNfqF0SV4z5r7xT for ; Thu, 27 Mar 2025 10:22:33 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNfqD1d6Lz3Tyg for ; Thu, 27 Mar 2025 10:22:32 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=gCxunomh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shamim.shahriar@gmail.com designates 2607:f8b0:4864:20::1136 as permitted sender) smtp.mailfrom=shamim.shahriar@gmail.com Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-6fee50bfea5so7434327b3.1 for ; Thu, 27 Mar 2025 03:22:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743070951; x=1743675751; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=cUiJnMPRZZy9OI38gszjHiFxnVMGIihzoE0nSoowfDc=; b=gCxunomhBoNFGJS+ebwYtL3wHVxT3lWGTgm3jq23d4pA/+V1NlWeRC5ARGo+SBUNE4 9fwduOG0AERKmNec3/5A58M1mYragxgFMtFJIqBDBJapF7CcbuXhYOx4v1bEiIGnYGH3 zRi+p6okhRqJHWQ7iseK1Sjpp61lZVMLjbd6M14Ogl4Hrs1ro1wDffoEKb8wOkbc9uWw ucRUxSNef3uPoMq5moMiR8QUsitMz02a9S138p6IucQOhV1XHqv5V4a/8G0O+xrDUfBX O7sD/NTU4X64cAf1r/SOXfNbBoisPav9zj+4AQR4pXimDw+7cO0+uh6ff8wTKkBSnk2b 2CjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743070951; x=1743675751; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cUiJnMPRZZy9OI38gszjHiFxnVMGIihzoE0nSoowfDc=; b=UyeGzr84cZ38W3qqVgPof5Yu3nWZkm6LEUSyx7EjAemRdjarxQZzEMzBtX+vBHvyoA BlJTN1B5SIm7qcOrHiE8tkxSsuNH3rWJtx8UJw6hHzOpOIAgtuz+ednQX5EJR2KrDzFd XbywUR7P5XxzizHgYqArobNR+64TpA+uqKezOv6YR0EsyIAk8NZbic2vrxYkEHKcvTYQ fghJAtNZTUl3XLQKfoDPFv4P5FUXGAyeSinZkNJ0WybUYSrg48Q1cYPnNaO/EJhw8qlh 3vHZ9XR3Qcn0LZmp/BbgsBqEcXtJYev0UC99PQ0kjagRb0RUxE/B3K3lNjvIZvx4qKqK QS4g== X-Gm-Message-State: AOJu0YyHYsmJvAHsA3hnixf6J3SVG60Gf6EmwwFmNZUnOPdcysRymQsw oC0t+A158F9aKDWU1GcBGCKxtiAn2JSWByrEKAj27IEKS7uZHbc3FmWUZF/wwCM9wHAw5iGwjxc 8Wfc7f6LRhn7vvFuVQouytn4CeQ== X-Gm-Gg: ASbGncsolwq1oKvvL+IZsrdsARmlxeWrE9i3oD0KbMSS4WfgR9gQFUMuPcSFgcJhmX6 p7m4asF5RMm1MxO194y645if6X1Sm//vHyww8GLIszo3OU4e6ylBa0pqDSfIIflr4Ktg23hATtf 7FMCNyFSw+ysP5dZjKJVpJ9w== X-Google-Smtp-Source: AGHT+IFPxjmEHy3rOfyXHGBBXzSBD67yBcKbbORxtBy6Or5eI7qGsBYsUDjLvI0ygMcpi0tPfsWnStDlZWXSc2dIjog= X-Received: by 2002:a05:690c:6906:b0:6f9:7ce7:b107 with SMTP id 00721157ae682-702250c41a1mr37409167b3.29.1743070950787; Thu, 27 Mar 2025 03:22:30 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Shamim Shahriar Date: Thu, 27 Mar 2025 10:22:18 +0000 X-Gm-Features: AQ5f1JoVCKeqT0KlPuv7ysDne3RX1qBtnvNXi0VKnVbcBP2157n2HsLKyDMIqfI Message-ID: Subject: Re: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" To: Alex <9vlc@proton.me> Cc: "freebsd-virtualization@FreeBSD.org" Content-Type: multipart/alternative; boundary="000000000000d71a8d0631505401" X-Spamd-Result: default: False [-3.41 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.97)[-0.969]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_LONG(-0.44)[-0.443]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TAGGED_FROM(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1136:from] X-Rspamd-Queue-Id: 4ZNfqD1d6Lz3Tyg X-Spamd-Bar: --- --000000000000d71a8d0631505401 Content-Type: text/plain; charset="UTF-8" Good morning Alex, and thank you for the detailed help, it is very much appreciated. For some reason your email went into google's spam box, I just noticed it this morning. I'll try this as well and keep the list posted as to how it goes. Best regards SS On Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote: > This is a hefty help email for Shamim Shahriar in regards to passing > through Nvidia GPU to a bhyve VM. > > Hi. > I have recently gotten this mailing list message forwarded to me by > Michael Dexter > https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html > and it alone made me sign up here. > > So, here's a big mail containing a bunch of useful GPU passthrough info! > > To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, > let me help you out make it work correctly! > > Parts and short description: > > 1. Pass through the GPU audio and other devices (if exist) > Modern GPUs usually create two PCI devices that you can see in > pciconf - VGA and audio. They come on the same PCI function (i think it is > called that), for example GPU 19/0/0 will also have 19/0/1 for audio and > may have 19/0/2 for usb. > > 2. Give the GPU a VGA BIOS as PCI Option ROM > GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a > trimmed VGA BIOS since the PCI option rom that is used by the VM is nested > inside the ROM file. > (from now on i will refer to the vga bios as vbios) > > 3. Possibly, re-BAR in BIOS? > Some UEFI firmware have a feature called re-BAR and "above 4G > decoding / mining", it makes some GPUs in some configurations not work in > passthru. > > 5. Possibly, Try newer EDK2 > The edk2-bhyve package (provides UEFI firmware for bhyve) on the > repos is still using the 2023 release. You can compile a 2025 version with > my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 > > Full parts and HOWTOs: > > !!!! MISC GPU DEVICES > > As decribed before, pass through all the devices that your GPU creates in > pciconf right after the GPU device on the same PCI function. > All audio, USB, etc. devices belonging to the GPU should be passed through > with it since they are a part of it, else the passthru will NOT work > correctly. > If that is not the case and there is only a single GPU VGA device with > nothing else Nvidia to be found, skip this step. > > !!!! VBIOS > > You first need to get the pure vBIOS file. To do that you can either: > 1. Use a Linux system and dump it with > https://github.com/ilya-zlobintsev/LACT > 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz > 3. Download one that someone has dumped from their GPU using GPU-z from > https://www.techpowerup.com/vgabios > > If you don't want to install a different OS on the PC you're currently > running, we can just find and download a correct vBIOS from Techpowerup. > But we stumble across an issue! You specified "NVidia Quadro RTX > 6000/8000", so which GPU is it really? Those are two different GPUs with > different specifications! > Thankfully, we have your pciconf output which says "subdevice=0x13d8" and > if we refer to Nvidias documentation, on page 7 over here: > > https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf > we can see that it is a RTX 8000! > Here is an unverified but most likely working vBIOS: > https://www.techpowerup.com/vgabios/256271/256271 > > Now after we've gotten the vBIOS, we need to trim the start of the file to > get to the embedded PCI Option ROM. > For that we will use my tool "trim_nvidia_vbios" from > https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios > Compile and run the resulting executable with the first argument being the > downloaded vBIOS and the second argument being an output PCI OpROM. Example: > $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c > $ ./trim.elf techpowerup.rom trimmed-vbios.rom > > And we come upon a road block: vm-bhyve does not support passing a PCI > Option ROMs in "passthru=", so we're going to use "bhyve_options=" instead > to define the GPU VGA device: > bhyve_options="-s 6:0,passthru,19/0/0,rom=" > Other devices that do not require a PCI Option ROM can be added with > "passthru=". These devices include the rest of the GPU pci functions like > audio and usb. > > !!!! RE-BAR > > If after everything you do get the GPU to work in the VM at least in the > edk2 firmware, > Go to your regular UEFI BIOS setup utility and search for options titled > "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for > Mining". > If you do see re-BAR but cannot disable it, you could try using > https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI > settings. Be warned though, smokeless is a heck of an unstable MESS. > > !!!! UPDATED EDK2 > > As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some > rare cases, that might cause issues. > I have submitted a package update bug report for sysutils/edk2 to FreeBSD > bugzilla over here: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 > You can download the patch, apply it to your ports tree, then compile and > install the edk2 package with "FLAVOR=bhyve" to get the fresh UEFI firmware. > > ~~~~ > > Hope I didn't forget anything! > I wish you success in the GPU passthrough journey!! > - Alexey > > > --000000000000d71a8d0631505401 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Good morning Alex, and thank you for the detailed help, it= is very much appreciated.

For some reason your email we= nt into google's spam box, I just noticed it this morning. I'll try= this as well and keep the list posted as to how it goes.

Best regards
SS

On Tue, 25 Mar= 2025 at 13:27, Alex <9vlc@proton.me> wrote:
Th= is is a hefty help email for Shamim Shahriar in regards to passing through = Nvidia GPU to a bhyve VM.

Hi.
I have recently gotten this mailing list message forwarded to me by Michael= Dexter https://list= s.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html an= d it alone made me sign up here.

So, here's a big mail containing a bunch of useful GPU passthrough info= !

To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, l= et me help you out make it work correctly!

Parts and short description:

1. Pass through the GPU audio and other devices (if exist)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Modern GPUs usually create two PCI devices that= you can see in pciconf - VGA and audio. They come on the same PCI function= (i think it is called that), for example GPU 19/0/0 will also have 19/0/1 = for audio and may have 19/0/2 for usb.

2. Give the GPU a VGA BIOS as PCI Option ROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GPUs need a VGA BIOS to work. In addition, Nvid= ia GPUs need a trimmed VGA BIOS since the PCI option rom that is used by th= e VM is nested inside the ROM file.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (from now on i will refer to the vga bios as vb= ios)

3. Possibly, re-BAR in BIOS?
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Some UEFI firmware have a feature called re-BAR= and "above 4G decoding / mining", it makes some GPUs in some con= figurations not work in passthru.

5. Possibly, Try newer EDK2
=C2=A0 =C2=A0 =C2=A0 =C2=A0 The edk2-bhyve package (provides UEFI firmware = for bhyve) on the repos is still using the 2023 release. You can compile a = 2025 version with my patch: https://bugs.f= reebsd.org/bugzilla/show_bug.cgi?id=3D285554

Full parts and HOWTOs:

!!!! MISC GPU DEVICES

As decribed before, pass through all the devices that your GPU creates in p= ciconf right after the GPU device on the same PCI function.
All audio, USB, etc. devices belonging to the GPU should be passed through = with it since they are a part of it, else the passthru will NOT work correc= tly.
If that is not the case and there is only a single GPU VGA device with noth= ing else Nvidia to be found, skip this step.

!!!! VBIOS

You first need to get the pure vBIOS file. To do that you can either:
1. Use a Linux system and dump it with https://github.com/ily= a-zlobintsev/LACT
2. Use a Windows system and dump it with https://www.techpowerup.com= /gpuz
3. Download one that someone has dumped from their GPU using GPU-z from https://www.techpowerup.com/vgabios

If you don't want to install a different OS on the PC you're curren= tly running, we can just find and download a correct vBIOS from Techpowerup= .
But we stumble across an issue! You specified "NVidia Quadro RTX 6000/= 8000", so which GPU is it really? Those are two different GPUs with di= fferent specifications!
Thankfully, we have your pciconf output which says "subdevice=3D0x13d8= " and if we refer to Nvidias documentation, on page 7 over here:
https://www.nvidia.c= om/content/dam/en-zz/Solutions/design-visualization/quadro-product-literatu= re/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf
we can see that it is a RTX 8000!
Here is an unverified but most likely working vBIOS: https://www.techpowerup.com/vgabios/256271/256271

Now after we've gotten the vBIOS, we need to trim the start of the file= to get to the embedded PCI Option ROM.
For that we will use my tool "trim_nvidia_vbios" from https= ://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios
Compile and run the resulting executable with the first argument being the = downloaded vBIOS and the second argument being an output PCI OpROM. Example= :
$ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c
$ ./trim.elf techpowerup.rom trimmed-vbios.rom

And we come upon a road block: vm-bhyve does not support passing a PCI Opti= on ROMs in "passthru=3D", so we're going to use "bhyve_o= ptions=3D" instead to define the GPU VGA device:
bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D</path/to/trimmed-vbi= os.rom>"
Other devices that do not require a PCI Option ROM can be added with "= passthru=3D". These devices include the rest of the GPU pci functions = like audio and usb.

!!!! RE-BAR

If after everything you do get the GPU to work in the VM at least in the ed= k2 firmware,
Go to your regular UEFI BIOS setup utility and search for options titled &q= uot;BAR", "PCI Re-size BAR", "re-BAR", "Above= 4G Encoding" or "Above 4G for Mining".
If you do see re-BAR but cannot disable it, you could try using https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI = settings. Be warned though, smokeless is a heck of an unstable MESS.

!!!! UPDATED EDK2

As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some ra= re cases, that might cause issues.
I have submitted a package update bug report for sysutils/edk2 to FreeBSD b= ugzilla over here: https://bugs.freebsd.or= g/bugzilla/show_bug.cgi?id=3D285554
You can download the patch, apply it to your ports tree, then compile and i= nstall the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UE= FI firmware.

~~~~

Hope I didn't forget anything!
I wish you success in the GPU passthrough journey!!
- Alexey


--000000000000d71a8d0631505401-- From nobody Thu Mar 27 10:37:26 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNg986nL3z5r904 for ; Thu, 27 Mar 2025 10:38:04 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNg981Sd1z3ZW2 for ; Thu, 27 Mar 2025 10:38:04 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=C+PQm+5W; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::62a as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-227d6b530d8so15565375ad.3 for ; Thu, 27 Mar 2025 03:38:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743071883; x=1743676683; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jBoOVGqT6Ip57UzpxL5s7JLChKH43ndtcOfzqI57IIo=; b=C+PQm+5WsUvia+pzkthuKfILgeMHtbSy/QoDEq/A/aT4IpWnrfYb5SBQG8a05pJtA+ KqY9mRKep3BVaYSPcoQEJg7xngAvIPSCmg9uVuRZh6VYfiuYWkiAc3OFS1Xx2wJU64CW zG2CeuLeRUveLKTvJX+OGAnRInHmEWu1Ndo24rtgGZYipnG21QlOl39eDuCR6W4Yc3ik mjlSfuB1DQPLfSd5M/SIpFuLPvGg1LhPVCOqGeO1nghi0baPy+LPi6JQ9RRXqA3I3DdK Fg6P6AFt17Wxjt+OyPofgmCMo+jlDOSYUqtLW2U03wtgvsHJHwpfish5zUirQP/nD/2N 6mIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743071883; x=1743676683; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jBoOVGqT6Ip57UzpxL5s7JLChKH43ndtcOfzqI57IIo=; b=hNAHbkuuV3/WIzBFk3JGk2FMERPpm3WF2ZX1TAiTYWJ2mheX5ts5hCrPlcvZDZsq8P asizWOSk1cOHiDqQAW33TahM/Ojdw8wwYLijOWIwNCYO7kAC0MObiY8lSLEjPeKmTlN4 E407zfSsTP92BemE+lrq8oos3DNLBwbpN1T2i91Lbv15aA1FhPROLWU1AY23M0C0a/Ah 3DfQKziwe8wT07F7gDOLiadp/2o34dxAMX3kMh03MjDd3GtrQe9ne6DY4vHU89pKLlRu LeLuicc5hdHkpysNT7isSXV1GM3Qo2QWd1l1iu6yvSSMr4eCOZ2nPgm5RvZlIyUNMj6i FNRw== X-Forwarded-Encrypted: i=1; AJvYcCXErkKKAdqzWMFxsfRGtgXnWnoJLwA+nDokkm1ucLEyrS2WQvZPQ9s+WEEceoGmLCelxkYiqSRf2g1wiGbv8SpDZpPbslxf@freebsd.org X-Gm-Message-State: AOJu0Ywi+/PNdkCQKQ19tlxpIAKpiqczIS0LeWBrYVjQZ1mzI1604BqM JwmXu/VSs/3dySAkzzUTbNTRBLdgV4UHQG/IGjQzy1C021uKcTGkdVXeoio2rt1Ddm+YJjy04P/ 2o2dXdrcR14kw50eMypxal0aX+qL339mb X-Gm-Gg: ASbGncsd76vqjXUDfM8tpXCaQRH8jBlEugm527mBOn+Uuf7j4FkyMJeqpqd3n+am8f5 LcJ7nlVIv218YSDzDxMwiVI7tAB49eT7cZ1VMAV6W8q9VWt16TPBq0bGRzxt+LvjAVFRvflBTgg nI1Oy61DYbaNBkfho5uk164TpUI4k= X-Google-Smtp-Source: AGHT+IEmC4c5ERv0jTpnKSIwFwsdyl7uGd7x/wL9txPipcH4tsdz8lGFXT5Kx9UuljlBLKsR8Mk1lcYgVNHLLmDNkTk= X-Received: by 2002:a17:90b:2f4d:b0:2fe:8a84:e033 with SMTP id 98e67ed59e1d1-303a7b5a137mr4820698a91.2.1743071882581; Thu, 27 Mar 2025 03:38:02 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Thu, 27 Mar 2025 11:37:26 +0100 X-Gm-Features: AQ5f1JpRm8Jn8y2TQs_gqKT4U7F-XkuiIOrU0wmUqZKHQf127WShs3Nqq8GXGco Message-ID: Subject: Re: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" To: Shamim Shahriar Cc: Alex <9vlc@proton.me>, "freebsd-virtualization@FreeBSD.org" Content-Type: multipart/alternative; boundary="0000000000006123850631508c90" X-Spamd-Result: default: False [-3.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.99)[-0.988]; NEURAL_HAM_LONG(-0.81)[-0.812]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::62a:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4ZNg981Sd1z3ZW2 X-Spamd-Bar: --- --0000000000006123850631508c90 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Shamim. Without applying the patches that I sent to you will not be able to pass through your nVidia Gpu. Probably your problem is due to this. You can try what you want,but the risk is that you go nowhere. If I were you,I would try first to apply the patches of Corvin. On Thu, Mar 27, 2025 at 11:22=E2=80=AFAM Shamim Shahriar wrote: > Good morning Alex, and thank you for the detailed help, it is very much > appreciated. > > For some reason your email went into google's spam box, I just noticed it > this morning. I'll try this as well and keep the list posted as to how it > goes. > > Best regards > SS > > On Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote: > >> This is a hefty help email for Shamim Shahriar in regards to passing >> through Nvidia GPU to a bhyve VM. >> >> Hi. >> I have recently gotten this mailing list message forwarded to me by >> Michael Dexter >> https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002= 818.html >> and it alone made me sign up here. >> >> So, here's a big mail containing a bunch of useful GPU passthrough info! >> >> To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config= , >> let me help you out make it work correctly! >> >> Parts and short description: >> >> 1. Pass through the GPU audio and other devices (if exist) >> Modern GPUs usually create two PCI devices that you can see in >> pciconf - VGA and audio. They come on the same PCI function (i think it = is >> called that), for example GPU 19/0/0 will also have 19/0/1 for audio and >> may have 19/0/2 for usb. >> >> 2. Give the GPU a VGA BIOS as PCI Option ROM >> GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a >> trimmed VGA BIOS since the PCI option rom that is used by the VM is nest= ed >> inside the ROM file. >> (from now on i will refer to the vga bios as vbios) >> >> 3. Possibly, re-BAR in BIOS? >> Some UEFI firmware have a feature called re-BAR and "above 4G >> decoding / mining", it makes some GPUs in some configurations not work i= n >> passthru. >> >> 5. Possibly, Try newer EDK2 >> The edk2-bhyve package (provides UEFI firmware for bhyve) on the >> repos is still using the 2023 release. You can compile a 2025 version wi= th >> my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285554 >> >> Full parts and HOWTOs: >> >> !!!! MISC GPU DEVICES >> >> As decribed before, pass through all the devices that your GPU creates i= n >> pciconf right after the GPU device on the same PCI function. >> All audio, USB, etc. devices belonging to the GPU should be passed >> through with it since they are a part of it, else the passthru will NOT >> work correctly. >> If that is not the case and there is only a single GPU VGA device with >> nothing else Nvidia to be found, skip this step. >> >> !!!! VBIOS >> >> You first need to get the pure vBIOS file. To do that you can either: >> 1. Use a Linux system and dump it with >> https://github.com/ilya-zlobintsev/LACT >> 2. Use a Windows system and dump it with https://www.techpowerup.com/gpu= z >> 3. Download one that someone has dumped from their GPU using GPU-z from >> https://www.techpowerup.com/vgabios >> >> If you don't want to install a different OS on the PC you're currently >> running, we can just find and download a correct vBIOS from Techpowerup. >> But we stumble across an issue! You specified "NVidia Quadro RTX >> 6000/8000", so which GPU is it really? Those are two different GPUs with >> different specifications! >> Thankfully, we have your pciconf output which says "subdevice=3D0x13d8" = and >> if we refer to Nvidias documentation, on page 7 over here: >> >> https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/= quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-= 1219.pdf >> we can see that it is a RTX 8000! >> Here is an unverified but most likely working vBIOS: >> https://www.techpowerup.com/vgabios/256271/256271 >> >> Now after we've gotten the vBIOS, we need to trim the start of the file >> to get to the embedded PCI Option ROM. >> For that we will use my tool "trim_nvidia_vbios" from >> https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios >> Compile and run the resulting executable with the first argument being >> the downloaded vBIOS and the second argument being an output PCI OpROM. >> Example: >> $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c >> $ ./trim.elf techpowerup.rom trimmed-vbios.rom >> >> And we come upon a road block: vm-bhyve does not support passing a PCI >> Option ROMs in "passthru=3D", so we're going to use "bhyve_options=3D" i= nstead >> to define the GPU VGA device: >> bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D" >> Other devices that do not require a PCI Option ROM can be added with >> "passthru=3D". These devices include the rest of the GPU pci functions l= ike >> audio and usb. >> >> !!!! RE-BAR >> >> If after everything you do get the GPU to work in the VM at least in the >> edk2 firmware, >> Go to your regular UEFI BIOS setup utility and search for options titled >> "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for >> Mining". >> If you do see re-BAR but cannot disable it, you could try using >> https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI >> settings. Be warned though, smokeless is a heck of an unstable MESS. >> >> !!!! UPDATED EDK2 >> >> As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some >> rare cases, that might cause issues. >> I have submitted a package update bug report for sysutils/edk2 to FreeBS= D >> bugzilla over here: >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285554 >> You can download the patch, apply it to your ports tree, then compile an= d >> install the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UEFI fir= mware. >> >> ~~~~ >> >> Hope I didn't forget anything! >> I wish you success in the GPU passthrough journey!! >> - Alexey >> >> >> --=20 Mario. --0000000000006123850631508c90 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Shamim.

Without applyi= ng the patches that I sent to you will not be able to pass through your nVi= dia Gpu. Probably your problem is due to this. You can try what you want,bu= t the risk is that you go nowhere. If I were you,I would try first to apply= the patches of Corvin.

On Thu, Mar 27, 2025 at= 11:22=E2=80=AFAM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Good morning Alex, and th= ank you for the detailed help, it is very much appreciated.

<= div>For some reason your email went into google's spam box, I just noti= ced it this morning. I'll try this as well and keep the list posted as = to how it goes.

Best regards
SS

On = Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote:
This is a hefty help email for Shamim Shahriar i= n regards to passing through Nvidia GPU to a bhyve VM.

Hi.
I have recently gotten this mailing list message forwarded to me by Michael= Dexter https://list= s.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html an= d it alone made me sign up here.

So, here's a big mail containing a bunch of useful GPU passthrough info= !

To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, l= et me help you out make it work correctly!

Parts and short description:

1. Pass through the GPU audio and other devices (if exist)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Modern GPUs usually create two PCI devices that= you can see in pciconf - VGA and audio. They come on the same PCI function= (i think it is called that), for example GPU 19/0/0 will also have 19/0/1 = for audio and may have 19/0/2 for usb.

2. Give the GPU a VGA BIOS as PCI Option ROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GPUs need a VGA BIOS to work. In addition, Nvid= ia GPUs need a trimmed VGA BIOS since the PCI option rom that is used by th= e VM is nested inside the ROM file.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (from now on i will refer to the vga bios as vb= ios)

3. Possibly, re-BAR in BIOS?
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Some UEFI firmware have a feature called re-BAR= and "above 4G decoding / mining", it makes some GPUs in some con= figurations not work in passthru.

5. Possibly, Try newer EDK2
=C2=A0 =C2=A0 =C2=A0 =C2=A0 The edk2-bhyve package (provides UEFI firmware = for bhyve) on the repos is still using the 2023 release. You can compile a = 2025 version with my patch: https://bugs.f= reebsd.org/bugzilla/show_bug.cgi?id=3D285554

Full parts and HOWTOs:

!!!! MISC GPU DEVICES

As decribed before, pass through all the devices that your GPU creates in p= ciconf right after the GPU device on the same PCI function.
All audio, USB, etc. devices belonging to the GPU should be passed through = with it since they are a part of it, else the passthru will NOT work correc= tly.
If that is not the case and there is only a single GPU VGA device with noth= ing else Nvidia to be found, skip this step.

!!!! VBIOS

You first need to get the pure vBIOS file. To do that you can either:
1. Use a Linux system and dump it with https://github.com/ily= a-zlobintsev/LACT
2. Use a Windows system and dump it with https://www.techpowerup.com= /gpuz
3. Download one that someone has dumped from their GPU using GPU-z from https://www.techpowerup.com/vgabios

If you don't want to install a different OS on the PC you're curren= tly running, we can just find and download a correct vBIOS from Techpowerup= .
But we stumble across an issue! You specified "NVidia Quadro RTX 6000/= 8000", so which GPU is it really? Those are two different GPUs with di= fferent specifications!
Thankfully, we have your pciconf output which says "subdevice=3D0x13d8= " and if we refer to Nvidias documentation, on page 7 over here:
https://www.nvidia.c= om/content/dam/en-zz/Solutions/design-visualization/quadro-product-literatu= re/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf
we can see that it is a RTX 8000!
Here is an unverified but most likely working vBIOS: https://www.techpowerup.com/vgabios/256271/256271

Now after we've gotten the vBIOS, we need to trim the start of the file= to get to the embedded PCI Option ROM.
For that we will use my tool "trim_nvidia_vbios" from https= ://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios
Compile and run the resulting executable with the first argument being the = downloaded vBIOS and the second argument being an output PCI OpROM. Example= :
$ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c
$ ./trim.elf techpowerup.rom trimmed-vbios.rom

And we come upon a road block: vm-bhyve does not support passing a PCI Opti= on ROMs in "passthru=3D", so we're going to use "bhyve_o= ptions=3D" instead to define the GPU VGA device:
bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D</path/to/trimmed-vbi= os.rom>"
Other devices that do not require a PCI Option ROM can be added with "= passthru=3D". These devices include the rest of the GPU pci functions = like audio and usb.

!!!! RE-BAR

If after everything you do get the GPU to work in the VM at least in the ed= k2 firmware,
Go to your regular UEFI BIOS setup utility and search for options titled &q= uot;BAR", "PCI Re-size BAR", "re-BAR", "Above= 4G Encoding" or "Above 4G for Mining".
If you do see re-BAR but cannot disable it, you could try using https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI = settings. Be warned though, smokeless is a heck of an unstable MESS.

!!!! UPDATED EDK2

As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some ra= re cases, that might cause issues.
I have submitted a package update bug report for sysutils/edk2 to FreeBSD b= ugzilla over here: https://bugs.freebsd.or= g/bugzilla/show_bug.cgi?id=3D285554
You can download the patch, apply it to your ports tree, then compile and i= nstall the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UE= FI firmware.

~~~~

Hope I didn't forget anything!
I wish you success in the GPU passthrough journey!!
- Alexey




--
Ma= rio.
--0000000000006123850631508c90-- From nobody Thu Mar 27 13:48:34 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNlP01fqRz5rP5t for ; Thu, 27 Mar 2025 13:48:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNlNy5m6hz3NsK for ; Thu, 27 Mar 2025 13:48:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743083314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0hakIdCGFnBeYQlyYixXfNAp/d9PNBkkmhMvp5/QXdM=; b=k6UNEQQp+wAqTzuqVNrWrk4DjFl7sixLy68uyiKPdZIJkuCS0meR9aTO1e9Cmh7Fthiebf FttOAcf6K/JqiBBDmt1n8pc2iZEYd2MIYfAn2ZQITlrDzb067ZYXDWmSTVjECGDfV2W1/A T9bLId7EZS4mwnIdx1sO3jp0y4nW1kiKzIdLUa56db44AJ331os+rgXPsaSGjdBaWcXNrf gWKFd/grIHPloc/fyXr7VPj1Uk9/wuEhkRaq7V8hZ+CbVcPZWNSU8C90+NrjzEKkq1zrll NIwx+nGUn10fHP+Tq57gCXnsxLdWd2kPYZ0SUCyqmBUhpNTTP1OlbZ6tCP4hNQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743083314; a=rsa-sha256; cv=none; b=bSDwmQH8wrAWdZETNS5XBS/PUfAzEKuVhN89h249sUT/VD/dQBZWSl8OWTtXM8Byn14vxV IR0iZd/uDqb5uaTTMiha6h9NLhwcXL8NHg63kBrv+zFBvGGcKYLBzsewQpLF4b0rgMNng9 YUUrOnmFiX+3ls/uCiNvqbxMj/oyssgme8WmVZMpFdU7T2mtCP9M8CkNLoevyM371L99cQ D5bpeZImWa4SoyqrTUs2gfevgWpVpcTIolRPM1BPfCvT4ZMZLlaCRQMeFu/V6Jl0trFdUt Hcasm+9Coyh9RPl1iAjuNuDFUbQ7WP/ngZ1Zo7Q8eIbxMlffPQiJuIBSn41O3g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743083314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0hakIdCGFnBeYQlyYixXfNAp/d9PNBkkmhMvp5/QXdM=; b=lBtfPGIXoRj3+ugl1kE3GweqF7mhIFQWARflfyZzWjDAcBImkJN0aU7tV6MGLJ2AlmXfws Qk+iVQWCKq+VI+d6LxXpagIcdmqjxdNCOkEpkNBijbVSC4AW6qeUOSGFqN9O9Bt4hWjwD/ 7gVcBiGJ3TBvbaY18rsYiGifOAVEOMd6C0gVD0ub8qCAzuZAYX2p3Bv+rdN1xZU0upcZrp AtfFZK8bUzzxbctYVo0RS/x1BgGlAEmKgdGPdnt2KmmyahdRaK5w2iXpZTDbMc131VAMDA p6xFHgQqeteQdvMKalCivlwJ8oQE+U1/jdu9DkEuRtiOz7UqEpwBrZZILu2fPA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZNlNy5MlzzyrC for ; Thu, 27 Mar 2025 13:48:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52RDmYwO057165 for ; Thu, 27 Mar 2025 13:48:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52RDmYIZ057160 for virtualization@FreeBSD.org; Thu, 27 Mar 2025 13:48:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285051] [bhyve] ampere emag freezes after kldload vmm, reboot needed not even a panic Date: Thu, 27 Mar 2025 13:48:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285051 --- Comment #2 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D872fc1b8de1ac2a10465a315457cd69b4= 268cbe9 commit 872fc1b8de1ac2a10465a315457cd69b4268cbe9 Author: Andrew Turner AuthorDate: 2025-03-27 13:17:30 +0000 Commit: Andrew Turner CommitDate: 2025-03-27 13:47:37 +0000 arm64/vmm: Don't load on Ampere eMAG Loading vmm.ko on Ampere eMAG locks up the system. Block loading for now until it can be fixed. PR: 285051 Reviewed by: tuexen Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D49155 sys/arm64/vmm/vmm.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 27 18:05:52 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNs5r3TFFz5rhqT for ; Thu, 27 Mar 2025 18:05:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNs5r30nfz3Yxs for ; Thu, 27 Mar 2025 18:05:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743098752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8kj4aN22dP75Mt4vaWVR48PsgRdN21mammtabitEtU=; b=sVDB2tg4B7PR7Hd/HwyekHVw7VP1tE8gv/9OCjB3q3gThixF6FGe/WCg5mimAv4IyVOFcv V4jz+2tlmZ+6t+KU1NluWidY4MC8Cs0+NL1HZyUZIjZD6Yevm7XVwBUE19pNjmvmgR0htq 8yWywns6GRsJlJy8dtxwj3hmRNIbx89BNUfOUm7SSt9oEZIR82WowsWI0lku6RJ83I7osu 28Ayemo4UyceJWXAMkXJzl/EjFLeYBfMBmYEpM3CHEchgY3PrxKLhp8hr5+kMp59bMyL1/ OuDbVl9/CNVmDOrr3HIULeKrpfEppYOGA+YWDydL/lbWQTL1cPIxTxakf0kERQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743098752; a=rsa-sha256; cv=none; b=UcEphHJmSWJnpSTeOLy6RwS2kHcnPyXUEit7ZdlFx96fNhmxLY9bO5BvA8qv7AOCZwfPL/ gLn+m6si1HoS8uB5bKxKl2K+5FaDzoIHswdISqLzFBUROt0GvF+SrDxdXngaRlL1EWhutd IM9qls442EMsjm+iwP0epw9LeDYruQyP6EklBble/ZQ6faz2OmXKXBiqWh/C/QjzQIChsl lX8NhnA7XG1MtkDEb+5dBVyy2lkwq3o6okVvKEsONuEtg/w6yi0yXtyHVXmMCfQzhGACRk T9YmWv1h3kH+05KqnGFwj2XDNTmjw/wfw+WVbp6XqMMuuiSFa0/WPlC6YI+H4A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743098752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8kj4aN22dP75Mt4vaWVR48PsgRdN21mammtabitEtU=; b=e9YxOYrBltWT5Zff8VmJfQf33rw2GldO7W2rabP6S4iP53DAYyIkiRn54uRZEBMHvMe97S 4ZN1T+SirRoAXJlSgQLYrCvY8X1gAV9i0kZY0/QBVq9trjZm2YMw3Y9R13oyqWaiqu9jk0 S2yVh9uHkDR1mWaCZed6gpxYyP3N6jHsqO+GJPFzjGo5bHdmu/d3YE0xy/NbuoijMdM72K +9GDtiokpPIsMLEGQzw9Su67EhEv6pFRke6VwP0Svf8vsivnwXKpCQj05QbhT+2CLq3HO/ 8Aa8RQsuWPyQ8xoprhq12wPg+/8a2n/xZV8QaNvYKkpSY4kYR1U18TR8lmxZUg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZNs5r28yNz16XJ for ; Thu, 27 Mar 2025 18:05:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52RI5qcj049407 for ; Thu, 27 Mar 2025 18:05:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52RI5qWQ049406 for virtualization@FreeBSD.org; Thu, 27 Mar 2025 18:05:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Thu, 27 Mar 2025 18:05:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #4 from Dimitry Andric --- (In reply to Mark Johnston from comment #3) I started initially with 3G, to keep below a 4G barrier. But since i386 see= ms to have PAE I increased it to 8G. But I can configure it in any way you wou= ld need. Interestingly the patch seems to fix the issue! It boots to multi-user just fine now, and survives a "zpool scrub zroot". --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 28 08:50:14 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZPDkG3RPbz5rW9x for ; Fri, 28 Mar 2025 08:50:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZPDkG2rmyz3NMd for ; Fri, 28 Mar 2025 08:50:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743151814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RvnJ/B9Bz3zpQHhnXO6+fmbPhFMfcNoXq35U8NooaJo=; b=MaDVO6r/xTrJ2GCP/3PqOuzChVTEe0/AblIjcDZEnN+txGDoJOSlTKWDOd2fGHuOavUSWR SjntLDE+EaLzaS2akWMJw9iBBuMTlqdLjh+bZTaHCPSekHe9dDw/uah2NsYCklcDKZK026 oodRD3U3kaIDhHuh1chcI4oTLovL5rCluw8/cFY3OUH6GOglIMB9h03W5Jkdz6+zwZX2eh Av+EKMmptjA1mvA90eBDNCuZLnK4iWT+RNC6LpzStuDCpb4sEkbe6IDQZVxgTN6wDUFcbM Uo0LphB8AjYBwyOu0Gs//o6DLYg0OH3fe8Vt+w+xV80bSNZ10MqoG4yPBd5dwg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743151814; a=rsa-sha256; cv=none; b=r9NWwArCHUFj4VMHSseWxGpN5ajpWH5k2/i5Qhx/4cfx73XUyzmJ/br7VXa9TYzy+oW/eF ZJbIoJMKjQp9bYSiVL9u4WtDklQ2vyF0IvZ//ZFHd9sSebnBIDKWI+w6A5muzNcNc/UKO+ ma2HYlpNT5eP0C1TuFDG++SjAZGpEXbGRVCSzEDjWX/oOVbgr+fgdN6xxqt6POSCaSg4pq 0D6hfV09mvjh32yAbFsn/QGhpEG+Vq9chGNN2MPbozWM7h2WKdTMmmTe7mkGWpzm+C2jpo 3oY7KHokTwX2tM0P+cKHCz2APMJFWKaZLZ/rf08PjcFobU3KG2LnME0jDBMZHg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743151814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RvnJ/B9Bz3zpQHhnXO6+fmbPhFMfcNoXq35U8NooaJo=; b=IDlYivQuxEaderjSGG36J5vFDxrnmdhOJOlAKKcQ4tmBi5som2zTnBdpmnvLcd+lKqMbG1 vvfJhjrFVwU2NqQhlTNE8aroK2cNFFL5qpHsta3C2gkz6nxdXjbrYR6xl1RvXyfcQInaPY 5BetoqvcDBYWJOHMN5emGKztE+DLkOgC+lWuZucIK+ca/fmGL7gNoqIYlyy1Pll0I4eX03 O/m9ad63SMoOoMSbfdtZk+3ub2uV79KF/ii6H2uJnCVGbZ2XKjC+xUvtpbLRu4cWNInBbv ZHj00JKoUa4vGzPVnco9mYYrqAy90Flkl5m0u5R/TcFbv8NxQtgZIwSQmBnfEg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZPDkG2Q5XzcVy for ; Fri, 28 Mar 2025 08:50:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52S8oE0f053750 for ; Fri, 28 Mar 2025 08:50:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52S8oE7Y053748 for virtualization@FreeBSD.org; Fri, 28 Mar 2025 08:50:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Fri, 28 Mar 2025 08:50:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open CC| |whu@FreeBSD.org --- Comment #5 from Mark Johnston --- (In reply to Dimitry Andric from comment #4) I'm wondering if the panic is reproducible with less than 4GB of RAM. In particular, I guess we're bouncing the I/O pages, and since the driver does= n't keep track of the 1st page offset, this flag fixes the problem. Could you please try, say, 3GB of RAM, and without the patch? In particular I note t= hat the page address passed by storvsc_xferbuf_prepare() is a uint64_t, so mayb= e we can avoid bouncing entirely on i386. If that works, it still doesn't explain why we need to disable the assertio= n on arm64. Can we try the same test from comment 2 on arm64? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 28 13:48:20 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZPMLD0lrSz5rsWP for ; Fri, 28 Mar 2025 13:48:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZPMLC6ZlBz3YDy for ; Fri, 28 Mar 2025 13:48:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743169699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f7par/1/CJEjcCsyiXClA62LrQackU/z9956FVqjgpw=; b=k2NLYrYWn/jYsFJCWrTKwAxTUbV+dcSktknRzhzos1DypzeJnd6H9c//FJ17xSymDTbDUA h1qQ5j1VaRUMTUFYTSa7WCXeYAm0ntKymBZNXokWGGalxP2eovdvOM5kqD09cz1FzWgfq2 lihkuSSUb7XMQErEnrWpfo2qEW0pu7O8KVQlZmc8NbkgtA94BvgNfa5CMYhT1Nt9Etdn+2 glRBem/EoixRHk8vfMJdQK/y3igEPsnSImkWtbEGZCEWITaOeKDvnufQC9fU21CfEs8PKx Qw7xd8a9aJMid+iL19aznqK/Uis5/yRj+RgEV1mXgtrlgKjp2osDc0igj4VmJw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743169699; a=rsa-sha256; cv=none; b=YleyhKciOoQSnWIX7ZnjVYoY2nDNn0+KEXeqojcrLvq5jll9I0ZN0TQd63uV7JnQ1ZpS0r NzXZi7+2z1Bp2b8KSUoYbIFUbpukLz/ZoBrkPezXR3SGW+wS3DcW8obnjccm8o+r5m/Nl/ Vv9Zr5a+3SC2XUlCg/nNUPHlghO0WG5ZzU3wnIDmac1ZnQESyAHRi6uAlUIsQ6x84opRpz Mv93w55VjjrdiltVLnrYlxURioOHbH4qZunV0BZL5JwuhMFs3RgZgHZSEkhMF8ko2HKU93 4w585tJV6SXC1Sz/P+MZpt9iuDLtpe10aLJ1IoQlmSUJ+rA8Pkqxm4mK+enthw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743169699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f7par/1/CJEjcCsyiXClA62LrQackU/z9956FVqjgpw=; b=YF039sybuYbemtYLfzE1HXDmb2rraypo8LsX7iapdp+A7u4Ci7vgf/9RX0czijaCaV2G8O Y5Zvb7FHaChZQnFuLwPzF+D0XkFIDVwP7de07sNV/HThv3JxqT6y4d90msd1OBRzd3HIwP D3JxotvwTGJBXpilM5V6AeALYFXq1cGuzI4F1v4rQpUGIjVE5Rga06lX93zd9Bs0xjEYBA IT4/0+5zqOTiBOafzcYpQjaYhPNjHNi1Kwfafrd4vg0/pApA/+hB8kMnM62mCzLyT/GVpH q7pfI4htnJEqLhigt+PxBzm9vdGHokSw8jEcioJ0WzI6a8qtR4/N+DBpS1ZSZQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZPMLC6BGdzm1X for ; Fri, 28 Mar 2025 13:48:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52SDmJqb017068 for ; Fri, 28 Mar 2025 13:48:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52SDmJuO017067 for virtualization@FreeBSD.org; Fri, 28 Mar 2025 13:48:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285051] [bhyve] ampere emag freezes after kldload vmm, reboot needed not even a panic Date: Fri, 28 Mar 2025 13:48:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jfc@mit.edu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285051 John F. Carr changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jfc@mit.edu --- Comment #3 from John F. Carr --- See also bug 277559. The vmm module works reliably on some systems, hangs = some of the time on my RockPro64, and always hangs on Ampere eMAG. I have physical access to an Ampere eMAG server. If there is a JTAG or sim= ilar debug tool I can plug in I will give it a try. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 28 20:01:19 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZPWcc0MWRz5rMPh for ; Fri, 28 Mar 2025 20:01:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZPWcb4pYDz3xFT for ; Fri, 28 Mar 2025 20:01:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743192079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zx2kg8c3L/JlhUv2dtDJ8oY4rdPCEP688kS80t+Hpss=; b=aToGlrmT6bh06TZDuZasS/MAg2mL/ZVr3m8q63pwNcjSMgZvxCXmccDE5MpVpxt5qEg35G ZqzCt/Hl23JX1QsqRQXzs7KMLQaE3PLPC8Z4lvqOd0st84WH4IQemy7sZWC3jLfGDyJB3N tA/fB3r4kMSKi7SUIRl8J1sKG/4aiy+hIuGHfdXeBJGo6v+n9ACZq6KXKgxmeAlmg7QCnk I6lHS5GCITmkY16NGs8EYE2f8uSfVe0WtNf9HmKfcGbxwFYtIia7ZA4RZ1z6DCTB9QNZMa bohYl5aJinbNnHrK3tf6n9GcLWzBUOAKX+Oy+5d+H+YJWuzq9wE9AxmMZttOIA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743192079; a=rsa-sha256; cv=none; b=uwMT+/FqdNAx9KMpsAVxEa8+gP3ARWaSHru3J/2C6xZ5ojhh583gdNpg6VWjoErKANTEan jbmilzbCralfU5D2brVN3XgbhKIi48CrSjqywiHgJkYRQ0ljvvIcuU/ntmbDvxO7wMQhHq WJU3wmKDRbw1lByw36UO4S2hzFqvjrkopAKBiL7z6ROa186GsCxOoerfuwDF1UqfRScmL9 BM07xlGY5W94Npv5Hokd79S1nRvveNdEWYkmdALHf2agwYM1/MVda27DmFA6krywkXGDTo D5bqie9w0+6PzrTyq/fXFn9Y54M/Kdm5APw1e1EiArQTHBNsilefqhX6Vv41hw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743192079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zx2kg8c3L/JlhUv2dtDJ8oY4rdPCEP688kS80t+Hpss=; b=RdyvjA4xf79JOrfDPAiabspkBeyD2KZDqCG9aZKkyZYf7iW/zG8hCQjtMKLlkeEHrHX7+y nIU0XamO1Un21Grs8sYlpeI6tzdL5c8sRBhmlRCz//l/QgQFANxFxWTx+pQUC+I/ZYpvq5 D+v11HP2vYYJjkXaolXWSmshmNB+lrUJmfDzjOpG8W4+DzGI8rwJCESUU+MydN4ztbvaX+ KRQFmVHP23q+LNgnwWurnSL3XqzdWF+3eHlrn46z5BjjAeF2VMIH0x3S2eHEb+2+LG7zHT VJdFd89eBVyzUdQgzSify9jiQ3k2JjNv/fB12VBIWHmej2bwWXHDAoBQzc2I7Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZPWcb4KyCzx0D for ; Fri, 28 Mar 2025 20:01:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52SK1Jxe027555 for ; Fri, 28 Mar 2025 20:01:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52SK1JPQ027553 for virtualization@FreeBSD.org; Fri, 28 Mar 2025 20:01:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Fri, 28 Mar 2025 20:01:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #6 from Dimitry Andric --- (In reply to Mark Johnston from comment #5) > Could you please try, say, 3GB of RAM, and without the patch? With 3G of RAM it boots fine, without the patch! If I increase to 4G, it panics with "storvsc recvbuf is not large enough". = If I increase to 5G or larger, it panics with "invalid 1st page, ofs 0x378f000, = len 2048", so I suppose it then switches to PAE mode? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Mar 29 08:38:53 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZPrQk3ks6z5sJC6 for ; Sat, 29 Mar 2025 08:38:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZPrQj5DnRz3F2c for ; Sat, 29 Mar 2025 08:38:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743237533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ngoHaTxuTojFa5ly9FlNHKB1Sh3GJ9jF/wjjAc66SU=; b=lL1Z94CaQXu1SWd6+neVKYReLPTCH+2YW+BRWHwc7ob79qXne5cCtmbH0MPvQCjYyeFUgl G9QCTH0ZshjNQpoJtodFfo9h2S1zJb75+Z7F2KytQJOsnUV/Dc3TrGNuSSMq38FYLaXdXG fMPaXBN0yYnAqatfDEcz4kEGyJ4xVdMULYTYJZpEJP0bXshwzTOPJsh0lrdGXjkngAXGQw yjdBbQQl7Eu91dXob87CF26wm/MzxgDYI5/cJpLV+TF4jworxHYGSOoo+9y4H6Ve44reAr peBVuC8CDgW2TvT4znQCKR1mqWPP6RBqdzXdvLFkOgjfov7dtrOp7PksjrjiPw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743237533; a=rsa-sha256; cv=none; b=wqDvT6O+UDNeSvSkkWSMZCTpwMX7sywu8hUlXjqPc+xjB5qSBOdjDzYhz+dxtvEtK/b2EG C7RG/Wu+Y+5UitogUDjAdjD3LUQral8z60LRElKST81w8a58nNDFbX4dnmC8Nn+aiE6cjJ knS6MOKzdbgDczZio5UrPjToL2jKsHPqwkJc0Nwr/Opy+NPjGpwSc+yMPwbZxsi/jtJrnU gOVIP0IbGUR5iLIQ7JZqRqKfaVjs5GjnvSJIhTH7mjzYmqO2xwpwybYUWhfv4AUwCyfGoa KTuubV/EuP0eTosjOonCy3fJ9Vft4W375Lijp24uo8haI2KWIYM4BajilfTyvw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743237533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ngoHaTxuTojFa5ly9FlNHKB1Sh3GJ9jF/wjjAc66SU=; b=A3jomoSWgHKYiOqAa+TB15zkFyVkNquUfREXjhqmak0IIzd3AQj85sy2uZsmC5AFEwps9m Pl5Mezyqr/1Lm2hdpzMnWD4k2Az6giuXy/3cQnKlnwvCvYyFRhUx52orOthcX7KGb50EwD JBxstYIp5M9bvPmylZY0ClNGzqoLMfB+1FWIbjKA8JfxceEj6rdZ3I/uwit0MbtCKNW9GH ygs03SCc0vB52pGnlPeT/7pdFV9jluWnJwBDJAO9CIMTSV4UESOvFCCkAywoKjGmkrZN6a 2tc3uPBJskZeXOvxcrJFdgjRFwxXz+U5wQKQkwbZoRx+ZgNLDqw92iAzXTIDdA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZPrQj3z9Yz6JD for ; Sat, 29 Mar 2025 08:38:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52T8cr8r065214 for ; Sat, 29 Mar 2025 08:38:53 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52T8cr4H065213 for virtualization@FreeBSD.org; Sat, 29 Mar 2025 08:38:53 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sat, 29 Mar 2025 08:38:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.isobsolete attachments.created Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #259079|0 |1 is obsolete| | --- Comment #7 from Mark Johnston --- Created attachment 259144 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D259144&action= =3Dedit possible patch (In reply to Dimitry Andric from comment #6) > If I increase to 4G, it panics with "storvsc recvbuf is not large enough". Weird, I have no idea about that. I guess we should look at the packet len= gth in vmbus_chan_recv(). A stack trace would also be useful. But, does it ha= ppen at all with the patch in comment 3? I think that patch is the correct solution. The attached patch should also eliminate unnecessary bouncing on i386 systems with physical RAM above 4GB.= I think what's happening is that we are getting a 4KB I/O across two pages ab= ove 4GB, and busdma is bouncing it into the first 2KB of each of two pages below 4GB. The storvsc driver isn't able to cope with that. Again, it would be really nice to understand why the assertion fails on arm= 64, and whether the attached patch fixes that as well. A storage driver like t= his really shouldn't be bouncing on 64-bit systems. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Mar 29 12:43:07 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZPxrX25m2z5rNnS for ; Sat, 29 Mar 2025 12:43:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZPxrW6fPyz3ckC for ; Sat, 29 Mar 2025 12:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743252187; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/bTctr3D1YRyzNfB18thnbnLj5B1dJtUxtMR/Mmy4cc=; b=P1mzQPEe/nRehzTb1hIMEUn6SX+4+gZd526J5sUj9PSUkuFmiMQQsJC4pZ2XMkxnqSUSOl kP80UBDjvdkgehhTOhtcT9Kz0VEdTAm7qJkIAOLjq0P6ltlWvlhJrLBGKo//fWVC8DT3LU CDQ32SsM+3FFAQMGe7Whte3ZpCA8E/hc4SEPHYwM/iUVVjTEt5PILVo9aVmR8VKksKdvxp 8WWCuH8aBl3ZcYP3OHcAmvM2HlkGulOQOdRhBtUy1RP+xDI4nKrtc3vQ07g3yfOPpEV/qU uhDaXZ8+1TbPYA6zi4S2QJ3zUqC3//dpDzLEwlg9ppRveoFm8BQTRHrVGGQZKw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743252187; a=rsa-sha256; cv=none; b=OTxD53MVWV4cUNiAQ0TgIHj4vO9EUsO7O/ZAAyRp5e0FME82QStcuCSuB4lNzuHolmaLXA gpzHHSdgZ/fwE8d6PU8ZOoENF38iHAnmEJSqT9NvKR/lr0BqSLoBFCgex2xz8VqIG+vrFQ +zCuuu43WJjCnJo9fuVr/Trf32fa+KG01LAe9RpjkdgEaUPkyfuRofuwZ+iNuxQs6XkSBF HH6F1u9hvBfXGsYzBFWziH7HsF3p3K8V25ncZkJoiViMjxdhgBPuV4X0iUb6GSa1QgmgBw +YmomnHyepl+39VXBFKgA1DQ1BMXQlc9Jji87CR0Imv+xgefYvHn9SvZPrJ+9Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743252187; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/bTctr3D1YRyzNfB18thnbnLj5B1dJtUxtMR/Mmy4cc=; b=TjB5hAHtdfYFHjFBDlYq31AIRwLqJ1+M0cTQ0YllBFDHgLAyBtTBJgLgLAmUOz/vRyvRuz +4pcbNZUjY2XtQMf0GI1OsE3D5k1btB1cgbQq3mE2h3TEIixYuT8nqhf2aO7vG4zdk0ciP d290tDfydS5dUjvKiubuf0qVtIolRi2DicxJ18lz2XRW75H6GWa8UkNjRomFmCy0LtYXPq xr4iLozu90UQNoRAmMzZdsJUkotflL5Vi5uQOvimyAnEyqG1z09oKZyuVFq5Bgp+/dh+S7 Nsl7yEid5Drb/GCTlVlTCjHrOT6mpnvuDrjjwSio4ESvQb9V3ftI/bNA6iVZDg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZPxrW6FdczTZW for ; Sat, 29 Mar 2025 12:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52TCh7wO027759 for ; Sat, 29 Mar 2025 12:43:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52TCh7gP027758 for virtualization@FreeBSD.org; Sat, 29 Mar 2025 12:43:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sat, 29 Mar 2025 12:43:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #8 from Dimitry Andric --- (In reply to Mark Johnston from comment #7) > Weird, I have no idea about that. I guess we should look at the packet l= ength in vmbus_chan_recv(). A stack trace would also be useful. Annoyingly, I cannot reproduce it anymore. Must have been a glitch. > But, does it happen at all with the patch in comment 3? No, that fixes it for all memory sizes. > The attached patch should also eliminate unnecessary bouncing on i386 sys= tems with physical RAM above 4GB. I applied this latest one instead, and it also boots fine with different me= mory sizes. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Mar 29 21:12:14 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQ97y4D15z5s5JX for ; Sat, 29 Mar 2025 21:12:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQ97y2ZKsz3Fbj for ; Sat, 29 Mar 2025 21:12:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743282734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gvdOaXurSvXpcJJEDfs3Oy7hwtbBTWiQvMAq4CBHDLE=; b=nVOGToX+VJL7zSHQyH7vgO/vnYO3Uw9+PmRiwCIjdia1gasmc3CdJF2fk/2Y2bKyiOAWES ikB/JPLfm97D1jRFsVw04YtaoLIzZv5ENBvYiHgHzmGTrspNqKJBPV4BZaDBGiJ3Kjyt+U Gyk01V/yjRW1vDLea4ZCADvTEZO4emP4UbLL3e1LAuVDgUv7lUR4eydxDMII1y/djTgkci /6Vc9R6qwpcf70eJze0SmJyxGtobZBpMbMXkyS2vD0emanI1EXFqFfEA6rJmhhB9bIOAWv aegqIAZMAkVmV9G8dFXrWzWRP1UW0+gL1dYN2QvIzbC9hDdZ7WB4pt7mfdpFkg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743282734; a=rsa-sha256; cv=none; b=fURAe9tgaWjygyP3LfD024GypktQXcXQGs2rxXOPG3EqmqzrLi2+XCs1GrEtirlFPYIOYM 6dmHuDTypQVn8eFT80nyWApwWNbvsGEspp0pD61eJY81vc/5U3N5uQbGznGKbXxX/8KBIT LPiDwhuWfPeY0XVlDzmIiR4hs8YZbvWezB7IMP2Gpsp6/g93hI+mhwZqQuUBu/QebwN1ez /CcQ0Fo6A2mE2NQ0cGtnNn9f9tojQakzYzOsbktawKg+HmaGmvhL9Xca/4NxcOU7E9LPQv SAEgC+nv+4hS148Bs62XsXTuvSPuhvUmJcenKa76dsx8j/DF1WyLW7CZffWP0Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743282734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gvdOaXurSvXpcJJEDfs3Oy7hwtbBTWiQvMAq4CBHDLE=; b=MuSKZN3K6XRB2e0kh4+x3C5UDWvEd4P5yu50OMRj7OvmzI0apSGbLm+D+zNBWgSgqYFGx9 s+kSueVlbog79jVb5Rs+62umj7WIwEYtk0wCSP1yhtHKTKtzQvTcgzQkawadY4cmKOcEq+ T1KlG7NWnvHiOSqSnEOOzILpsw+Z2oQKqznww8wffUm4VN27xnTTEvQGNqoLj5aeQYR9HI RGsnXFo9gomCnEuLVRZFnpZg1cbfD5FkkAMUNdz0J/GDga4iiSsQLutcuY+L9YxTxjZvKp 8RTHQ5AAZeEIA8PCqWmUo1kmjOyt8TyafPacW2cDiww8oEBWgce9nOEvPu1Eng== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQ97y21fVzltJ for ; Sat, 29 Mar 2025 21:12:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52TLCEiM005367 for ; Sat, 29 Mar 2025 21:12:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52TLCEEs005366 for virtualization@FreeBSD.org; Sat, 29 Mar 2025 21:12:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sat, 29 Mar 2025 21:12:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 Mark Millard changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marklmi26-fbsd@yahoo.com --- Comment #9 from Mark Millard --- (In reply to Mark Johnston from comment #7) I've never been able to get FreeBSD to complete much of the boot sequence under Hyper-V on the Windows DevKit 2023 (aarch64) that has Windows 11 Pro. The console output stops after the masks line of the EFI framebuffer information or somewhat later. The farthest I've seen is Event Timer line from the kernel output. It has stopped between those points otherwise. No failure notices when ti stops: just no more output. It does not get far enough for Hyper-V to be able to do a shutdown. Hyper-V indicates the VM was still not ready, even with a long wait first. arm64 Windows 11 only supports v2 VMs. I converted a downloaded FreeBSD .vhd to .vhdx in Hyper-V to have something official to try. (V2 only supports .vhdx .) (My personal builds have historically behaved similarly to the above.) I've never figured out how to get a serial console and named pipe configuration to work. So I'm dependent on the monitor being operational. Anyway, my retry at this got no farther and I was unable to get anywhere near figuring out how to test. I've no clue how specific the Hyper-V problems may be to the Windows DevKit 2023 type of context. So I'm not sure if anyone else would be able to test. I do not know if any of it is tied to Warner's UEFI console related adjustments that he made. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 12:29:49 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQYVj4zD7z5s60q for ; Sun, 30 Mar 2025 12:29:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQYVj3Z57z48jD for ; Sun, 30 Mar 2025 12:29:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743337789; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RE+wV6iyOg9uhSlTHz58lgah9CsRzv1C93TMcHHAWc8=; b=n2/D+5zGDKl7cTe5EAN7HRvBFyEFjD6FET4xw5zBDpKy386XHfHjP6dyGfiW/wWcjMXQJx TtYuuSDPT1bqyCK0aNgzNFSQGgNhcdds7aSOdZ3n+cLg7fMzuL3m/wyEC3BJ/FPt1tj8FE KloIfP7CQP9glCH3wAPFsZHXowiGqz0Jf/FZvOHgYuhIIo5/WZAfEteHXnOF3Gs/msCv8R CCk/o9Y+C5NQos5QzEjQ4L/4Dq7D2B9jyfFJtpoFja+S8xKswCAtBX102izy4nDr6In9F4 u8w83e7H2EN9uiPWy7fGyWkgkfqqUs1sxdMS5OC/w2tYNRxrEbW+3Nl9pU98wA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743337789; a=rsa-sha256; cv=none; b=cmcZDSU2I5f6UZluR1le7msp0fEJM0Kf0feV3SJeyUVAFdqAjoy/WtW2pZFpnQlm3CJi8n BTPy4GYSMCP7rQvF44OnMO3vvfpR/c0TRnhjwEWGn89KByv+SKWFx5aKJLWd9xD9sxMGWT xfVHmcGlMxL1ffJ11Elbmvd8IC/5OkpbormoMfIkQ8f4QRvIUxGpKZWWg2AG3enQPGG437 jZj/PJz0HAfVb0gTQx6SWq/j++Lb4yzlimDRuQk0xS4vawUui6JuBx4BQAM3eQm69RvZaF 97dZ3RbOj9hpOhWdcp9/ztbMtGG4Uno7GVUWJipeZSLeoYJFIOZvLB/dW0pGrg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743337789; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RE+wV6iyOg9uhSlTHz58lgah9CsRzv1C93TMcHHAWc8=; b=nWKq+Ixsj1timIssFvDSaQvKNWL/YArck78WX8PK2+0EyPiMZehtRbk6mT7A/kqqsQNt6a +dXatrsWgWdYPdP1YcYVhO3XwFDLHFpPm4LZ43NF8bwSSOuMBbW09POtTWUtql8Vhp/mui jDwGWx93k/yVzyyjRyYjHYH+HLeTCD9Uf/JJD1pWMcdcfPwCNtHpJE/3d8jmq4Q4mqi6yH pKKY3Flyck94qasNhJKzqDkeSoenJG+RcC269ficMZRo1HAKwmYshj0edA6X+kQn0ruan/ PbFsy8g0SqrWQkO5DMEQ+UKZIwZ8TZpT/DCrcms9yQsS5wtbkZoKXcoZ3oHzOQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQYVj39pMz1DgP for ; Sun, 30 Mar 2025 12:29:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UCTnQW054481 for ; Sun, 30 Mar 2025 12:29:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UCTnjT054480 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 12:29:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 12:29:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #10 from Mark Johnston --- (In reply to Dimitry Andric from comment #8) Thanks for testing. Are you able to verify that the patch doesn't regress anything on amd64 and/or arm64? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 13:25:27 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQZkv6vnVz5rQgZ for ; Sun, 30 Mar 2025 13:25:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQZkv5brmz3cD5 for ; Sun, 30 Mar 2025 13:25:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743341127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j97dmdDezDeECMBDYALScykd7iuHylWtixKi15PeGRc=; b=uN2dFRt675ixEs+2wf+qZ8NQNNnYuv67dG3Onl1zzd8tTzwV3NmxHB7IXAGUFPWZAjjEzp gs/1JUcTg8jr4IKyNFaG1gbALx+RkvollJ652pakbTwZl3ruyXJEgK0HQzQFh099n3gfuL qk0iTd9rXOw/s9wtaLB578kAFD6bvBwRJrr3JbDZ7Bh8QTgbJokNfF/1XlHA5615LHfu2n 1h+zhKn9i4jfN+Pdf8atQ+lEEo3BtphiTfDzuipY9wA+XYB042XBXrKxSxGE92qZOlwmRS 6TO3JZErvOtyPqffD2erTdFi17u4s5xiQYTq0UDJCEcpOw72g/G31V4RqDseKQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743341127; a=rsa-sha256; cv=none; b=wzBBfscS7EDpY153zKhdIOxKL0pwLxoqIqEdZzDDWAAY1/DhTtNO8N89cUlQpyoWXiZPoG v27OWiIzex2A39Lx28IYpTHXbjpCaxFOGDf3j66HRw0PlU9M7JlZLFYY5/QUfZAJNpi/Lo DC6QLIU5yrJyyD2izhui2dTYSLaUM61hyKPpFCA1CLgvl1bNb/MaMvDKV/yK3j9gK5X2pn O0heOFptM7Kdjh2O2bKusX46MWURTIyXD5oKrazQS4xdvfQWvKNE0IC4alIqPIg5PuogZF g6drVNQ66pRgK+YIOpxv1fbbVqQdryqtuMC0NCVcobUZKoFupkdPEmWaWZ1pDw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743341127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j97dmdDezDeECMBDYALScykd7iuHylWtixKi15PeGRc=; b=jDYcnh7IsE7r0zGnhZ3RwWdb9kZLkej/C1kb3NmUTfxDSiQ68hYESyxYfZSzu/XWWqI2rk XO5hbdXHaf2qVaiVtieF0Ekitna5oxNOPJhow6lOt/iwPISM833XGz4ZMLIQlLFF4O2Dwf ION50C3F5PywQ7BJ3poFhIPd6BRQ4usjWXTCXpcdlJq1mUVVshkGHMtsO72Ce/qYcKYb0d cUr58Pt+/ZI66wEA72FNDb/+bATvPm3lAdUnG4GywKHKWaiYaxqs0tPsVWeTsIT1Uo0xuV Of3tCTKwF31hvuEJcRZG4FSHxuvVc3nLHGrkrVWpy/Rh0i4l+GWWuJavmeF7KQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQZkv4HG5z1G6W for ; Sun, 30 Mar 2025 13:25:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UDPRZ1057989 for ; Sun, 30 Mar 2025 13:25:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UDPRZp057988 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 13:25:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 13:25:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: weh@microsoft.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 Wei Hu changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |weh@microsoft.com --- Comment #11 from Wei Hu --- (In reply to Mark Millard from comment #9) You are likely hitting the problem introduced by following commit: https://cgit.freebsd.org/src/commit/?h=3D1b9096cd1d2fce1edb7077aebd3512cc61= c54371 It causes arm64 boot hangs on Hyper-V. I heard Andrew Turner has fixed this recently. But I just tried boot from latest main on arm64 VM in Azure and i= t is hitting a different panic with stack like: panic: vm_fault failed: 0xffff000000627fa8 error 1 cpuid =3D 17 time =3D 1743320704 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x38 vpanic() at vpanic+0x1a0 panic() at panic+0x48 data_abort() at data_abort+0x28c handle_el1h_sync() at handle_el1h_sync+0x18 --- exception, esr 0x96000004 rib_notify() at rib_notify+0x38 add_route() at add_route+0xc8 add_route_flags() at add_route_flags+0x1bc rib_add_route() at rib_add_route+0x33c ifa_maintain_loopback_route() at ifa_maintain_loopback_route+0xfc in6_update_ifa() at in6_update_ifa+0xa44 in6_ifattach() at in6_ifattach+0x1c0 in6_if_up() at in6_if_up+0x98 if_up() at if_up+0xd4 ifhwioctl() at ifhwioctl+0xc18 ifioctl() at ifioctl+0x8bc kern_ioctl() at kern_ioctl+0x2e4 sys_ioctl() at sys_ioctl+0x140 do_el0_sync() at do_el0_sync+0x608 handle_el0_sync() at handle_el0_sync+0x4c --- exception, esr 0x56000000 KDB: enter: panic [ thread pid 202 tid 100285 ] Stopped at kdb_enter+0x48: str xzr, [x19, #2048] Anyway, 14.0 and 14.1 arm64 image should boot up fine on Hyper-V if you lik= e to try Mark's patch. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 16:04:29 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQfGQ2YWWz5rfdM for ; Sun, 30 Mar 2025 16:04:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQfGP66QCz3SqJ for ; Sun, 30 Mar 2025 16:04:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743350669; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wJXttXCRhSULtaClIHIz+b1rxJj8S1UyhEnp2MdTPx8=; b=M9KUXHbTEABpvzSpuEDT9KZgQNpkNK8KisVdbXZszhapygXX0b6wk9OVBns7HG1sBcUwK6 J0goHSA9hTqLtutzr2v2wLdGSIv4BDkESBi3JjAysdWV0X0tuZ/qG2b5Cv8ISTPoUqftKB lMjMnF77f1n0VvAAUhCKkpaGGvoR3eyrXktqcOw9eO1mlKt5yXtmA64+o/CiUZxsazV6jT uCks2SBPDjXE1PeXzsZa1iHjR9AhjEmJAtmg9yQ0e0ncp6zqkwuQpAfcCRLHRWE6/NkxjQ 5q6/bTMdnDA0DtcSzcx/hdDinxHEPmGyUxSNPRI9KqubGneKDJDAl++jtQwxwA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743350669; a=rsa-sha256; cv=none; b=Niye7G9plAoCdXimk3Gt4W62ELkAMzy2Ecr7u3N43TNhXHp0KjV2NugPo+nj7TaVefzgU7 7LQFEJ7Uh+iM4TMCFnq6SHlLld1zjvdaGNXMaqOq3/aiH1LnLrpUSJOve5xBt0JxFmnY81 pXbIpGmAy6x6wrdvLSdjQpTGTRiebPQhQGrQiYMMXShGqkCLgi85ER0Q7UewAqVihKfQD9 VuzZ3HOjG5JUnbMelXhU/bgcvqg/pzENCg+vsgNw7CAqBJvFpL5HQhEspDS6i/dFcIXxCt 5vSXKJrtL+sM1lUWkT0qQ1v1VHKZxZmyQj4NubWDRnck41OEG8zEwTsmuXbKpA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743350669; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wJXttXCRhSULtaClIHIz+b1rxJj8S1UyhEnp2MdTPx8=; b=URTIAfxbH4Gy0hX+SNeg9/94MXLDXGP2W4oaboCLbLlW9ZGe/Si9CjojerulnYMYfJuQ17 LjZgrdyVHOCzASSS2pUkTNYDqFnDfMH9gaPlDxiJymUB7/rO8OM+IPpE+m3PYIh4NyBoaN ZLaiZO4YdyZ0ludo6hAsyK9ENO5DkX0IoCvxmZCNvndfc+8SDPQjxgjuy81Hv1U9aANOku hvPx1rP323qB5BzJjfQZMxwnFlgUU1LS3nVeYsb/uBJ8lcydXDAqcw21roVVu/Vg/iEs5t gUd+ZEoVuR6/W0ss4V2hcYE7s16kFGdM2J1ESdoTUCH5L+rIne196CKfPn8wnA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQfGP4kkMz1L5L for ; Sun, 30 Mar 2025 16:04:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UG4TLs040300 for ; Sun, 30 Mar 2025 16:04:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UG4TD8040299 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 16:04:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 16:04:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #12 from Mark Millard --- (In reply to Wei Hu from comment #11) What I was trying to boot was from the main [so: 15] 2025-Mar-27 VM image that I downloaded yesterday from: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/15.0-CURRENT/aarch64/L= atest/FreeBSD-15.0-CURRENT-arm64-aarch64-zfs.vhd.xz My pictures include showing: main-n276101-fd52a9e11c52 with a 2025-Mar-27 date. So recent, for sure. But the observed behavior is not new in my context. I normally run main, either official builds or a personal variant. If I use an alternate it is normally an officially built stable/* (stable/14 at this point). I converted the .vhd to .vhdx for use with Hyper-V v2. FYI: aarch64 no longer has console=3Dcomconcole as a possibility in the loader: just efi and eficom . So instructions on https://wiki.freebsd.org/HyperV about named pipe use for serial console use for v2 are out of date and cannot be followed. There is no 14.0 any more listed in: https://download.freebsd.org/ftp/releases/VM-IMAGES/ There is no 14.0 or 14.1 any more listed in: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/ So I'm guessing you are recommending testing: https://download.freebsd.org/ftp/releases/VM-IMAGES/14.1-RELEASE/aarch64/La= test/FreeBSD-14.1-RELEASE-arm64-aarch64-ufs.vhd.xz that the web page is showing 2024-Jun-04 22:30 in the Date column for. But that is far from being recent. But may be you mean: FreeBSD-14.2-STABLE-arm64-aarch64-ufs.vhd.xz that has the Date column showing: 2025-Mar-27 10:59 that would be recent? Anyway, I'm unclear on just which available VM download you are suggesting that I try just to get to the point of some example operational Hyper-V session. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 16:30:39 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQfrc4kKnz5rhZy for ; Sun, 30 Mar 2025 16:30:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQfrc1mvZz3mkr for ; Sun, 30 Mar 2025 16:30:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743352240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1aTwZU2FXlW9UeLvWcjHSBm6PpT7NEjT6FS6ODibObQ=; b=dhEBvH/wrbUHw1GY5nCaVBrSnWlxeakbnw6lY6+n3OvvYg42R3NbaJq5N6a0PVUqyEG2/u cmyc9cjqUV6uoio2Fc0Khaw8L36QKFeX8DZDtljw5Avu+0HliJOc26PcD6WU1s29tI0GP/ 420PgWKwEwESJ0wfj9vLzr1DsSRu7zYNkjwHX5W96VJS5lQ9RKn7SgxXf999kuM+uEWkLm MbpqsI70Xvs9deqLsluZCv5lsGNyJ/iuN9LI1++hgQpBI/XvPVXspAXNTay5ULdsgWs4bu ArTH6BxkBEY+OuklFNTE/lfuWC8PLdrmT2pKE5uHj3fBWoP/MGIE5z8Ye4Orig== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743352240; a=rsa-sha256; cv=none; b=pChgbAO6RVa+CIb/rOqEXrEMfTnKHrvdiwdzwBLQzzohkdKLkVwH84CiUeDKW4opPW5rjO RvYM6XYbB4UoH5cGsNTCveXXAvUsQPcXVi/3YREN0LB8bVJhqFypndmvjJzY57jtu1qboc /ficx+PfMttbuOV2iXk7yIiPnQHn9nP76AaWZ8TbFq3VcIU++XwNe+0rwQ8qBxH79iS33w huPJeZ26SiJDFdvVBbIe0uMJh/slr8R/IJ7UniMZF9VvwfGQAL5Y1Curqa31/pMluX69q1 xaQ0SBydU+Km4I4piyrKZXlbQZGF/kTfqJmlDV48/tXYVvUUK6PPhMEeiRY0lQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743352240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1aTwZU2FXlW9UeLvWcjHSBm6PpT7NEjT6FS6ODibObQ=; b=lci4li2HCCX2v9KuwNezTFaAmHloR9+c+j+AJV9qV4h4zcoZE8I0q3X6FeJoNt24VbDiXj CMOIA9mFGhwma/D5O7IUDXQmSZVJY1AHpCPiPuYQVxKbD/CHzcHOasodaGZ9hwGNOsiS6e xzDvSsIbPN6DUD+xOc/ehhNXAoUbbRIcwcFKi87cF/Wu7Zh7dBSJgbslGUdBTcCYxb5AAl zVthfuOxuqM6gkks1Hs6ZcVLTlj/YvWf2lW21ZeV06V3lGTtNQPX/i1EEVGVOSJHp4eDax V6rJQz/fl3igbHw+qQ+SibcjW1qGhTLX0RUcvy6Dk27uey8ljnxbrSp8Fp3w7A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQfrc0QrBz1L6h for ; Sun, 30 Mar 2025 16:30:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UGUeI3084837 for ; Sun, 30 Mar 2025 16:30:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UGUe52084836 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 16:30:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 16:30:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #13 from Mark Millard --- (In reply to Mark Millard from comment #12) (In reply to Wei Hu from comment #11) 14.1-RELEASE from: https://download.freebsd.org/ftp/releases/VM-IMAGES/14.1-RELEASE/aarch64/La= test/FreeBSD-14.1-RELEASE-arm64-aarch64-ufs.vhd.xz (converted to .vhdx) did work for having an operational Hyper-V session on the Windows DevKit 2023. Thanks. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 16:36:58 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQfzt3fTWz5rj0J for ; Sun, 30 Mar 2025 16:36:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQfzt2mBWz3rNh for ; Sun, 30 Mar 2025 16:36:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743352618; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0sOGLAvfhL6uVScGGlicTcdApPIJantIZyBPIoQSwBU=; b=uirHV5B0K2vahYhzNch5vKTLE93+EYnQpWB02msO7Or+wGOfFiPzXHxoXHiUCZCfqmpaBZ FyUbubqxoMa/e6dWpyPySI/yZU774wcYQI6R+k+vYOaU1ExBTngZD6LJv24vGZ8a3msUg+ d+CTaWUQQnsKZLuYSTfqCMGMbshqSAyi1izC8Qh6NAtCx6T/PSamS2So832uhsud0HwviO 7iZBb2Rem++pz0jK02Wi5A3lwUBCYjtv1LBbBHCo4rXyByn4REDS1fRPi2AduL5cQRxlqQ IOJNxR3rZBVxDP7xNszhFsbmNTEhhllH/oY0OLvVBSfJF6MQs47jXlth+i0bpg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743352618; a=rsa-sha256; cv=none; b=WjJIEdg0Qq39xovBulTvH2fL4E5MA71TGJnprZJPE3L5HIpAUVkzJ6exBcK+3UVnk//byc l96X3s83elE6a5EmG3E11NqT+GbAava2JiLNG/ni5QK+ZN56HYoSZmSLD5pfMXX+RBF+w2 N0pThkWDnyCYpWfPs9uWwnt8xAB95ctH1s0EjfchF2Noooql3roXHC99X78Dyvqb8E98nC tvw/Ror8yucthmXU6wvrSqa9qwjYHn4clK243Svntxf9Le27fdOYOkgQLhu+sZOtY1emJi tJvGXjCt+lkm0seoPztNZA9Zodkjz+hQVRyNvry/ZKfx852fvWX60iIcVksvAQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743352618; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0sOGLAvfhL6uVScGGlicTcdApPIJantIZyBPIoQSwBU=; b=PtRtuwhJf4v3cz5rIIDjOYwTFSuWZPfFwgmELk+lA1qOyZdphLlZHV2ymoYn/Pf/XrSUFA I2lgmelMLr6CRwAQW0zise00BKlOWiLgI+kFoX3/RU1DC7xcx5fExcuoh8XNPl/UDIBM72 HEU6RUEaNSAxP2Xh1r1NMzY2c5XO2eA0S+Be8uKBgyiaVQpOctKTBlfG9uNwFF5nmCKaZv gj2II2f9DrKukIILqRmjUAC4neo1E3F1dpJkrXEaMKXpN7Kxi4SdGDo3jOCV1bVfWcEitp 7VKn8OZS1XSbwDosdA/FZ+NuZZ64/fVhZL1dRoq81crmZAyJfVHORKTPm+1MLQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQfzt2N4pz1Lq2 for ; Sun, 30 Mar 2025 16:36:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UGawr4099230 for ; Sun, 30 Mar 2025 16:36:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UGawTM099229 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 16:36:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 16:36:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: weh@microsoft.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #14 from Wei Hu --- (In reply to Mark Millard from comment #12) I see. Maybe adding followings at boot will just work for you: console=3D"comconsole efi vidconsole" comconsole_speed=3D"115200" boot_multicons=3D"YES" boot_serial=3D"YES" These are directly copied from the FreeBSD images in Azure Marketplace. I d= on't have on-prem env. Everything I tested is from VMs in Azure. Now FreeBSD wor= ks on Gen2 Hyper-V for both amd64 and arm64. There seems some issues on larger arm64 VMs (with more than 4 vcpus), due to too much parallelization breaking the dependency at boot time. If you keep = you VM size under 4 vcpus, it is less likely to hit the panic mentioned in Comm= ent 11. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 19:00:32 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQk9Y24w3z5rvSc for ; Sun, 30 Mar 2025 19:00:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQk9X6h4lz3LDq for ; Sun, 30 Mar 2025 19:00:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743361233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3dXdkEUn1VM0f0nZbEG3k63fzvkGyesCRXOb1CAs5do=; b=Xy3UpRVpO3Ih9ZCJWVIF+XrXZj3reibKeyoAovaXQ+VrUs47Wa8tLq5Bs7KgdkYDHTAcVL 3SjlL8Z0ojkfq0sBtvXFrn8GyRoQbvOgooKB0R+CRe84erkIxsDc4xmQ6SrLWwXECVEdGR dJvJOYAFvo8ILBwv4oD6IO3TfEFe5kxzqPpLIiH17GpL51v1aZDFqpcrCLGGgVmhpo2frh g21c6JNK+kuHzKDs77+P/u8UK6N4u9O+scWyI9igdFOY9kUdG3IZVct1cgalHnoHXjdt5O JQt63GR/KcIlGipdFGjkxloGWLS0OHjrOgc69ozXilDg1v6QYMfkJjQvgmeF7A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743361233; a=rsa-sha256; cv=none; b=EizoTRbSgX6EtkG5F5ZPwZbfg14IJi5hb6JYG1oeqEJnQNecAeRbDusFg1APdF0H2do86M sVkbqtBr2Xh568vLXx+dCH26giSiYNm/6F20GOcIj/+46YCWScCzvzUjFlciFuXAkSbsl4 e1jncnV30XaEzlHX++Mdi8ABqPvk0etswkHqbRxtdzy0rNL+WAnk+Peirtl1dpeWF6JLZv YEc01KZhMOeDWzavwXZVCLRZSS7HkUloBU4y+dSY+Pt/7K0Wiq+gaV7EydbRMGZAcfTdV+ OiSJPkS8O7rpTvAKn4DmwvhmnWl76S1Dv3tMGr0PYvw4KpOZxNdWThQAXoYXFg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743361233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3dXdkEUn1VM0f0nZbEG3k63fzvkGyesCRXOb1CAs5do=; b=TkDYuqW3bNWRCFzkpx2sLTdr4ohAmQ7NP9miXdIKcswPjYp+iIUaVJ6ZNDbNx3RWK+GdCW tmAAXo223CupWxCV1qIfyz5slZtidaIv9GU77H8eW/CUw0ZIg/UjPQVwpFNN0X1mpNnStn TlipFWZtU2LTlsCa9qENz5QThujCH3Mz4LnUeTfr9g4YNh29K78qcFF0KiKmtAWChbVOGy +liRgDSf6/LB0ud+dSdw3qiYkj2PZDthOpI6PtQmBUIAm+vaTvteU26StjsnCSR6oid+P3 lufFndToUbM3UwKK9kTctIM4itDmRsqoP2Mw2JFbr9JrePi4+G4AVudvUhLc3A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQk9X5lDSz1QWL for ; Sun, 30 Mar 2025 19:00:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UJ0Wes049232 for ; Sun, 30 Mar 2025 19:00:32 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UJ0WO4049231 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 19:00:32 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 19:00:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #15 from Mark Millard --- (In reply to Mark Millard from comment #13) (In reply to Wei Hu from comment #11) Also working for being operational in my context is 14.2-RELEASE from: https://download.freebsd.org/ftp/releases/VM-IMAGES/14.2-RELEASE/aarch64/La= test/FreeBSD-14.2-RELEASE-arm64-aarch64-ufs.vhd.xz The web page shows date: 2024-Nov-29 11:11 That is about a year after the 2023-11-21 10:02:58 +0000 commit that you referenced: https://cgit.freebsd.org/src/commit/?h=3D1b9096cd1d2fce1edb7077aebd3512cc61= c54371 May be the change was not MFC'd? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 19:19:07 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQkb008zfz5rwc8 for ; Sun, 30 Mar 2025 19:19:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQkZz6Tqrz3Zxv for ; Sun, 30 Mar 2025 19:19:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743362347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BACb88jijSj81C6UV4PlNgDscIGRKiKr+qwaFN50I9A=; b=CnvlumE+2kp1HYOLY9rURl9ITqzTvUjVmqfKRSszt4XSDndSEd3kYVY+P7lGq3+Aa88RE+ uQRZPja4nBlhkmiw2f1kANrT1o/H442nvosOjJbovRYRL7E5VxBBArQKFUp9V6tJDqHi8e pYklqdCTeN8nrWCy9Dv0VEs+OvKy+Y87dgfthbDEqbBC39mNTpj1/aqtvxdtqNemL7XzK4 YVLcfPiRHJ1h8i74q6OgGVFkVgxUjfmeV+VOCuNUDq1KP/ch9UiZNpsooHRjS696DyQYyh sJ2TuA9JWeqN0oJnX0pgyWtbiDYoX3hh8xS6RfkzItETPc4341dz1H0DtisTZg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743362347; a=rsa-sha256; cv=none; b=UFFp+5vUKy/QT3tTAQKDyNEjES3DfKSWmfO4yqxg+u4+NZBXD2wFVMBIwPt8Avi7yNEnEO Rdr9LhLz6T2tHEdG7HeEClmyM/vE9INyf5zeXHIbqYIJCG4rKEKDS0fiPs6VykCgom7JAL AiG4Hcm3+qc9LyIrMUGsds8KyasOdcOp/AOgzKpoO/+j5pwjlriYw2ZN6UDxRxhW6bq8Yn XOjTf7zx7TONNZkfgXYHzn3sTppYd15Y3bWBFR++AMu7/vvQ+xy/MAxrmv3ZX4k0f1/SdX GZjpvxGL/Vzmh29NcYmRjctIFcQ1Q1hQqIyu/+lKdOd3eLTQIy5r56BpsI8ghw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743362347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BACb88jijSj81C6UV4PlNgDscIGRKiKr+qwaFN50I9A=; b=RRJDs3gz2hqowOpwYaFq8fjLn1sVVkBSL6OKK73C+OfwYQOjy6ndkKzUy6QbvcZJULnk1A KEHJRbMRz9k2xl2stvNT18Toumtr00ztdZ2byAVyvSd28Z8WFLFAW6YMEEcKIbGq3BD8kG 2876rnQtD3+lTJMhzsp/YraGDtSeAhbGvMQNtwWEK5+p3OMt1xulikY3AQk6wUibW2zCxk Lzd19yO2WRHnOtWBY12QrySyqwftN0x0nkx6bYdlVaLbMR7i36nNjoZobS0ZdnjpaV7C7x n9Rzq2wQH8qX00R2d/Q2TbR3wRo08lxO/pQ+ZrW0LhGVQuHEO59owQtZeFjabQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQkZz5kVfz1RFf for ; Sun, 30 Mar 2025 19:19:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UJJ7C7080139 for ; Sun, 30 Mar 2025 19:19:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UJJ7nl080138 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 19:19:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 19:19:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #16 from Dimitry Andric --- (In reply to Mark Johnston from comment #10) As far as I can see, this code only affects i386 and aarch64? I haven't got access to an aarch64 machine that can do virtualization, unfortunately. Run= ning Windows 11 for arm64 in VMware on my Mac M1 works, but it doesn't support t= he nested virtualization feature required for Hyper-V. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 19:21:54 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQkfB70xSz5rwwJ for ; Sun, 30 Mar 2025 19:21:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQkfB4BShz3dJK for ; Sun, 30 Mar 2025 19:21:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743362514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4dnuC5GheCPL3UCpF9hPSmStDpat/WKcgO+qW0ZZAO8=; b=DQlyKVZZ8UeFdXWUgc81OHtICGdv1tYiF4iiQG/ViUJQprz+8Hg3PI7h0lIVnF2wloHTnP LUYccY1Yl6cXCBVSY0fRR+VfqduDYPTaGEdcPHTispdyecCKiFvE7lq7rcV27nt1T0QJdy MTsikG0ForKcwfUV3lgsRKys2PKxeswVRGgXmJWcSnKw78Ko1epxRxeTy5ZDcWT0WwiEWH OyRCuUPlcCNQHv/tvqy6jgQxUYfC5MrsicMpZX3QcFflgvkofmai297rp0EngXBnNJgkJA xQ3BhUMoh4TdKwHSJehE7yfaLPGE4cTaKrHE+kp/fknVbgB1/VT8z7qqhj0w6g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743362514; a=rsa-sha256; cv=none; b=CuJdY2tEEiAyra1YKgaM/1YBN93VcNs5J21HQ9hNdN955zBOsA9wagEB9HhtRjLmxzDaQx e9VZXAxuGRx3ydQ79jtw+Efr5AmwyKTGY9Hq0/ljkQrvwI/eVihzgOt6Qnw8TCXpulMR/I f/00AlbQ+idm/2Gomy6XypPOP5seNBzLZf4yOeNp1u4Ba6SLRhQk7tUfT4YYfrRfcG9EzE F1zn0CfEtf68ESF5BeA61r4f8KqXpuKFdbpbvEFXqFRj15yahTTkCulLZ7/3OYE5oAm7yo f3rCg8GMf15YTs5ZttcZUj01KnlDon+ddsI6RFpmRR4QN98XYCj8geERGBt8yg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743362514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4dnuC5GheCPL3UCpF9hPSmStDpat/WKcgO+qW0ZZAO8=; b=I3Ha1XMMV//gQkTe3QAOGsik05bAkQq+q3E8KaUVp2eivP2esfLER2hn7wCqoMbEY625L1 6zDNwR0+ZtOTxAyp3nvVFqwWx+mgPjyPOp+5qC2Kp0HvID33sSGUgOcqWKOwu7PCjpcBku XNYdfV0H+1hXfN8FyRBQFSCrOeqvIESxofITGZTzAAAgrykpd27c10sJKPXF0eZAhwEkmv 1ckkxwmhCIxLoVDi+fAcRp9w0/p2SJw3CtNyeSm88maJo2GndUdihMEtO4OZT1rscr3n+6 7h4ZKr+XqnEBTlw2YovkLnJMV+vL/Hjz4gowLeMX472oVgpvWGMjyuUUWYnffg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQkfB3dpbz1Qqs for ; Sun, 30 Mar 2025 19:21:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UJLsF4090451 for ; Sun, 30 Mar 2025 19:21:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UJLswv090450 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 19:21:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 19:21:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #17 from Mark Millard --- (In reply to Mark Millard from comment #15) (In reply to Wei Hu from comment #11) 14.2-STABLE: Also working for being operational in my context is 14.2-STABLE from: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.2-STABLE/aarch64/La= test/FreeBSD-14.2-STABLE-arm64-aarch64-ufs.vhd.xz The web page shows date: 2025-Mar-27 10:59 That is a few days ago. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 19:36:04 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQkyX3QQkz5rxmW for ; Sun, 30 Mar 2025 19:36:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQkyX2mCvz3ncT for ; Sun, 30 Mar 2025 19:36:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743363364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QMP+pf78XWzkziRz2TA/Wvoa2Medg116wGIstiE6Ph8=; b=kB7pleOQrmvM81fk2YOIjK9zPvFnbqz61He8hW43VVDMtWrD+47MRDIALrPayXZc2PYeo9 98msIDgiLwhKZ3bfX0LIg5S5NuR42Bm6MOjg2QueimaLRvm2C+ae1IbvSwwNRRqZxbzVeO 5dkdI43PzEkMQCSDSKja2n7zMeVCAdn9/aHkfbUZPydoTJRjyZQFkegRErcfpjxGVL3Vp1 rlmJ3fg9iVOpKIopW9xSSNHWUQxH8VAaiiPmfMrQw/N3phpB9sCDGgiT1xq0Ov6ES7zDDW i/RC9LdZqfnKdlcTG5mB2k7aOM8TX7a+VSBtwUK1/NnGuVl3ctjnxCWwdLR9Pg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743363364; a=rsa-sha256; cv=none; b=Cdu2NyLb10fXNWJAZO938wj3YbyTf7tasLQ2kdJmf1dJuQ02RpjRR7jdUQyh+I8g0g9Lp2 vU8cfFkon9TWEkfGOitDnSZrEuywtVqmAj1rDeCc27PmaKyOzIxHc5GLmUwVQyx/DcZuSY Hdm+neYF/7htC42kgIQAuQSCb5D500JxMkQgTXADzJTRbjusHxXuD5faVzM63KRelIlrlH V/H4ep2Dvk1KtpRn3fp8iqLdO0q787SlTbieNUalQT+sVdnqNoQXUswzvsDn5vWQFS9CmN O1Enbsom+wTnNu1q5r3XoUelUHH59poIVqc7zno7VFQG6Iu7wJbMfn90rFlI4w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743363364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QMP+pf78XWzkziRz2TA/Wvoa2Medg116wGIstiE6Ph8=; b=FhfchaLjvqPBQuSoFJe2cSi3f8lt6MrC7PXm7MvJX22C7KLwKq2QI+b5SaIvKDVrStOPI8 VTQZC8eXyym5UWv5Xrv2O5vlPci+UvcpP27XLi75cMh8pZKKSWgFL25zsuzx2oxFSw9L/9 0U+eT831lp6Sjae5pR8UIzeIrQKvv1YPAjPXLuTAj5fR7KXhfbQx+XaQn1qJSIRPanqxs7 bPzf/ycBQ4kkNjUHI291eP1J4R4uCCmy6F4seubB+sLkzuPr55CmIhE5qjH+kEN7gqVbVg UViU2865P4KjebpEXdy2l8FnoOCpTF1nmLQawe55bACVVIVWenLuOFcUSKyYyQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQkyX26xjz1RjW for ; Sun, 30 Mar 2025 19:36:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UJa4mG013593 for ; Sun, 30 Mar 2025 19:36:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UJa4QL013592 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 19:36:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 19:36:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #18 from Mark Millard --- (In reply to Wei Hu from comment #14) Going back to trying the VM for main's [15's] .vhdx . . . FYI: at the loader prompt: OK set console=3D"comconsole efi vidconsole" reports: console comconsole is unavailable console vidconsole is unavailable Also doing the other lines: OK set comconsole_speed=3D"115200" OK set boot_multicons=3D"YES" OK set boot_serial=3D"YES" and then: OK boot still has the same problem that I've reported. At least 14.* does not have the problem. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 20:00:44 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQlW072hbz5s05F for ; Sun, 30 Mar 2025 20:00:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQlW06Gfkz450S for ; Sun, 30 Mar 2025 20:00:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743364844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oAAUKRzc41ot7cDC3Iguuzd/WYo3WKmwRUi9lyFKZws=; b=Z8GuEIXXWFFkVrfMYJLVDZy30dV3DSyR78ehCGlL+YkKcSa86hbQta3SmGhgBgFHUZYiRt Me2Huf4AVkwY7IpHxLrQr2mCJIq2rxsTs9QUZDbXOrvSFWwXEUi2WrCWuqeZpU21sV33Wl NBPCQq9VpxhprClfyemrBbah8M4PdGQ05The65FV6LMMFnb7Y9XvqQ17ld/+QN5YAZDPDt MBfJFSD5n6iVB7WgD3XmFbace5738frPQbCLXaSYQxTRKISf/WFg4AHktC3l782I4gP1OZ jbavSevpgb2JaL5e4OtXM7XZFsRHqcER+yIaKWV33N6Sn9lbfZmCVHxuskXGBA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743364844; a=rsa-sha256; cv=none; b=LP+w+0+UnRkR8ZvIoPmk1NlKpIjMylPOVWVzMY2yuh+ujaZabFL5xCC4zJH7HSh8p4oKQ0 Cy8viRjqkifuZVGRpkVBQe18NPoNnbBaZlNHLgzZQjP04dJkl1qeUXcHuk5T2BkqJ2sMP4 V1v/Brcm0uT1R8eBxJ9MZBQlC2gR3XJtafhsvtk/nJ+a675uOyd2/MZemjwWyoD6gy+PCm NoUENolMDrwjTd57wxWcxexwkT2gqynp4PxG3Q0TcZbu93GJCyWa5scIumUPG2U9mHxUjo DrcgoEx2HChdaRFQA4e3IGstCkSX4h4AiebwVK2ykYjYQgrBcxWCEKCkGEq0hA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743364844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oAAUKRzc41ot7cDC3Iguuzd/WYo3WKmwRUi9lyFKZws=; b=gymVphehqg74b8hQsjkqhB41GT3pMMC1udd33diLZ36vjbF3J+6eeYnZFLrVWZiuzy+olk ljPPg1lBwrc+HrX4Xx3JCY5ROdIQ0pA2MwydhoGie5QyWTToCx2t/WfsIKFO3nASJPpPTn 5azDVNfHuCFIovcUPgG9Y9SqNxkU4Ker8M4NBX6BvXqNVYxJQyVKGlMIf8/z+BUHQ4aVA+ q3eaqMNV5tG2qrMnTMVN881X8ynLblkaWyOeCQgf3mB6xQg5xctOr/c7Z2kWXfs96Ue41r MLMy6/kFRq1FrABnUFgcoyjbOGVcWoE4DsCZBQ+dXnL/N+hYmTd9n74YCSA7zw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQlW05r2Cz1Rv7 for ; Sun, 30 Mar 2025 20:00:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UK0iga054103 for ; Sun, 30 Mar 2025 20:00:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UK0ikl054102 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 20:00:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 20:00:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #19 from Mark Millard --- (In reply to Mark Millard from comment #18) FYI comconsole for aarch64 was removed for 15+ by the following but the "shim" the commit references is only in place when: defined(__aarch64__) && __FreeBSD_version < 1500000 The commit is from 2023-May-11: From: Warner Losh Date: Thu, 11 May 2023 20:06:47 UTC The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=3Df93416d677432f3a713c71b79fb68e891= 62baca9 commit f93416d677432f3a713c71b79fb68e89162baca9 Author: Warner Losh AuthorDate: 2023-05-11 20:03:30 +0000 Commit: Warner Losh CommitDate: 2023-05-11 20:06:03 +0000 stand: add comconsole backwards compatibility shim for aarch64 Add a compat shim for the "comconsole" name so that people with a "console=3Dcomconsole" in their loader.conf on aarch64 will continue to work (though with a warning). This is only aarch64: it will never be there for amd64 (where comconsole always means talk to the hardware directly). To do that is too hard. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D39983 --- stand/efi/loader/conf.c | 7 +++++++ stand/efi/loader/efiserialio.c | 25 +++++++++++++++++++++++++ 2 files changed, 32 insertions(+) . . . --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 30 20:35:11 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQmHB1ShWz5s2XM for ; Sun, 30 Mar 2025 20:35:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQmH83n96z3Ty9 for ; Sun, 30 Mar 2025 20:35:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=XX1GQ31Q; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1743366926; bh=O6C8QwBuRaovQ9Ld6QrDOpNDiSyOV9GnbLiL+X4Gbow=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=XX1GQ31QT7kYW0Oen0BuESnE6ueEp1PA9kCBnaU6Qr09LUpmP+OkMOd9bXVCbzXOGPyoAeslsLqk6I4E4Bd74Bz7KjsMpqxjVywznupf6tCigspCSsVouTw58Ok4RPnliGlqjf0qqiA8tNsrg/7aOIABfTUdVzImH+N0q9bii6toeCP0eHltr2j1pY8iqD4GRd5Db07rfh2obdungXr05m/w0u1Jw6kshe02AycapXqEkpYfKOeZP6x5UTUcPkzqBd8ZeLR/HzJYy4IubHhEpfVLbTYZWNkke5DfaXPnwgBQjIXWE8444CEi7Pk+jXkPdD8dffKD41TDsqmy8Yb3iA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1743366926; bh=q3FKzSlnnrScGa9HRpI1ODkCLhfsUjtcAHqXyrSwbQL=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=TKuQFyn9lCc79Iv5zQAhti2qYpF6GKyuNkWhWt8ymbysXHS9MWgqmyFSo5NSGF/wxSJlozee3/tx7rRdmP+7lndRK5Ic2/GjIUPPMnFUdxklecmMHtz5BX+TuLUfI3A0+PqKn5z1kYsYautnuA97Dbn+aKMQCX4zt+/zF0vdn6wu+uhqyHf/weMqwR0clF1xll+bH8H56V9KDSTOA3gg0rV1ujFrI0Z3FZ+yUJBOXfhwYimoxUz+3HEFvuuPVB7NaID1sUQ2f6K4KPMB9dwjMakeBPUaBzLiTegHzN/sg9R9x3s5wb8qHcC0YciIxiMtYNQoIk3ApeOTRXfIFIO6PQ== X-YMail-OSG: m0djFDQVM1mp0UhFU3yVVEWUJkq.Ug8rZSPiLMuCNNtJqoGv7LGtXqMBa7qYuj. U8a3Yc5gLpxD08OAQzTjsZ8RNlqo7FLpCMMw87EsxAL49cwU59d174d0R3zL4mXt42KcQOhVlk6G FwzaM16O_3RnoHD1.sQOOEQ9gzQpZIJcTV5k7uNlwQIOm.B7U.rt2mNNEMNLxN7b4ru4dZDZvdzU h15BtLVPEibZWj2IQTBCUOxOhkt9hxf1AjCBiVBl._cXcN5oltPNJc4JtS2m3F6AgjOq3r5Xf3hN eVkq_BJCEF4sCuGQAFyNDVfGtvLoQq2.NktpIkft38MzPFjdVvuPlB30Zp0i7zBDOTKz1wJCpj8I Jas7tMMz5JJVp3OZGI5J9pMTgVQITi2qX.4CTuwJzF1C37CKRvDpbruFOiwZOfdhoEMQ_qoieZZp xd9sJCnFP3aqT6eK4u0fr8emhTigWjUCrjTuTd_PaTMNMvWF2Q0vKggOoA6zoOuP.Ga9D4jtXhlL Ky_zyyZGZ8EASvOwqz8NO4qaTZ5r.lWtqMZ8SVKum0EbKcoGPxNHC..spuhpiGvVMNeQAA.2as4R kxc69v2th2KlownEtvUOoD0FCj7KjBF7f9R3RJdoY1ftA614uJ6PEQ2O.AxFIHEF9SXTAwx0Lv__ dQ1tKqce.i8.W3uMuvpWlFobrlguVH9iqKxn9cj2.36QbPk1fiT.imEiIGi5OdCLl0ZxgSEV_4vP A4zwX1fMGUZhwkFOvW1Qr1yxiHO1NKK9y_l3DZbPc4YeN98bxzP2TOodnnvmO9mRXqfbzfbFaEFQ GombdrSM1YGhHZbmdvai0rumvCWTSXAy38bMMksW8bcuG5PrClouwRuYKM8oBxX4G3eYfda4Hnzl 8yU1Idh7AfnvLXQcmcEWSQJlxIUPNBHv0tUJem7gRmFHAxffh.IB16ffXxXHA4NuVxVBsKVhwscP 7cC8AhOwHLEu6ssoPMaW8zUTFZuqihmTq8F.L275LruWUAIJ6mibbzvzqm0L8f2V2MjVmTAKgAGK EOIY.j9BffwrAo.dap_6gP5EV_ncV6Hv5yQzakp7mGyiRICxr1qKRQkz.qjVaU7AVPjQ3vG9_4G_ yZO6AFepo795C68crWjv5aSMI8QERrQYUo9GcpN39VKGLdwiDGawI945aCu_MpHK3omDoO92ftgC BOaTw8a8aijo6BTx6WYaxApNXcmrnApNuyx.vqVj.TPn0GtFL4XndCmloGb78d6mqraJdsTU3l3B WVclVlhcFCDiyLBbj9KSHu1mRBYWmqvLmNMhQ.KEDQId_SVl4pvRVIoveZbW6Suk5WLSpVZMu3XN uuHbBGBuC1TonE87.V4JBqPRjcBBnKY.B2oa6NhJSa0hsTWl43jxU99iI091gSYFqP6fU7YXfUEl LQt23jCq_oOuFKl6pCY_iG_qBJ3_7QLFMnBpRzgU4lTi7cGdOcaUvzQpE9lmbvvg8ojSw3.GDB_i uhQuGqsxtznqxotZ05ghM3YTG7wEsHS2BcL8QdQH9KSmmonHCE_jgaJn9IYXcwSFDd86fardXjIO YL4whhHobVUbllPg5RwgiX.qrjDe5KtQUrWwZj0uXm.OfeKp0hRRMPsEiIwO3glgq5yQKhK6sX81 uFFC2MKZDQprgywwhPTV3xWbrpXQyUP7FVFgZaDmMMQ791PpcCJOX47FE25voc7dGKaSIxRuk2dh btjqJA3tZnxYLPHcRurKIEB3aCXxKNJH.Z_CtTjIjQcCCL3VYCbk_zuzI2ejdFGBb8pB_dW6k7B8 GWHxVqI8n5YqoivkODnziolRaNxE0io8Zt36m8Ql0zgypSPHDgemEbEwylSbjtG55do3KcovrA3e L6u6JMlQXYs7YpT7RwwBDbDvrsgDeHQ_0ZGDFh6pMblWBQCDKUA29OkWJtrHjoI4qahEJ0i_6KuG aSwo4sAv0AyqP321pl7cagNWeAQ6C54iPu9pPvZro4EcAduDLl_kd.K54sTrLLK9fxU9n6iQshUc b77LPn1.FP95tz7.AVl0ZuuhR602MXz7WkGLvc1eu7fFZagbmTyhgQa6MUIEBmqUs3P4cTWsfYxG JZ1kBuFkZJuK2hSEk0ml0RHrmoFnfUq2AXcORKh4FH.t42IPhRZHXo_N3LLFi_88nhKIAcH0ySjr f1squCW8qKDrF60..NEN87vmZHZgXGfqtBUouda7M3R3Oc0T.qAp5RgjX07kU4kMUmLK9XTT2nEX VLgLzIOmj4cql1U5m_0Bc2bik8lybRVPNrg2pNvzdIy2OohX4JqsK1uQFGKvXXElRgdhE1b_kzEN 5auJXjOmG_uCSSgFOLQW3 X-Sonic-MF: X-Sonic-ID: b2a6b56e-591b-4d71-84c0-085a37ff2536 Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sun, 30 Mar 2025 20:35:26 +0000 Received: by hermes--production-gq1-5c477bf655-9n48k (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 6b6bae667220c15e81d191b2abfcb110; Sun, 30 Mar 2025 20:35:22 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.400.131.1.6\)) Subject: FreeBSD-15.0-CURRENT-arm64-aarch64-zfs.vhd.xz fails to boot under Hyper-V v2 on Windows 11 Pro; possibly tied to: "stand: add comconsole backwards compatibility shim for aarch64" Message-Id: <0B5B79D3-A51D-43E8-9B96-2E89B787CEC0@yahoo.com> Date: Sun, 30 Mar 2025 13:35:11 -0700 To: Warner Losh , weh@microsoft.com, FreeBSD Current , FreeBSD ARM List , Virtualisation on FreeBSD X-Mailer: Apple Mail (2.3826.400.131.1.6) References: <0B5B79D3-A51D-43E8-9B96-2E89B787CEC0.ref@yahoo.com> X-Spamd-Result: default: False [-2.63 / 15.00]; RBL_SENDERSCORE_REPUT_9(-1.00)[98.137.64.84:from]; NEURAL_HAM_LONG(-0.99)[-0.991]; NEURAL_HAM_SHORT(-0.84)[-0.841]; NEURAL_SPAM_MEDIUM(0.70)[0.702]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.84:from]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.84:from]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-Rspamd-Queue-Id: 4ZQmH83n96z3Ty9 X-Spamd-Bar: -- In order to try to help test: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 I again tried to get main FreeBSD booting on the Windows DevKit 2023 under Windows 11 Pro, this time using: FreeBSD-15.0-CURRENT-arm64-aarch64-zfs.vhd.xz (expanded to .vhd and then the .vhd converted to .vhdx since aarch64 Hyper-V only supports v2 VM's and those have to use .vhdx format, which is not directly available). It fails as described in: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681#c9 In part (implicit context: use of main, as is normal for my context): QUOTE I've never been able to get FreeBSD to complete much of the boot sequence under Hyper-V on the Windows DevKit 2023 (aarch64) that has Windows 11 Pro. The console output stops after the masks line of the EFI framebuffer information or somewhat later. The farthest I've seen is Event Timer line from the kernel output. It has stopped between those points otherwise. No failure notices when ti stops: just no more output. END QUOTE Also, the boot does stop, with Hyper-V indicating a not-ready status for attempts to shutdown from Hyper-V. I have to have Hyper-V just stop the VM. I tested examples from: https://download.freebsd.org/ftp/releases/VM-IMAGES/*-*/aarch64/Latest/ and all 3 of the 14.* that I tested worked: FreeBSD-14.2-STABLE-arm64-aarch64-ufs.vhd.xz FreeBSD-14.2-RELEASE-arm64-aarch64-ufs.vhd.xz FreeBSD-14.1-RELEASE-arm64-aarch64-ufs.vhd.xz Just: FreeBSD-15.0-CURRENT-arm64-aarch64-zfs.vhd.xz fails of what I tested. I'll note that comconsole for aarch64 was removed for 15+ by the following and the "shim" the commit references is only in place when: defined(__aarch64__) && __FreeBSD_version < 1500000 The commit is from 2023-May-11: From: Warner Losh Date: Thu, 11 May 2023 20:06:47 UTC The branch main has been updated by imp: URL: = https://cgit.FreeBSD.org/src/commit/?id=3Df93416d677432f3a713c71b79fb68e89= 162baca9 commit f93416d677432f3a713c71b79fb68e89162baca9 Author: Warner Losh AuthorDate: 2023-05-11 20:03:30 +0000 Commit: Warner Losh CommitDate: 2023-05-11 20:06:03 +0000 stand: add comconsole backwards compatibility shim for aarch64 Add a compat shim for the "comconsole" name so that people with a "console=3Dcomconsole" in their loader.conf on aarch64 will continue to work (though with a warning). This is only aarch64: it will never be there for amd64 (where comconsole always means talk to the hardware directly). To do that is too hard. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D39983 --- stand/efi/loader/conf.c | 7 +++++++ stand/efi/loader/efiserialio.c | 25 +++++++++++++++++++++++++ 2 files changed, 32 insertions(+) . . . =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Mar 30 23:07:09 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZQqf54Kh2z5sDZX for ; Sun, 30 Mar 2025 23:07:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZQqf53dw0z3QXd for ; Sun, 30 Mar 2025 23:07:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743376029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nsT5m8PzxVSMJtbPOkbXNy8opPAKEI/qe2ulKb51dUc=; b=AuiAlNpOPol9xMImpAYqJTPzxbPNflNNOOmusYCYc/zsIsQ8o99+7r0XmJ7KYvkj9540Tw XsxemiIb/JtM6Z7ejZuHPuDY1bUXSZT1pmcH7Y2c41wARi29yT7IcxYyZgNXG3s4n6mbsw UBMP5vPbTfc4V29I5ZOP8vFgAJsfEdFf5CS6/a/u2VTmBRsRSkFwU0iQRGdMqjJJzmdEGl oAGVoyTaXymIAFCGKXI7pQnoz2Y93769oHgrgnEysS3O0UF01NMdPDWsGAldTaVGPuGiGP cLS/fBMw0GHUSrq47pHwowaFV61N+qYDJma83kJsA3YrqnSCMcNIiD28LIWwnA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743376029; a=rsa-sha256; cv=none; b=mjzB3eAW5FEEXhxTCCmHF3YJ9iZ1OvAMF6X6dWggrVsDkpIDI8BnsEpvXpldO2Q81MYsvs McGi7Ko6oMqIv0cxwafR2J3NPTlWyw5QdRcgInXS0Uh9eBQWWQ6pfy+57T9JLgeDI8AJt1 qlF3M72aEi7Kz2YttzPVocMFpp35xgwctQD/pKh20R/RqTBNAjYcPT7IDxc7T4zxnfeDAJ anVAvtcTBRVgJSWut30Fl48uJF13LLryMbyC+6rPUvJQILt+zTduRJtW5gNtyEHCYQf1Bh sMYvEvT5BPBWx234B7iEZBy9jVcTA8+qpn0ZouwhOEgppjoAdyDBVqHJlvsFlg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743376029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nsT5m8PzxVSMJtbPOkbXNy8opPAKEI/qe2ulKb51dUc=; b=s3AvwK4xnKvJ4keqUjgCcc5imC/L6pBpp+vnn9KvO5Tii2G0lCWVm6jIVcpRaHKfDwukMc Q0su0syPfcRsgLvX/XVQMd+y4FJIOEZ+pNiDJNKcRk5x3P9n6nLAKYqxlGGQLlQxYE7uSm j1MQJigDi7rVGN1jqPdvYEN50M6n/7k60sXh8JURILyIiX/d7QpDkTO0z39Yku+YkkAxH8 STcVoWQzC1WUvDc2WqVcXinDF0LheNB4uzZGe7oRlRyDeVtHFQsDs8DaqpTEFpwCpF7JO0 0hLr4l4g06nwrh1W+H//4WnFfkFXn0+wE0qD9k8LSuncNVmwRTYmCB61adAJ/A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZQqf52sD5z4Ml for ; Sun, 30 Mar 2025 23:07:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 52UN79SL095882 for ; Sun, 30 Mar 2025 23:07:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 52UN79fS095881 for virtualization@FreeBSD.org; Sun, 30 Mar 2025 23:07:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 285681] [Hyper-V] i386 panic during storvsc_xferbuf_prepare() Date: Sun, 30 Mar 2025 23:07:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: marklmi26-fbsd@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285681 --- Comment #20 from Mark Millard --- (In reply to Mark Millard from comment #19) Context: the problem with main booting that is preventing getting far enough to help test. FYI: substitution of the bootaa64.efi from the stable/14 based VM into the main based VM is not enough to avoid the problem in main, even with the likes of using: OK set console=3D"comconsole" OK set comconsole_speed=3D"115200" OK set boot_multicons=3D"YES" OK set boot_serial=3D"YES" OK boot Somehow main's FreeBSD kernel code is involved. May be there is common code between the loader and the kernel so that the substitution is incomplete? --=20 You are receiving this mail because: You are the assignee for the bug.=