From nobody Mon Sep 22 00:20:11 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVNzp1gdVz68Ds0 for ; Mon, 22 Sep 2025 00:20:22 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-24422.protonmail.ch (mail-24422.protonmail.ch [109.224.244.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVNzm0vHjz3xNW; Mon, 22 Sep 2025 00:20:19 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=VPvwhal8; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 109.224.244.22 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758500416; x=1758759616; bh=AxXIWcoBwHrGqgDuq4kDqj+T32LQKXwC7iRcrPVVFT4=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=VPvwhal8L4en1SYZUgJHwsfWIBnfJ1NgM534HiSsSrQQYXlXGAY+skTbp5vmUmMbu K17KaXRIvznwtu1zDBJ7Eze/2qr59biwqi8QhO1o+NEeUm4ldfN5IkXBpVVFX3xgP+ ieadfGlJEzW4+C5eNshVTdtt9Ae+ofaXGH7erUDoR3XP3Q6MJKV6bxfxTRqE9aaSsg nXLEy92k3rrnhOr/+hcHeuKbuMinmWyc8hNp04hCGlAJYd8MeUuhdPVdg7+ItgDq+C oJ/BpXz4esPgshNBKbUW+3F6t4GZi4O1ivNyeCoO1xm2tFzDX1HtY29v5gu7CvshWN 5W9C+ALE8YFYQ== Date: Mon, 22 Sep 2025 00:20:11 +0000 To: "virtualization@freebsd.org" From: Jonathan Vasquez Cc: Nuno Teixeira , Tomek CEDRO , =?utf-8?Q?Corvin_K=C3=B6hne?= Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: =?us-ascii?Q?<6CV-OY6BcErrWRit9jSpi6fWsYBG3E=5FZ3u6eTLPcz6foPAZV1gQpZYaZTR7JA=5F1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=3D@xyinn.org>________?= Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 505b7751c9d9b8b9ef78c8568761200b93087987 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1=_OdLJLr8sUAwsmR6QT9puQVzBTbPy2TZJZy1FfVY" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.28 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.976]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[109.224.244.22:from]; R_SPF_ALLOW(-0.20)[+ip4:109.224.244.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_BASE64_TEXT(0.10)[]; FREEFALL_USER(0.00)[jon]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[109.224.244.22:from]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:62371, ipnet:109.224.244.0/24, country:CH]; RCVD_COUNT_ZERO(0.00)[0]; RCPT_COUNT_THREE(0.00)[4]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cVNzm0vHjz3xNW --b1=_OdLJLr8sUAwsmR6QT9puQVzBTbPy2TZJZy1FfVY Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SGV5IGFsbCwKCkp1c3Qgd2FudGVkIHRvIHJlcG9ydCB0aGF0IEkgc3RpbGwgaGF2ZW4ndCByZWNl aXZlZCBvbmUgY3Jhc2ggYWZ0ZXIgbXkgYWRqdXN0bWVudHMgeWVzdGVyZGF5LiBTbyBmYXIgb3Zl ciAyMCsgaG91cnMgb2YgdXB0aW1lfi4gSSd2ZSBhbHNvIHJlYm9vdGVkIHRoZSBWTS9ob3N0IGEg ZmV3IHRpbWVzIGp1c3QgdG8gZW5zdXJlIHRoYXQgaXQgaXMgZGV0ZXJtaW5pc3RpY2FsbHkgc3Rh YmxlIGFuZCBzbyBmYXIgemVybyBpc3N1ZXMuIFNvIHRoaXMgaXMgcG9pbnRpbmcgaW4gYSBwb3Np dGl2ZSBkaXJlY3Rpb24uIEknbSB0ZW1wdGVkIHRvIG1hcmsgdGhpcyBhcyBhIHN0YWJsZSBzZXQg dXAgYnV0IEknbGwgbmVlZCB0byBwbGF5IGZvciBsb25nZXIgdG8gbWFrZSB0aGF0IGNhbGwuCgpK b25hdGhhbiBWYXNxdWV6ClBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBIEQ0OUYgRkI4NSA5 MEI3IEM0Q0EgNTI3OQpTZW50IHdpdGggUHJvdG9uTWFpbCBTZWN1cmUgRW1haWwKCk9uIFN1bmRh eSwgU2VwdGVtYmVyIDIxc3QsIDIwMjUgYXQgMDg6MjMsIEpvbmF0aGFuIFZhc3F1ZXogPGpvbkB4 eWlubi5vcmc+IHdyb3RlOgoKPiBTeXN0ZW0gaGFzIGJlZW4gdXAgZm9yIGFib3V0IDloNDBtLiBN b3N0IG9mIGl0IGlkbGUgYW5kIG5vIGNyYXNoZXMgeWV0LiBJJ20gZ29ubmEgcmVib290IHRoZSBo b3N0IGFuZCBzZWUgaWYgdGhlIHN0YWJpbGl0eSBzdGlsbCBjb250aW51ZXMuCj4KPiBKb25hdGhh biBWYXNxdWV6Cj4gUEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgRDQ5RiBGQjg1IDkwQjcg QzRDQSA1Mjc5Cj4gU2VudCB3aXRoIFByb3Rvbk1haWwgU2VjdXJlIEVtYWlsCj4KPiBTZW50IGZy b20gW1Byb3RvbiBNYWlsXShodHRwczovL3Byb3Rvbi5tZS9tYWlsL2hvbWUpIGZvciBBbmRyb2lk Lgo+Cj4gLS0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLS0tLQo+IE9uIFNhdHVyZGF5LCAw OS8yMC8yNSBhdCAyMzo0NSBKb25hdGhhbiBWYXNxdWV6IDxqb25AeHlpbm4ub3JnPiB3cm90ZToK Pgo+PiBKdXN0IGZpbmlzaGVkIGJlYXRpbmcgdGhlIG1haW4gc3RvcnkgbGluZSBpbiBDeWJlcnB1 bmsgMjA3NyBpbiB0aGUgVk0uIFNvIGZhciBubyBjcmFzaGVzIGFuZCBpdCdzIGJlZW4gcnVubmlu ZyBmb3IgYW4gaG91ci4gU28gdGhpcyBpcyBhIHBvc2l0aXZlIHNpZ25hbC4uLiB3ZSdsbCBzZWUg d2hhdCBoYXBwZW5zLiBJJ2xsIGxlYXZlIHRoZSBWTSBydW5uaW5nIGlkbGUgb3ZlciBuaWdodCBh bmQgc2VlIGlmIGl0J3Mgc3RpbGwgcnVubmluZyBpbiB0aGUgbW9ybmluZy4gU3RheSBzYWZlIGFs bCEKPj4KPj4gSm9uYXRoYW4gVmFzcXVlego+PiBQR1A6IDM0REEgODU4QyAxNDQ3IDUwOUUgQzc3 QSBENDlGIEZCODUgOTBCNyBDNENBIDUyNzkKPj4gU2VudCB3aXRoIFByb3Rvbk1haWwgU2VjdXJl IEVtYWlsCj4+Cj4+IE9uIFNhdHVyZGF5LCBTZXB0ZW1iZXIgMjB0aCwgMjAyNSBhdCAyMjoyOCwg Sm9uYXRoYW4gVmFzcXVleiA8am9uQHh5aW5uLm9yZz4gd3JvdGU6Cj4+Cj4+PiBJIGRpZCBhIGJ1 bmNoIG9mIG1vcmUgZXhwZXJpbWVudHMgdG9kYXkgYW5kIEkgd2FzIGFibGUgdG8gc3dpdGNoIGZy b20gdXNpbmcgdGhlIFVTQiBjb250cm9sbGVyIGNvbm5lY3RlZCB0byB0aGUgbXVsdGktZnVuY3Rp b25hbCBidXMgdGhhdCBoYXMgdGhlIGludGVncmF0ZWQgY2FyZCwgYW5kIHN3aXRjaGVkIHRvIGEg c2VwYXJhdGUgVVNCIGNvbnRyb2xsZXIgdGhhdCBvbmx5IGhhcyBVU0IgcG9ydHMuIEknbSBob3Bp bmcgdGhhdCB0aGlzIGluY3JlYXNlcyBWTSBzdGFiaWxpdHkgYW5kIHRoYXQgdGhlcmUgbWF5IGhh dmUgYmVlbiBzb21lIHdpcmVzIGJlaW5nIGNyb3NzZWQuIFRoaXMgd2FzIHRoZSBwcmV2aW91cyBi dXMgSSB3YXMgdXNpbmcgKEkgY291bGRuJ3QgaXNvbGF0ZSBqdXN0IHRoZSBVU0IgZnVuY3Rpb25z LCBpdCBoYWQgdG8gYmUgYWxsIG9yIG5vdGhpbmcgZm9yIGl0IHRvIGZ1bmN0aW9uKToKPj4+Cj4+ PiB2Z2FwY2kwQHBjaTA6MTg6MDowOiBjbGFzcz0weDAzMDAwMCByZXY9MHhjMSBoZHI9MHgwMCB2 ZW5kb3I9MHgxMDAyIGRldmljZT0weDE2NGUgc3VidmVuZG9yPTB4MTA0MyBzdWJkZXZpY2U9MHg4 ODc3Cj4+PiB2ZW5kb3IgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywgSW5jLiBbQU1EL0FUSV0n Cj4+PiBkZXZpY2UgPSAnUmFwaGFlbCcKPj4+IGNsYXNzID0gZGlzcGxheQo+Pj4gc3ViY2xhc3Mg PSBWR0EKPj4+IGhkYWMwQHBjaTA6MTg6MDoxOiBjbGFzcz0weDA0MDMwMCByZXY9MHgwMCBoZHI9 MHgwMCB2ZW5kb3I9MHgxMDAyIGRldmljZT0weDE2NDAgc3VidmVuZG9yPTB4MTA0MyBzdWJkZXZp Y2U9MHg4ODc3Cj4+PiB2ZW5kb3IgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywgSW5jLiBbQU1E L0FUSV0nCj4+PiBkZXZpY2UgPSAnUmVtYnJhbmR0IFJhZGVvbiBIaWdoIERlZmluaXRpb24gQXVk aW8gQ29udHJvbGxlcicKPj4+IGNsYXNzID0gbXVsdGltZWRpYQo+Pj4gc3ViY2xhc3MgPSBIREEK Pj4+IG5vbmUxQHBjaTA6MTg6MDoyOiBjbGFzcz0weDEwODAwMCByZXY9MHgwMCBoZHI9MHgwMCB2 ZW5kb3I9MHgxMDIyIGRldmljZT0weDE2NDkgc3VidmVuZG9yPTB4MTA0MyBzdWJkZXZpY2U9MHg4 ODc3Cj4+PiB2ZW5kb3IgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywgSW5jLiBbQU1EXScKPj4+ IGRldmljZSA9ICdGYW1pbHkgMTloIFBTUC9DQ1AnCj4+PiBjbGFzcyA9IGVuY3J5cHQvZGVjcnlw dAo+Pj4geGhjaTFAcGNpMDoxODowOjM6IGNsYXNzPTB4MGMwMzMwIHJldj0weDAwIGhkcj0weDAw IHZlbmRvcj0weDEwMjIgZGV2aWNlPTB4MTViNiBzdWJ2ZW5kb3I9MHgxMDQzIHN1YmRldmljZT0w eDg4NzcKPj4+IHZlbmRvciA9ICdBZHZhbmNlZCBNaWNybyBEZXZpY2VzLCBJbmMuIFtBTURdJwo+ Pj4gZGV2aWNlID0gJ1JhcGhhZWwvR3Jhbml0ZSBSaWRnZSBVU0IgMy4xIHhIQ0knCj4+PiBjbGFz cyA9IHNlcmlhbCBidXMKPj4+IHN1YmNsYXNzID0gVVNCCj4+PiB4aGNpMkBwY2kwOjE4OjA6NDog Y2xhc3M9MHgwYzAzMzAgcmV2PTB4MDAgaGRyPTB4MDAgdmVuZG9yPTB4MTAyMiBkZXZpY2U9MHgx NWI3IHN1YnZlbmRvcj0weDEwNDMgc3ViZGV2aWNlPTB4ODg3Nwo+Pj4gdmVuZG9yID0gJ0FkdmFu Y2VkIE1pY3JvIERldmljZXMsIEluYy4gW0FNRF0nCj4+PiBkZXZpY2UgPSAnUmFwaGFlbC9HcmFu aXRlIFJpZGdlIFVTQiAzLjEgeEhDSScKPj4+IGNsYXNzID0gc2VyaWFsIGJ1cyBzdWJjbGFzcyA9 IFVTQgo+Pj4KPj4+IEFsdGhvdWdoIG5vdyBJJ20gbG9uZ2VyIHVzaW5nIHRoaXMgYXQgYWxsIGZv ciBVU0IgZnVuY3Rpb25hbGl0eSBpbiB0aGUgVk0uIEkndmUgc3dpdGNoZWQgdG86Cj4+Pgo+Pj4g cHB0MkBwY2kwOjEzOjA6MDogY2xhc3M9MHgwYzAzMzAgcmV2PTB4MDEgaGRyPTB4MDAgdmVuZG9y PTB4MTAyMiBkZXZpY2U9MHg0M2Y3IHN1YnZlbmRvcj0weDFiMjEgc3ViZGV2aWNlPTB4MTE0Mgo+ Pj4gdmVuZG9yID0gJ0FkdmFuY2VkIE1pY3JvIERldmljZXMsIEluYy4gW0FNRF0nCj4+PiBkZXZp Y2UgPSAnNjAwIFNlcmllcyBDaGlwc2V0IFVTQiAzLjIgQ29udHJvbGxlcicKPj4+IGNsYXNzID0g c2VyaWFsIGJ1cyBzdWJjbGFzcyA9IFVTQgo+Pj4KPj4+IE11Y2ggY2xlYW5lciBhbmQgc3RpbGwg Y29udGFpbnMgYSB2YXJpZXR5IG9mIGhpZ2ggc3BlZWQgcG9ydHMuIEFsdGhvdWdoIHVsdGltYXRl bHkgSSdtIGp1c3QgdXNpbmcgb25lIG9mIHRoZW0gYmVjYXVzZSBJIGhhdmUgYSBLVk0gY29ubmVj dGVkIHRvIGl0IHRoYXQgYWxsb3dzIG1lIHRvIHBsdWcgaW4ga2V5Ym9hcmQvbW91c2UvMy41bW0g amFjayAoYXVkaW8pL2V0aGVybmV0IGFsbCB0aHJvdWdoIG9uZSBVU0Igd2lyZS4gQWx0aG91Z2gg dG8gcmVtaW5kIGV2ZXJ5b25lLCBJJ20gZ2FtaW5nIGluIGFuIG9mZmxpbmUgY2FwYWNpdHkgdG8g SSBpbnRlbnRpb25hbGx5IG1lc3MgdXAgdGhlIElQdjQgZ2F0ZXdheSBzbyBteSBWTSBjYW4gb25s eSBjb21tdW5pY2F0ZSBpbnRlcm5hbGx5IChhbmQgSSBjYW4gc2lkZWxvYWQgYXBwcy9nYW1lcykg YnV0IHByZXZlbnQgTWljcm9zb2Z0IGZyb20gcmVhY2hpbmcgbXkgbWFjaGluZSBhbmQgY29sbGVj dGluZyB0ZWxlbWV0cnkgaW5mbywgYW5kIGRvaW5nIGZvcmNlZCB1cGRhdGVzLiBQbHVzLCBJIGhh dmUgbm8gaW50ZW50aW9uIG9mIG1vdmluZyB0byBXaW5kb3dzIDExIHdpdGggYWxsIG9mIHRoZWly IG5ldyByZXF1aXJlbWVudHMgYW5kIGZvcmNlZCBvbmxpbmUgYWN0aXZhdGlvbiAoSSBrbm93IG9m IHRoZSB3b3JrYXJvdW5kcywgYnV0IEknbSBhZ2FpbnN0IG9uIG1hbmRhdGluZyBhIE1TQSBvbiBw cmluY2lwbGUpLiBQZW9wbGUgc2hvdWxkIGJlIGFibGUgdG8gZnVsbHkgdXNlIHRoZWlyIG1hY2hp bmVzIGV2ZW4gZHVyaW5nIGluc3RhbGxhdGlvbi9PT0JFIHdpdGhvdXQgaGF2aW5nIHRvIGF1dGhl bnRpY2F0ZSB0byBhIGNlbnRyYWwgc2VydmVyLiBBbnl3YXlzLCBzbWFsbCByYW50IG92ZXIgbG9s Lgo+Pj4KPj4+IEknbGwgdXBkYXRlIGV2ZXJ5b25lIG9uIG15IHByb2dyZXNzIHdpdGggdGhpcyBy ZS1jb25maWd1cmF0aW9uLgo+Pj4KPj4+IEpvbmF0aGFuIFZhc3F1ZXoKPj4+IFBHUDogMzREQSA4 NThDIDE0NDcgNTA5RSBDNzdBIEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3OQo+Pj4gU2VudCB3aXRo IFByb3Rvbk1haWwgU2VjdXJlIEVtYWlsCj4+Pgo+Pj4gT24gU2F0dXJkYXksIFNlcHRlbWJlciAy MHRoLCAyMDI1IGF0IDIwOjAyLCBKb25hdGhhbiBWYXNxdWV6IDxqb25AeHlpbm4ub3JnPiB3cm90 ZToKPj4+Cj4+Pj4gSGV5IE51bm8sCj4+Pj4KPj4+PiAtIEknbSBub3QuIEkgcGx1Z2dlZCBpbiBt eSBtb25pdG9yJ3MgSERNSSBjYWJsZSBkaXJlY3RseSBpbnRvIHRoZSBHUFUncyBIRE1JIHBvcnQu IE9uY2UgYmh5dmUgc3RhcnRzIHVwIChpZiB5b3VyIHBhc3N0aHJvdWdoIGlzIHdvcmtpbmcgY29y cmVjdGx5IGFuZCB5b3VyIEdQVSdzIGRyaXZlcnMgYXJlIGluc3RhbGxlZCksIGl0IHdpbGwgb3V0 cHV0IHRvIHlvdXIgbW9uaXRvci4gWW91IHdvdWxkIG9ubHkgbmVlZCBWTkMgZm9yIGluaXRpYWwg aW5zdGFsbGF0aW9uIG9mIFdpbmRvd3MuCj4+Pj4gLSBJJ20gbm90IHN1cmUgb2YgdGhlIE5WSURJ QSBjYXNlLiBJIGRvIGtub3cgdGhlcmUgaXMgc29tZW9uZSB3aG8gYmxvZ2dlZCBhYm91dCBob3cg dGhleSBnb3QgdGhlaXJzIHdvcmtpbmc6IGh0dHBzOi8vZGZsdW5kLnNlL35nZXR6L05vdGVzLzIw MjQvZnJlZWJzZC1ncHUvCj4+Pj4KPj4+PiBBdG0gSSdtIGN1cnJlbnRseSB0cnlpbmcgdG8gY29s bGVjdCBtb3JlIGluZm8gb24gd2h5IG15IGJoeXZlIFZNIHJhbmRvbWx5IGNyYXNoZXMgc29tZXRp bWVzLiBTbyBmYXIgSSd2ZSBjb2xsZWN0ZWQgdGhhdCB0aGUgZXhpdCByZWFzb24gYmh5dmUgaXMg c2hvd2luZyBpcyAweDYwLCB3aGljaCBwb2ludHMgdG8gdGhlIGZvbGxvd2luZyBlbnRyaWVzIGlu IHRoZSBhbWQgZnJlZWJzZCBrZXJuZWwgc291cmNlczoKPj4+Pgo+Pj4+IDB4NjAKPj4+PiBodHRw czovL2dpdGh1Yi5jb20vZnJlZWJzZC9mcmVlYnNkLXNyYy9ibG9iLzRjM2E4NjhkMTNjMDUzZWYx NzMyNjhjZGZlMTM2NTk3OGEyODIxNzgvc3lzL2FtZDY0L3ZtbS9hbWQvdm1jYi5oI0wxOTUKPj4+ PiBodHRwczovL2dpdGh1Yi5jb20vZnJlZWJzZC9mcmVlYnNkLXNyYy9ibG9iLzRjM2E4NjhkMTNj MDUzZWYxNzMyNjhjZGZlMTM2NTk3OGEyODIxNzgvc3lzL2FtZDY0L3ZtbS9hbWQvdm1jYi5oI0wx MjhDMS1MMTI4QzMwCj4+Pj4KPj4+PiBYU0VUQlYKPj4+PiBodHRwczovL2dpdGh1Yi5jb20vZnJl ZWJzZC9mcmVlYnNkLXNyYy9ibG9iLzRjM2E4NjhkMTNjMDUzZWYxNzMyNjhjZGZlMTM2NTk3OGEy ODIxNzgvc3lzL2FtZDY0L3ZtbS9hbWQvdm1jYi5oI0w5NAo+Pj4+Cj4+Pj4gSSBhbHNvIGFza2Vk IFByb3RvbidzIEx1bW8rIEFJIHdoYXQgaXMgMHg2MCB0byBzZWUgd2hhdCBpdCBjb21lcyB1cCB3 aXRoLCBpdCBzYWlkIHRoZSBmb2xsb3dpbmc6Cj4+Pj4KPj4+PiBleGl0X3JlYXNvblswXSA9IDB4 NjAgaXMgdGhlIFZN4oCRZXhpdCBjb2RlIHRoYXQgYmh5dmUgcmVwb3J0cyB3aGVuIHRoZSBndWVz dCB0cmlnZ2VycyB0aGUgWFNFVEJWIGluc3RydWN0aW9uLgo+Pj4+IFdoeSBpdCBoYXBwZW5zCj4+ Pj4KPj4+PiBYU0VUQlYgd3JpdGVzIHRvIHRoZSBleHRlbmRlZCBjb250cm9sIHJlZ2lzdGVyIFhD UjAuCj4+Pj4gSW4gSW50ZWzigK9WTVggKHRoZSB2aXJ0dWFsaXphdGlvbiB0ZWNobm9sb2d5IGJo eXZlIHVzZXMgb24geDg2KSwgZXhlY3V0aW9uIG9mIFhTRVRCViBpcyBub3QgYWxsb3dlZCB0byBy dW4gZGlyZWN0bHkgaW4gdGhlIGd1ZXN0OyB0aGUgQ1BVIGZvcmNlcyBhIFZN4oCRZXhpdCBzbyB0 aGUgaHlwZXJ2aXNvciBjYW4gdmFsaWRhdGUgdGhlIG9wZXJhdGlvbi4KPj4+PiBiaHl2ZSByZWNv cmRzIHRoZSBleGl0IHJlYXNvbiBhcyAweDYwLCB3aGljaCBtYXBzIHRvIHRoZSBWTVggZXhpdCBy ZWFzb24g4oCcWFNFVEJW4oCdIGluIHRoZSBJbnRlbCBzcGVjaWZpY2F0aW9uLgo+Pj4+Cj4+Pj4g V2hhdCBpdCBtZWFucyBmb3IgeW91Cj4+Pj4KPj4+PiBUaGUgZ3Vlc3QgdHJpZWQgdG8gZW5hYmxl IG9yIGNoYW5nZSBBVljigJE1MTIgLyBYU0FWRSBmZWF0dXJlIGJpdHMgdmlhIFhTRVRCVi4KPj4+ PiBCZWNhdXNlIGJoeXZlIGRvZXMgbm90IGN1cnJlbnRseSBlbXVsYXRlIHRoYXQgaW5zdHJ1Y3Rp b24sIGl0IGFib3J0cyB0aGUgZ3Vlc3TigJlzIGluc3RydWN0aW9uIHN0cmVhbSBhbmQgcmVwb3J0 cyB0aGUgZXhpdCBjb2RlLgo+Pj4+IEluIG1vc3QgY2FzZXMgdGhlIGd1ZXN0IHdpbGwgY29udGlu dWUgYWZ0ZXIgYmh5dmUgaGFuZGxlcyB0aGUgZXhpdCwgYnV0IGlmIHRoZSBoeXBlcnZpc29yIGlz buKAmXQgc2V0IHVwIHRvIGVtdWxhdGUgWFNFVEJWIHlvdeKAmWxsIHNlZSB0aGUgVk0gc3RvcCBv ciBjcmFzaC4KPj4+Pgo+Pj4+IFR5cGljYWwgd2F5cyB0byBkZWFsIHdpdGggaXQKPj4+Pgo+Pj4+ IFVwZGF0ZSBiaHl2ZSDigJMgbmV3ZXIgcmVsZWFzZXMgaW5jbHVkZSBiYXNpYyBYU0VUQlYgaGFu ZGxpbmcgZm9yIGNvbW1vbiBmZWF0dXJlIHNldHMuCj4+Pj4gQWRqdXN0IHRoZSBndWVzdOKAmXMg Q1BVIGZsYWdzIOKAkyBzdGFydCB0aGUgVk0gd2l0aCBhIENQVSBtb2RlbCB0aGF0IGRvZXNu4oCZ dCBleHBvc2UgdGhlIFhTQVZFL1hHRVRCViBmZWF0dXJlcyB0aGUgZ3Vlc3QgZXhwZWN0cyAoZS5n LiwgLWMgaG9zdCB2cy4gYSBzcGVjaWZpYyBtb2RlbCkuCj4+Pj4gUGF0Y2ggYmh5dmUg4oCTIGlm IHlvdeKAmXJlIGNvbWZvcnRhYmxlIHdpdGggQywgeW91IGNhbiBhZGQgYSBoYW5kbGVyIGZvciBW TVhfRVhJVF9SRUFTT05fWFNFVEJWIHRoYXQgZWl0aGVyIGVtdWxhdGVzIHRoZSBpbnN0cnVjdGlv biBvciBzaW1wbHkgaWdub3JlcyBpdCB3aGVuIHRoZSByZXF1ZXN0ZWQgYml0cyBhcmUgYWxyZWFk eSBlbmFibGVkLgo+Pj4+Cj4+Pj4gSWYgeW91IGtlZXAgc2VlaW5nIHRoZSBzYW1lIGV4aXQgcmVw ZWF0ZWRseSwgaXQgdXN1YWxseSBwb2ludHMgdG8gYSBtaXNtYXRjaCBiZXR3ZWVuIHRoZSBndWVz dOKAmXMgZXhwZWN0YXRpb25zIChvZnRlbiBhIG1vZGVybiBMaW51eCBrZXJuZWwgb3IgYSBXaW5k b3dzIGJ1aWxkIHRoYXQgd2FudHMgQVZY4oCRNTEyKSBhbmQgdGhlIGNhcGFiaWxpdGllcyBiaHl2 ZSBpcyBwcmVzZW50aW5nLiBBbGlnbmluZyB0aG9zZSBleHBlY3RhdGlvbnPigJRlaXRoZXIgYnkg dHdlYWtpbmcgdGhlIFZN4oCZcyBDUFUgY29uZmlndXJhdGlvbiBvciBieSB1cGdyYWRpbmcgYmh5 dmXigJRnZW5lcmFsbHkgcmVzb2x2ZXMgdGhlIGlzc3VlLgo+Pj4+Cj4+Pj4gSm9uYXRoYW4gVmFz cXVlego+Pj4+IFBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBIEQ0OUYgRkI4NSA5MEI3IEM0 Q0EgNTI3OQo+Pj4+IFNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbAo+Pj4+Cj4+Pj4g T24gU2F0dXJkYXksIFNlcHRlbWJlciAyMHRoLCAyMDI1IGF0IDE5OjQ3LCBOdW5vIFRlaXhlaXJh IDxlZHVhcmRvQGZyZWVic2Qub3JnPiB3cm90ZToKPj4+Pgo+Pj4+PiBIZWxsbyBKb25hdGhhbiEK Pj4+Pj4KPj4+Pj4gVGhhbmtzIGZvciBleGNlbGxlbnQgcXVhbGl0eSB2aWRlb3Mgb24geXQsIEkg YWxyZWFkeSBzdWJzY3JpYmVkIGl0IDopCj4+Pj4+Cj4+Pj4+IEp1c3QgYSBxdWljayBxdWVzdGlv bnM6Cj4+Pj4+Cj4+Pj4+IDEtIEFyZSB5b3UgdXNpbmcgZnJlZXJkcDMgdG8gY29ubmVjdCB0byB3 aW4xMXBybz8gKEFza2luZyB0aGlzIGJlY2F1c2Ugdm5jIGlzIHZlcnkgc2xvdykKPj4+Pj4gMi0g RG8geW91IGtub3cgaG93IGlzIG52aWRpYSBzdGF0dXMgY29tcGFyZWQgdG8geW91ciB1c2UgY2Fz ZT8KPj4+Pj4KPj4+Pj4gVGhhbmtzIQo+Pj4+Pgo+Pj4+PiBKb25hdGhhbiBWYXNxdWV6IDxqb25A eHlpbm4ub3JnPiBlc2NyZXZldSAoc8OhYmFkbywgMjAvMDkvMjAyNSDDoChzKSAwMjo0Mik6Cj4+ Pj4+Cj4+Pj4+PiBIZXkgYWxsLCBJJ3ZlIHNwZW50IHNvbWUgdGltZSB0b2RheSBkb2luZyBhIGdh bWluZyBkZW1vLiBJJ3ZlIHVwbG9hZGVkIHRoZSB2aWRlbyBpbiAySyBzbyB5b3UgY2FuIGdldCB0 aGUgbWF4aW11bSBlZmZlY3QuIEl0J3MgYmVlbiBhbiBpbnRlcmVzdGluZyBleHBlcmllbmNlIHNv IGZhciwgYW5kIEknbSBnb2luZyB0byB0cnkgdG8gbWFrZSBpdCBteSBwcmltYXJ5IGdhbWluZyBt YWNoaW5lLiBXZSdsbCBzZWUgaG93IGl0IGdvZXMuIEl0J3MgZGVmaW5pdGVseSBxd2lya3ksIGFu ZCBzb21ldGltZXMgSSBuZWVkIHRvIHJlYm9vdCB0aGUgVk0gNC01IHRpbWVzIGJlZm9yZSBpdCAi c3RhYmlsaXplcyIgKHdoaWNoIG1lYW5zIHRoZSBWTSB3b24ndCBqdXN0IHJhbmRvbWx5IGNyYXNo IG9yIGhhdmUgbWFzc2l2ZSBsYWcgaW4gZ2FtZSkuIEknbGwgbmVlZCB0byBjb2xsZWN0IG1vcmUg ZGF0YSB0aHJvdWdoIHBsYXkgdGVzdGluZy4KPj4+Pj4+Cj4+Pj4+PiBZb3UgY2FuIHRha2UgYSBs b29rIGF0IHRoZSB2aWRlbyBoZXJlIChJJ2xsIHBvc3QgdGltZXN0YW1wcyBpbiB0aGUgdmlkZW8g c29vbik6Cj4+Pj4+Pgo+Pj4+Pj4gaHR0cHM6Ly95b3V0dS5iZS9fY3owUlVBdzVwOAo+Pj4+Pj4K Pj4+Pj4+IEpvbmF0aGFuIFZhc3F1ZXoKPj4+Pj4+IFBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBD NzdBIEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3OQo+Pj4+Pj4gU2VudCB3aXRoIFByb3Rvbk1haWwg U2VjdXJlIEVtYWlsCj4+Pj4+Pgo+Pj4+Pj4gT24gV2VkbmVzZGF5LCBTZXB0ZW1iZXIgMTd0aCwg MjAyNSBhdCAyMjozMywgSm9uYXRoYW4gVmFzcXVleiA8am9uQHh5aW5uLm9yZz4gd3JvdGU6Cj4+ Pj4+Pgo+Pj4+Pj4+IEFscmlnaHQhIFRoZSBZb3VUdWJlIFZpZGVvIGlzIHVwLiBMZXQgbWUga25v dyB3aGF0IHlvdSB0aGluay4gSSdtIHJlYWxseSBleGNpdGVkIGFib3V0IHRoaXMuCj4+Pj4+Pj4K Pj4+Pj4+PiBodHRwczovL3lvdXR1LmJlL09iNC12N2RUSkdzCj4+Pj4+Cj4+Pj4+IC0tCj4+Pj4+ Cj4+Pj4+IE51bm8gVGVpeGVpcmEKPj4+Pj4gRnJlZUJTRCBVTklYOiA8ZWR1YXJkb0BGcmVlQlNE Lm9yZz4gV2ViOiBbaHR0cHM6Ly9GcmVlQlNELm9yZ10oaHR0cHM6Ly9mcmVlYnNkLm9yZy8p --b1=_OdLJLr8sUAwsmR6QT9puQVzBTbPy2TZJZy1FfVY Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0 cHg7Ij5IZXkgYWxsLDwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1z ZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5 OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+SnVzdCB3YW50ZWQgdG8gcmVw b3J0IHRoYXQgSSBzdGlsbCBoYXZlbid0IHJlY2VpdmVkIG9uZSBjcmFzaCBhZnRlciBteSBhZGp1 c3RtZW50cyB5ZXN0ZXJkYXkuIFNvIGZhciBvdmVyIDIwKyBob3VycyBvZiB1cHRpbWV+LiBJJ3Zl IGFsc28gcmVib290ZWQgdGhlIFZNL2hvc3QgYSBmZXcgdGltZXMganVzdCB0byBlbnN1cmUgdGhh dCBpdCBpcyBkZXRlcm1pbmlzdGljYWxseSBzdGFibGUgYW5kIHNvIGZhciB6ZXJvIGlzc3Vlcy4g U28gdGhpcyBpcyBwb2ludGluZyBpbiBhIHBvc2l0aXZlIGRpcmVjdGlvbi4gSSdtIHRlbXB0ZWQg dG8gbWFyayB0aGlzIGFzIGEgc3RhYmxlIHNldCB1cCBidXQgSSdsbCBuZWVkIHRvIHBsYXkgZm9y IGxvbmdlciB0byBtYWtlIHRoYXQgY2FsbC48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTog QXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsiPjxicj48L2Rpdj4NCjxkaXYgc3R5 bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyIgY2xh c3M9InByb3Rvbm1haWxfc2lnbmF0dXJlX2Jsb2NrIj4NCiAgICA8ZGl2IGNsYXNzPSJwcm90b25t YWlsX3NpZ25hdHVyZV9ibG9jay11c2VyIj4NCiAgICAgICAgPGRpdj5Kb25hdGhhbiBWYXNxdWV6 PGJyPjwvZGl2PjxkaXY+UEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgIEQ0OUYgRkI4NSA5 MEI3IEM0Q0EgNTI3OTxicj48L2Rpdj48ZGl2PlNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBF bWFpbDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj4NCiAgICA8L2Rpdj4NCiAgICANCiAgICAgICAg ICAgIDxkaXYgY2xhc3M9InByb3Rvbm1haWxfc2lnbmF0dXJlX2Jsb2NrLXByb3RvbiBwcm90b25t YWlsX3NpZ25hdHVyZV9ibG9jay1lbXB0eSI+DQogICAgICAgIA0KICAgICAgICAgICAgPC9kaXY+ DQo8L2Rpdj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9u dC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2PjxkaXYgY2xhc3M9InByb3Rvbm1haWxfcXVvdGUiPg0K ICAgICAgICBPbiBTdW5kYXksIFNlcHRlbWJlciAyMXN0LCAyMDI1IGF0IDA4OjIzLCBKb25hdGhh biBWYXNxdWV6ICZsdDtqb25AeHlpbm4ub3JnJmd0OyB3cm90ZTo8YnI+DQogICAgICAgIDxibG9j a3F1b3RlIGNsYXNzPSJwcm90b25tYWlsX3F1b3RlIiB0eXBlPSJjaXRlIj4NCiAgICAgICAgICAg IFN5c3RlbSBoYXMgYmVlbiB1cCBmb3IgYWJvdXQgOWg0MG0uIE1vc3Qgb2YgaXQgaWRsZSBhbmQg bm8gY3Jhc2hlcyB5ZXQuIEknbSBnb25uYSByZWJvb3QgdGhlIGhvc3QgYW5kIHNlZSBpZiB0aGUg c3RhYmlsaXR5IHN0aWxsIGNvbnRpbnVlcy48YnI+PGJyPjxkaXYgY2xhc3M9InByb3Rvbm1haWxf c2lnbmF0dXJlX2Jsb2NrLXVzZXIiPjxkaXY+Sm9uYXRoYW4gVmFzcXVlejxicj48L2Rpdj48ZGl2 PlBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBICBENDlGIEZCODUgOTBCNyBDNENBIDUyNzk8 YnI+PC9kaXY+PGRpdj5TZW50IHdpdGggUHJvdG9uTWFpbCBTZWN1cmUgRW1haWw8YnI+PC9kaXY+ PGRpdj48YnI+PC9kaXY+PC9kaXY+PGJyPjxicj5TZW50IGZyb20gPGEgdGFyZ2V0PSJfYmxhbmsi IGhyZWY9Imh0dHBzOi8vcHJvdG9uLm1lL21haWwvaG9tZSIgcmVsPSJub3JlZmVycmVyIG5vZm9s bG93IG5vb3BlbmVyIj5Qcm90b24gTWFpbDwvYT4gZm9yIEFuZHJvaWQuPGRpdiBjbGFzcz0icHJv dG9ubWFpbF9xdW90ZSI+PGJyPjxicj4tLS0tLS0tLSBPcmlnaW5hbCBNZXNzYWdlIC0tLS0tLS0t PGJyPk9uIFNhdHVyZGF5LCAwOS8yMC8yNSBhdCAyMzo0NSBKb25hdGhhbiBWYXNxdWV6ICZsdDtq b25AeHlpbm4ub3JnJmd0OyB3cm90ZTo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9InByb3Rvbm1haWxf cXVvdGUiPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1z aXplOiAxNHB4Ij5KdXN0IGZpbmlzaGVkIGJlYXRpbmcgdGhlIG1haW4gc3RvcnkgbGluZSBpbiBD eWJlcnB1bmsgMjA3NyBpbiB0aGUgVk0uIFNvIGZhciBubyBjcmFzaGVzIGFuZCBpdCdzIGJlZW4g cnVubmluZyBmb3IgYW4gaG91ci4gU28gdGhpcyBpcyBhIHBvc2l0aXZlIHNpZ25hbC4uLiB3ZSds bCBzZWUgd2hhdCBoYXBwZW5zLiBJJ2xsIGxlYXZlIHRoZSBWTSBydW5uaW5nIGlkbGUgb3ZlciBu aWdodCBhbmQgc2VlIGlmIGl0J3Mgc3RpbGwgcnVubmluZyBpbiB0aGUgbW9ybmluZy4gU3RheSBz YWZlIGFsbCE8L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7 IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2Pg0KPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFy aWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiIGNsYXNzPSJwcm90b25tYWlsX3NpZ25h dHVyZV9ibG9jayI+DQogICAgPGRpdiBjbGFzcz0icHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2st dXNlciI+DQogICAgICAgIDxkaXY+Sm9uYXRoYW4gVmFzcXVlejxicj48L2Rpdj48ZGl2PlBHUDog MzREQSA4NThDIDE0NDcgNTA5RSBDNzdBICBENDlGIEZCODUgOTBCNyBDNENBIDUyNzk8YnI+PC9k aXY+PGRpdj5TZW50IHdpdGggUHJvdG9uTWFpbCBTZWN1cmUgRW1haWw8YnI+PC9kaXY+PGRpdj48 YnI+PC9kaXY+DQogICAgPC9kaXY+DQoNCiAgICAgICAgICAgIDxkaXYgY2xhc3M9InByb3Rvbm1h aWxfc2lnbmF0dXJlX2Jsb2NrLXByb3RvbiBwcm90b25tYWlsX3NpZ25hdHVyZV9ibG9jay1lbXB0 eSI+DQoNCiAgICAgICAgICAgIDwvZGl2Pg0KPC9kaXY+DQo8ZGl2IHN0eWxlPSJmb250LWZhbWls eTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2PjxkaXYgY2xh c3M9InByb3Rvbm1haWxfcXVvdGUiPg0KICAgICAgICBPbiBTYXR1cmRheSwgU2VwdGVtYmVyIDIw dGgsIDIwMjUgYXQgMjI6MjgsIEpvbmF0aGFuIFZhc3F1ZXogJmx0O2pvbkB4eWlubi5vcmcmZ3Q7 IHdyb3RlOjxicj4NCiAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9InByb3Rvbm1haWxfcXVvdGUi IHR5cGU9ImNpdGUiPg0KICAgICAgICAgICAgPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFs LCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPkkgZGlkIGEgYnVuY2ggb2YgbW9yZSBleHBl cmltZW50cyB0b2RheSBhbmQgSSB3YXMgYWJsZSB0byBzd2l0Y2ggZnJvbSB1c2luZyB0aGUgVVNC IGNvbnRyb2xsZXIgY29ubmVjdGVkIHRvIHRoZSBtdWx0aS1mdW5jdGlvbmFsIGJ1cyB0aGF0IGhh cyB0aGUgaW50ZWdyYXRlZCBjYXJkLCBhbmQgc3dpdGNoZWQgdG8gYSBzZXBhcmF0ZSBVU0IgY29u dHJvbGxlciB0aGF0IG9ubHkgaGFzIFVTQiBwb3J0cy4gSSdtIGhvcGluZyB0aGF0IHRoaXMgaW5j cmVhc2VzIFZNIHN0YWJpbGl0eSBhbmQgdGhhdCB0aGVyZSBtYXkgaGF2ZSBiZWVuIHNvbWUgd2ly ZXMgYmVpbmcgY3Jvc3NlZC4gVGhpcyB3YXMgdGhlIHByZXZpb3VzIGJ1cyBJIHdhcyB1c2luZyAo SSBjb3VsZG4ndCBpc29sYXRlIGp1c3QgdGhlIFVTQiBmdW5jdGlvbnMsIGl0IGhhZCB0byBiZSBh bGwgb3Igbm90aGluZyBmb3IgaXQgdG8gZnVuY3Rpb24pOjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQt ZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48YnI+PC9kaXY+PGRp diBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgi PjxzcGFuPnZnYXBjaTBAcGNpMDoxODowOjA6CWNsYXNzPTB4MDMwMDAwIHJldj0weGMxIGhkcj0w eDAwIHZlbmRvcj0weDEwMDIgZGV2aWNlPTB4MTY0ZSBzdWJ2ZW5kb3I9MHgxMDQzIHN1YmRldmlj ZT0weDg4Nzc8L3NwYW4+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IHZlbmRvciAmbmJzcDsgJm5i c3A7ID0gJ0FkdmFuY2VkIE1pY3JvIERldmljZXMsIEluYy4gW0FNRC9BVEldJzwvc3Bhbj48L2Rp dj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgZGV2aWNlICZuYnNwOyAmbmJzcDsgPSAnUmFwaGFl bCc8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IGNsYXNzICZuYnNwOyAmbmJz cDsgJm5ic3A7PSBkaXNwbGF5PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBz dWJjbGFzcyAmbmJzcDsgPSBWR0E8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj5oZGFjMEBwY2kwOjE4 OjA6MToJY2xhc3M9MHgwNDAzMDAgcmV2PTB4MDAgaGRyPTB4MDAgdmVuZG9yPTB4MTAwMiBkZXZp Y2U9MHgxNjQwIHN1YnZlbmRvcj0weDEwNDMgc3ViZGV2aWNlPTB4ODg3Nzwvc3Bhbj48L2Rpdj48 ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgdmVuZG9yICZuYnNwOyAmbmJzcDsgPSAnQWR2YW5jZWQg TWljcm8gRGV2aWNlcywgSW5jLiBbQU1EL0FUSV0nPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5i c3A7ICZuYnNwOyBkZXZpY2UgJm5ic3A7ICZuYnNwOyA9ICdSZW1icmFuZHQgUmFkZW9uIEhpZ2gg RGVmaW5pdGlvbiBBdWRpbyBDb250cm9sbGVyJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNw OyAmbmJzcDsgY2xhc3MgJm5ic3A7ICZuYnNwOyAmbmJzcDs9IG11bHRpbWVkaWE8L3NwYW4+PC9k aXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IHN1YmNsYXNzICZuYnNwOyA9IEhEQTwvc3Bhbj48 L2Rpdj48ZGl2PjxzcGFuPm5vbmUxQHBjaTA6MTg6MDoyOgljbGFzcz0weDEwODAwMCByZXY9MHgw MCBoZHI9MHgwMCB2ZW5kb3I9MHgxMDIyIGRldmljZT0weDE2NDkgc3VidmVuZG9yPTB4MTA0MyBz dWJkZXZpY2U9MHg4ODc3PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyB2ZW5k b3IgJm5ic3A7ICZuYnNwOyA9ICdBZHZhbmNlZCBNaWNybyBEZXZpY2VzLCBJbmMuIFtBTURdJzwv c3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgZGV2aWNlICZuYnNwOyAmbmJzcDsg PSAnRmFtaWx5IDE5aCBQU1AvQ0NQJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJz cDsgY2xhc3MgJm5ic3A7ICZuYnNwOyAmbmJzcDs9IGVuY3J5cHQvZGVjcnlwdDwvc3Bhbj48L2Rp dj48ZGl2PjxzcGFuPnhoY2kxQHBjaTA6MTg6MDozOgljbGFzcz0weDBjMDMzMCByZXY9MHgwMCBo ZHI9MHgwMCB2ZW5kb3I9MHgxMDIyIGRldmljZT0weDE1YjYgc3VidmVuZG9yPTB4MTA0MyBzdWJk ZXZpY2U9MHg4ODc3PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyB2ZW5kb3Ig Jm5ic3A7ICZuYnNwOyA9ICdBZHZhbmNlZCBNaWNybyBEZXZpY2VzLCBJbmMuIFtBTURdJzwvc3Bh bj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgZGV2aWNlICZuYnNwOyAmbmJzcDsgPSAn UmFwaGFlbC9HcmFuaXRlIFJpZGdlIFVTQiAzLjEgeEhDSSc8L3NwYW4+PC9kaXY+PGRpdj48c3Bh bj4mbmJzcDsgJm5ic3A7IGNsYXNzICZuYnNwOyAmbmJzcDsgJm5ic3A7PSBzZXJpYWwgYnVzPC9z cGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBzdWJjbGFzcyAmbmJzcDsgPSBVU0I8 L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj54aGNpMkBwY2kwOjE4OjA6NDoJY2xhc3M9MHgwYzAzMzAg cmV2PTB4MDAgaGRyPTB4MDAgdmVuZG9yPTB4MTAyMiBkZXZpY2U9MHgxNWI3IHN1YnZlbmRvcj0w eDEwNDMgc3ViZGV2aWNlPTB4ODg3Nzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJz cDsgdmVuZG9yICZuYnNwOyAmbmJzcDsgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywgSW5jLiBb QU1EXSc8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IGRldmljZSAmbmJzcDsg Jm5ic3A7ID0gJ1JhcGhhZWwvR3Jhbml0ZSBSaWRnZSBVU0IgMy4xIHhIQ0knPC9zcGFuPjwvZGl2 PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBjbGFzcyAmbmJzcDsgJm5ic3A7ICZuYnNwOz0gc2Vy aWFsIGJ1czwvc3Bhbj48L2Rpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IHN1YmNsYXNzICZuYnNwOyA9 IFVTQjwvc3Bhbj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2Vy aWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBB cmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij5BbHRob3VnaCBub3cgSSdtIGxvbmdl ciB1c2luZyB0aGlzIGF0IGFsbCBmb3IgVVNCIGZ1bmN0aW9uYWxpdHkgaW4gdGhlIFZNLiBJJ3Zl IHN3aXRjaGVkIHRvOjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1z ZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6 IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxzcGFuPnBwdDJAcGNpMDoxMzow OjA6CWNsYXNzPTB4MGMwMzMwIHJldj0weDAxIGhkcj0weDAwIHZlbmRvcj0weDEwMjIgZGV2aWNl PTB4NDNmNyBzdWJ2ZW5kb3I9MHgxYjIxIHN1YmRldmljZT0weDExNDI8L3NwYW4+PGRpdj48c3Bh bj4mbmJzcDsgJm5ic3A7IHZlbmRvciAmbmJzcDsgJm5ic3A7ID0gJ0FkdmFuY2VkIE1pY3JvIERl dmljZXMsIEluYy4gW0FNRF0nPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBk ZXZpY2UgJm5ic3A7ICZuYnNwOyA9ICc2MDAgU2VyaWVzIENoaXBzZXQgVVNCIDMuMiBDb250cm9s bGVyJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgY2xhc3MgJm5ic3A7ICZu YnNwOyAmbmJzcDs9IHNlcmlhbCBidXM8L3NwYW4+PC9kaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBz dWJjbGFzcyAmbmJzcDsgPSBVU0I8L3NwYW4+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6 IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rpdj48ZGl2IHN0eWxl PSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+TXVjaCBj bGVhbmVyIGFuZCBzdGlsbCBjb250YWlucyBhIHZhcmlldHkgb2YgaGlnaCBzcGVlZCBwb3J0cy4g QWx0aG91Z2ggdWx0aW1hdGVseSBJJ20ganVzdCB1c2luZyBvbmUgb2YgdGhlbSBiZWNhdXNlIEkg aGF2ZSBhIEtWTSBjb25uZWN0ZWQgdG8gaXQgdGhhdCBhbGxvd3MgbWUgdG8gcGx1ZyBpbiBrZXli b2FyZC9tb3VzZS8zLjVtbSBqYWNrIChhdWRpbykvZXRoZXJuZXQgYWxsIHRocm91Z2ggb25lIFVT QiB3aXJlLiBBbHRob3VnaCB0byByZW1pbmQgZXZlcnlvbmUsIEknbSBnYW1pbmcgaW4gYW4gb2Zm bGluZSBjYXBhY2l0eSB0byBJIGludGVudGlvbmFsbHkgbWVzcyB1cCB0aGUgSVB2NCBnYXRld2F5 IHNvIG15IFZNIGNhbiBvbmx5IGNvbW11bmljYXRlIGludGVybmFsbHkgKGFuZCBJIGNhbiBzaWRl bG9hZCBhcHBzL2dhbWVzKSBidXQgcHJldmVudCBNaWNyb3NvZnQgZnJvbSByZWFjaGluZyBteSBt YWNoaW5lIGFuZCBjb2xsZWN0aW5nIHRlbGVtZXRyeSBpbmZvLCBhbmQgZG9pbmcgZm9yY2VkIHVw ZGF0ZXMuIFBsdXMsIEkgaGF2ZSBubyBpbnRlbnRpb24gb2YgbW92aW5nIHRvIFdpbmRvd3MgMTEg d2l0aCBhbGwgb2YgdGhlaXIgbmV3IHJlcXVpcmVtZW50cyBhbmQgZm9yY2VkIG9ubGluZSBhY3Rp dmF0aW9uIChJIGtub3cgb2YgdGhlIHdvcmthcm91bmRzLCBidXQgSSdtIGFnYWluc3Qgb24gbWFu ZGF0aW5nIGEgTVNBIG9uIHByaW5jaXBsZSkuIFBlb3BsZSBzaG91bGQgYmUgYWJsZSB0byBmdWxs eSB1c2UgdGhlaXIgbWFjaGluZXMgZXZlbiBkdXJpbmcgaW5zdGFsbGF0aW9uL09PQkUgd2l0aG91 dCBoYXZpbmcgdG8gYXV0aGVudGljYXRlIHRvIGEgY2VudHJhbCBzZXJ2ZXIuIEFueXdheXMsIHNt YWxsIHJhbnQgb3ZlciBsb2wuPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBz YW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZh bWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+SSdsbCB1cGRhdGUgZXZl cnlvbmUgb24gbXkgcHJvZ3Jlc3Mgd2l0aCB0aGlzIHJlLWNvbmZpZ3VyYXRpb24uPC9kaXY+PGRp diBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgi Pjxicj48L2Rpdj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsg Zm9udC1zaXplOiAxNHB4IiBjbGFzcz0icHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2siPg0KICAg IDxkaXYgY2xhc3M9InByb3Rvbm1haWxfc2lnbmF0dXJlX2Jsb2NrLXVzZXIiPg0KICAgICAgICA8 ZGl2PkpvbmF0aGFuIFZhc3F1ZXo8YnI+PC9kaXY+PGRpdj5QR1A6IDM0REEgODU4QyAxNDQ3IDUw OUUgQzc3QSAgRDQ5RiBGQjg1IDkwQjcgQzRDQSA1Mjc5PGJyPjwvZGl2PjxkaXY+U2VudCB3aXRo IFByb3Rvbk1haWwgU2VjdXJlIEVtYWlsPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2Pg0KICAgIDwv ZGl2Pg0KDQogICAgICAgICAgICA8ZGl2IGNsYXNzPSJwcm90b25tYWlsX3NpZ25hdHVyZV9ibG9j ay1wcm90b24gcHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2stZW1wdHkiPg0KDQogICAgICAgICAg ICA8L2Rpdj4NCjwvZGl2Pg0KPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNl cmlmOyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rpdj48ZGl2IGNsYXNzPSJwcm90b25tYWlsX3F1 b3RlIj4NCiAgICAgICAgT24gU2F0dXJkYXksIFNlcHRlbWJlciAyMHRoLCAyMDI1IGF0IDIwOjAy LCBKb25hdGhhbiBWYXNxdWV6ICZsdDtqb25AeHlpbm4ub3JnJmd0OyB3cm90ZTo8YnI+DQogICAg ICAgIDxibG9ja3F1b3RlIGNsYXNzPSJwcm90b25tYWlsX3F1b3RlIiB0eXBlPSJjaXRlIj4NCiAg ICAgICAgICAgIDxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9u dC1zaXplOiAxNHB4Ij5IZXkgTnVubyw8L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJp YWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZv bnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48b2wgc3R5bGU9 Im1hcmdpbi10b3A6IDA7IG1hcmdpbi1ib3R0b206IDAiPjxsaSBzdHlsZT0ibGlzdC1zdHlsZS10 eXBlOiAmcXVvdDsxLiAmcXVvdDsiPjxzcGFuPkknbSBub3QuIEkgcGx1Z2dlZCBpbiBteSBtb25p dG9yJ3MgSERNSSBjYWJsZSBkaXJlY3RseSBpbnRvIHRoZSBHUFUncyBIRE1JIHBvcnQuIE9uY2Ug Ymh5dmUgc3RhcnRzIHVwIChpZiB5b3VyIHBhc3N0aHJvdWdoIGlzIHdvcmtpbmcgY29ycmVjdGx5 IGFuZCB5b3VyIEdQVSdzIGRyaXZlcnMgYXJlIGluc3RhbGxlZCksIGl0IHdpbGwgb3V0cHV0IHRv IHlvdXIgbW9uaXRvci4gWW91IHdvdWxkIG9ubHkgbmVlZCBWTkMgZm9yIGluaXRpYWwgaW5zdGFs bGF0aW9uIG9mIFdpbmRvd3MuPC9zcGFuPjwvbGk+PGxpIHN0eWxlPSJsaXN0LXN0eWxlLXR5cGU6 ICZxdW90OzIuICZxdW90OyI+PHNwYW4+SSdtIG5vdCBzdXJlIG9mIHRoZSBOVklESUEgY2FzZS4g SSBkbyBrbm93IHRoZXJlIGlzIHNvbWVvbmUgd2hvIGJsb2dnZWQgYWJvdXQgaG93IHRoZXkgZ290 IHRoZWlycyB3b3JraW5nOiZuYnNwOzxzcGFuPjxhIHRhcmdldD0iX2JsYW5rIiByZWw9Im5vcmVm ZXJyZXIgbm9mb2xsb3cgbm9vcGVuZXIiIGhyZWY9Imh0dHBzOi8vZGZsdW5kLnNlL35nZXR6L05v dGVzLzIwMjQvZnJlZWJzZC1ncHUvIj5odHRwczovL2RmbHVuZC5zZS9+Z2V0ei9Ob3Rlcy8yMDI0 L2ZyZWVic2QtZ3B1LzwvYT48L3NwYW4+PC9zcGFuPjwvbGk+PC9vbD48ZGl2Pjxicj48L2Rpdj48 ZGl2PkF0bSBJJ20gY3VycmVudGx5IHRyeWluZyB0byBjb2xsZWN0IG1vcmUgaW5mbyBvbiB3aHkg bXkgYmh5dmUgVk0gcmFuZG9tbHkgY3Jhc2hlcyBzb21ldGltZXMuIFNvIGZhciBJJ3ZlIGNvbGxl Y3RlZCB0aGF0IHRoZSBleGl0IHJlYXNvbiBiaHl2ZSBpcyBzaG93aW5nIGlzIDB4NjAsIHdoaWNo IHBvaW50cyB0byB0aGUgZm9sbG93aW5nIGVudHJpZXMgaW4gdGhlIGFtZCBmcmVlYnNkIGtlcm5l bCBzb3VyY2VzOjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PHNwYW4+MHg2MDwvc3Bhbj48ZGl2 PjxzcGFuPjxhIHRhcmdldD0iX2JsYW5rIiByZWw9Im5vcmVmZXJyZXIgbm9mb2xsb3cgbm9vcGVu ZXIiIGhyZWY9Imh0dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL2Jsb2IvNGMz YTg2OGQxM2MwNTNlZjE3MzI2OGNkZmUxMzY1OTc4YTI4MjE3OC9zeXMvYW1kNjQvdm1tL2FtZC92 bWNiLmgjTDE5NSI+aHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMvYmxvYi80 YzNhODY4ZDEzYzA1M2VmMTczMjY4Y2RmZTEzNjU5NzhhMjgyMTc4L3N5cy9hbWQ2NC92bW0vYW1k L3ZtY2IuaCNMMTk1PC9hPjwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPjxhIHRhcmdldD0iX2JsYW5r IiByZWw9Im5vcmVmZXJyZXIgbm9mb2xsb3cgbm9vcGVuZXIiIGhyZWY9Imh0dHBzOi8vZ2l0aHVi LmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL2Jsb2IvNGMzYTg2OGQxM2MwNTNlZjE3MzI2OGNkZmUx MzY1OTc4YTI4MjE3OC9zeXMvYW1kNjQvdm1tL2FtZC92bWNiLmgjTDEyOEMxLUwxMjhDMzAiPmh0 dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL2Jsb2IvNGMzYTg2OGQxM2MwNTNl ZjE3MzI2OGNkZmUxMzY1OTc4YTI4MjE3OC9zeXMvYW1kNjQvdm1tL2FtZC92bWNiLmgjTDEyOEMx LUwxMjhDMzA8L2E+PC9zcGFuPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PHNwYW4+WFNFVEJW PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+PGEgdGFyZ2V0PSJfYmxhbmsiIHJlbD0ibm9yZWZlcnJl ciBub2ZvbGxvdyBub29wZW5lciIgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJl ZWJzZC1zcmMvYmxvYi80YzNhODY4ZDEzYzA1M2VmMTczMjY4Y2RmZTEzNjU5NzhhMjgyMTc4L3N5 cy9hbWQ2NC92bW0vYW1kL3ZtY2IuaCNMOTQiPmh0dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL2Zy ZWVic2Qtc3JjL2Jsb2IvNGMzYTg2OGQxM2MwNTNlZjE3MzI2OGNkZmUxMzY1OTc4YTI4MjE3OC9z eXMvYW1kNjQvdm1tL2FtZC92bWNiLmgjTDk0PC9hPjwvc3Bhbj48L2Rpdj48c3Bhbj48L3NwYW4+ PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JIGFsc28gYXNrZWQgUHJvdG9uJ3MgTHVtbysgQUkg d2hhdCBpcyAweDYwIHRvIHNlZSB3aGF0IGl0IGNvbWVzIHVwIHdpdGgsIGl0IHNhaWQgdGhlIGZv bGxvd2luZzo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxzcGFuPmV4aXRfcmVhc29uWzBdID0g MHg2MCBpcyB0aGUgVk3igJFleGl0IGNvZGUgdGhhdCBiaHl2ZSByZXBvcnRzIHdoZW4gdGhlIGd1 ZXN0IHRyaWdnZXJzIHRoZSBYU0VUQlYgaW5zdHJ1Y3Rpb24uPC9zcGFuPjxkaXY+PHNwYW4+V2h5 IGl0IGhhcHBlbnM8L3NwYW4+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsg Jm5ic3A7IFhTRVRCViB3cml0ZXMgdG8gdGhlIGV4dGVuZGVkIGNvbnRyb2wgcmVnaXN0ZXIgWENS MC48L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IEluIEludGVs4oCvVk1YICh0 aGUgdmlydHVhbGl6YXRpb24gdGVjaG5vbG9neSBiaHl2ZSB1c2VzIG9uIHg4NiksIGV4ZWN1dGlv biBvZiBYU0VUQlYgaXMgbm90IGFsbG93ZWQgdG8gcnVuIGRpcmVjdGx5IGluIHRoZSBndWVzdDsg dGhlIENQVSBmb3JjZXMgYSBWTeKAkWV4aXQgc28gdGhlIGh5cGVydmlzb3IgY2FuIHZhbGlkYXRl IHRoZSBvcGVyYXRpb24uPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBiaHl2 ZSByZWNvcmRzIHRoZSBleGl0IHJlYXNvbiBhcyAweDYwLCB3aGljaCBtYXBzIHRvIHRoZSBWTVgg ZXhpdCByZWFzb24g4oCcWFNFVEJW4oCdIGluIHRoZSBJbnRlbCBzcGVjaWZpY2F0aW9uLjwvc3Bh bj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxzcGFuPldoYXQgaXQgbWVhbnMgZm9yIHlvdTwv c3Bhbj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgVGhlIGd1 ZXN0IHRyaWVkIHRvIGVuYWJsZSBvciBjaGFuZ2UgQVZY4oCRNTEyIC8gWFNBVkUgZmVhdHVyZSBi aXRzIHZpYSBYU0VUQlYuPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBCZWNh dXNlIGJoeXZlIGRvZXMgbm90IGN1cnJlbnRseSBlbXVsYXRlIHRoYXQgaW5zdHJ1Y3Rpb24sIGl0 IGFib3J0cyB0aGUgZ3Vlc3TigJlzIGluc3RydWN0aW9uIHN0cmVhbSBhbmQgcmVwb3J0cyB0aGUg ZXhpdCBjb2RlLjwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgSW4gbW9zdCBj YXNlcyB0aGUgZ3Vlc3Qgd2lsbCBjb250aW51ZSBhZnRlciBiaHl2ZSBoYW5kbGVzIHRoZSBleGl0 LCBidXQgaWYgdGhlIGh5cGVydmlzb3IgaXNu4oCZdCBzZXQgdXAgdG8gZW11bGF0ZSBYU0VUQlYg eW914oCZbGwgc2VlIHRoZSBWTSBzdG9wIG9yIGNyYXNoLjwvc3Bhbj48L2Rpdj48ZGl2Pjxicj48 L2Rpdj48ZGl2PjxzcGFuPlR5cGljYWwgd2F5cyB0byBkZWFsIHdpdGggaXQ8L3NwYW4+PC9kaXY+ PGRpdj48YnI+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IFVwZGF0ZSBiaHl2ZSDigJMg bmV3ZXIgcmVsZWFzZXMgaW5jbHVkZSBiYXNpYyBYU0VUQlYgaGFuZGxpbmcgZm9yIGNvbW1vbiBm ZWF0dXJlIHNldHMuPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBBZGp1c3Qg dGhlIGd1ZXN04oCZcyBDUFUgZmxhZ3Mg4oCTIHN0YXJ0IHRoZSBWTSB3aXRoIGEgQ1BVIG1vZGVs IHRoYXQgZG9lc27igJl0IGV4cG9zZSB0aGUgWFNBVkUvWEdFVEJWIGZlYXR1cmVzIHRoZSBndWVz dCBleHBlY3RzIChlLmcuLCAtYyBob3N0IHZzLiBhIHNwZWNpZmljIG1vZGVsKS48L3NwYW4+PC9k aXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IFBhdGNoIGJoeXZlIOKAkyBpZiB5b3XigJlyZSBj b21mb3J0YWJsZSB3aXRoIEMsIHlvdSBjYW4gYWRkIGEgaGFuZGxlciBmb3IgVk1YX0VYSVRfUkVB U09OX1hTRVRCViB0aGF0IGVpdGhlciBlbXVsYXRlcyB0aGUgaW5zdHJ1Y3Rpb24gb3Igc2ltcGx5 IGlnbm9yZXMgaXQgd2hlbiB0aGUgcmVxdWVzdGVkIGJpdHMgYXJlIGFscmVhZHkgZW5hYmxlZC48 L3NwYW4+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48c3Bhbj5JZiB5b3Uga2VlcCBzZWVpbmcg dGhlIHNhbWUgZXhpdCByZXBlYXRlZGx5LCBpdCB1c3VhbGx5IHBvaW50cyB0byBhIG1pc21hdGNo IGJldHdlZW4gdGhlIGd1ZXN04oCZcyBleHBlY3RhdGlvbnMgKG9mdGVuIGEgbW9kZXJuIExpbnV4 IGtlcm5lbCBvciBhIFdpbmRvd3MgYnVpbGQgdGhhdCB3YW50cyBBVljigJE1MTIpIGFuZCB0aGUg Y2FwYWJpbGl0aWVzIGJoeXZlIGlzIHByZXNlbnRpbmcuIEFsaWduaW5nIHRob3NlIGV4cGVjdGF0 aW9uc+KAlGVpdGhlciBieSB0d2Vha2luZyB0aGUgVk3igJlzIENQVSBjb25maWd1cmF0aW9uIG9y IGJ5IHVwZ3JhZGluZyBiaHl2ZeKAlGdlbmVyYWxseSByZXNvbHZlcyB0aGUgaXNzdWUuPC9zcGFu PjwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZh bWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCIgY2xhc3M9InByb3Rvbm1h aWxfc2lnbmF0dXJlX2Jsb2NrIj4NCiAgICA8ZGl2IGNsYXNzPSJwcm90b25tYWlsX3NpZ25hdHVy ZV9ibG9jay11c2VyIj4NCiAgICAgICAgPGRpdj5Kb25hdGhhbiBWYXNxdWV6PGJyPjwvZGl2Pjxk aXY+UEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgIEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3 OTxicj48L2Rpdj48ZGl2PlNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbDxicj48L2Rp dj48ZGl2Pjxicj48L2Rpdj4NCiAgICA8L2Rpdj4NCg0KICAgICAgICAgICAgPGRpdiBjbGFzcz0i cHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2stcHJvdG9uIHByb3Rvbm1haWxfc2lnbmF0dXJlX2Js b2NrLWVtcHR5Ij4NCg0KICAgICAgICAgICAgPC9kaXY+DQo8L2Rpdj4NCjxkaXYgc3R5bGU9ImZv bnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48YnI+PC9kaXY+ PGRpdiBjbGFzcz0icHJvdG9ubWFpbF9xdW90ZSI+DQogICAgICAgIE9uIFNhdHVyZGF5LCBTZXB0 ZW1iZXIgMjB0aCwgMjAyNSBhdCAxOTo0NywgTnVubyBUZWl4ZWlyYSAmbHQ7ZWR1YXJkb0BmcmVl YnNkLm9yZyZndDsgd3JvdGU6PGJyPg0KICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0icHJvdG9u bWFpbF9xdW90ZSIgdHlwZT0iY2l0ZSI+DQogICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj48ZGl2 PkhlbGxvIEpvbmF0aGFuITwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhhbmtzIGZvciBleGNl bGxlbnQgcXVhbGl0eSB2aWRlb3Mgb24geXQsIEkgYWxyZWFkeSBzdWJzY3JpYmVkIGl0IDopPC9k aXY+PGRpdj48YnI+PC9kaXY+PGRpdj5KdXN0IGEgcXVpY2sgcXVlc3Rpb25zOjwvZGl2PjxkaXY+ PGJyPjwvZGl2PjxkaXY+MS0gQXJlIHlvdSB1c2luZyBmcmVlcmRwMyB0byBjb25uZWN0IHRvIHdp bjExcHJvPyAgKEFza2luZyB0aGlzIGJlY2F1c2Ugdm5jIGlzIHZlcnkgc2xvdyk8L2Rpdj48ZGl2 PjItIERvIHlvdSBrbm93IGhvdyBpcyBudmlkaWEgc3RhdHVzIGNvbXBhcmVkIHRvIHlvdXIgdXNl IGNhc2U/PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGFua3MhPC9kaXY+PC9kaXY+PGJyPjxk aXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IGRpcj0ibHRyIiBjbGFzcz0iZ21haWxfYXR0ciI+ Sm9uYXRoYW4gVmFzcXVleiAmbHQ7PGEgaHJlZj0ibWFpbHRvOmpvbkB4eWlubi5vcmciIHRhcmdl dD0iX2JsYW5rIiByZWw9Im5vcmVmZXJyZXIgbm9mb2xsb3cgbm9vcGVuZXIiPmpvbkB4eWlubi5v cmc8L2E+Jmd0OyBlc2NyZXZldSAoc8OhYmFkbywgMjAvMDkvMjAyNSDDoChzKSAwMjo0Mik6PGJy PjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjogMCAw IDAgLjhleDsgYm9yZGVyLWxlZnQ6IDFweCBzb2xpZCAjY2NjOyBwYWRkaW5nLWxlZnQ6IDFleCI+ SGV5IGFsbCwgSSd2ZSBzcGVudCBzb21lIHRpbWUgdG9kYXkgZG9pbmcgYSBnYW1pbmcgZGVtby4g SSd2ZSB1cGxvYWRlZCB0aGUgdmlkZW8gaW4gMksgc28geW91IGNhbiBnZXQgdGhlIG1heGltdW0g ZWZmZWN0LiBJdCdzIGJlZW4gYW4gaW50ZXJlc3RpbmcgZXhwZXJpZW5jZSBzbyBmYXIsIGFuZCBJ J20gZ29pbmcgdG8gdHJ5IHRvIG1ha2UgaXQgbXkgcHJpbWFyeSBnYW1pbmcgbWFjaGluZS4gV2Un bGwgc2VlIGhvdyBpdCBnb2VzLiBJdCdzIGRlZmluaXRlbHkgcXdpcmt5LCBhbmQgc29tZXRpbWVz IEkgbmVlZCB0byByZWJvb3QgdGhlIFZNIDQtNSB0aW1lcyBiZWZvcmUgaXQgInN0YWJpbGl6ZXMi ICh3aGljaCBtZWFucyB0aGUgVk0gd29uJ3QganVzdCByYW5kb21seSBjcmFzaCBvciBoYXZlIG1h c3NpdmUgbGFnIGluIGdhbWUpLiBJJ2xsIG5lZWQgdG8gY29sbGVjdCBtb3JlIGRhdGEgdGhyb3Vn aCBwbGF5IHRlc3RpbmcuPGJyPg0KPGJyPg0KWW91IGNhbiB0YWtlIGEgbG9vayBhdCB0aGUgdmlk ZW8gaGVyZSAoSSdsbCBwb3N0IHRpbWVzdGFtcHMgaW4gdGhlIHZpZGVvIHNvb24pOjxicj4NCjxi cj4NCjxhIGhyZWY9Imh0dHBzOi8veW91dHUuYmUvX2N6MFJVQXc1cDgiIHJlbD0ibm9yZWZlcnJl ciBub2ZvbGxvdyBub29wZW5lciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8veW91dHUuYmUvX2N6 MFJVQXc1cDg8L2E+PGJyPg0KPGJyPg0KSm9uYXRoYW4gVmFzcXVlejxicj4NClBHUDogMzREQSA4 NThDIDE0NDcgNTA5RSBDNzdBIEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3OTxicj4NClNlbnQgd2l0 aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbDxicj4NCjxicj4NCjxicj4NCjxicj4NCk9uIFdlZG5l c2RheSwgU2VwdGVtYmVyIDE3dGgsIDIwMjUgYXQgMjI6MzMsIEpvbmF0aGFuIFZhc3F1ZXogJmx0 OzxhIGhyZWY9Im1haWx0bzpqb25AeHlpbm4ub3JnIiB0YXJnZXQ9Il9ibGFuayIgcmVsPSJub3Jl ZmVycmVyIG5vZm9sbG93IG5vb3BlbmVyIj5qb25AeHlpbm4ub3JnPC9hPiZndDsgd3JvdGU6PGJy Pg0KPGJyPg0KJmd0OyBBbHJpZ2h0ISBUaGUgWW91VHViZSBWaWRlbyBpcyB1cC4gTGV0IG1lIGtu b3cgd2hhdCB5b3UgdGhpbmsuIEknbSByZWFsbHkgZXhjaXRlZCBhYm91dCB0aGlzLjxicj4NCiZn dDsgPGJyPg0KJmd0OyA8YSBocmVmPSJodHRwczovL3lvdXR1LmJlL09iNC12N2RUSkdzIiByZWw9 Im5vcmVmZXJyZXIgbm9mb2xsb3cgbm9vcGVuZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwczovL3lv dXR1LmJlL09iNC12N2RUSkdzPC9hPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT48L2Rpdj48ZGl2 PjxiciBjbGVhcj0iYWxsIj48L2Rpdj48YnI+PHNwYW4gY2xhc3M9ImdtYWlsX3NpZ25hdHVyZV9w cmVmaXgiPi0tIDwvc3Bhbj48YnI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25hdHVy ZSI+PGRpdiBkaXI9Imx0ciI+PGRpdj48Zm9udCBjb2xvcj0iIzg4ODg4OCI+TnVubyBUZWl4ZWly YTwvZm9udD48L2Rpdj48ZGl2PjxkaXY+PGZvbnQgY29sb3I9IiM4ODg4ODgiPg0KRnJlZUJTRCBV TklYOiAgJmx0O2VkdWFyZG9ARnJlZUJTRC5vcmcmZ3Q7ICAgV2ViOiAgPGEgaHJlZj0iaHR0cHM6 Ly9mcmVlYnNkLm9yZy8iIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciIgdGFyZ2V0 PSJfYmxhbmsiPmh0dHBzOi8vRnJlZUJTRC5vcmc8L2E+PGJyPjwvZm9udD48L2Rpdj48L2Rpdj48 L2Rpdj48L2Rpdj4NCg0KICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwvZGl2Pg0KICAg ICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwvZGl2Pg0KICAgICAgICA8L2Jsb2NrcXVvdGU+ PGJyPg0KICAgIDwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj4NCiAgICAgICAgICAgICAgICAgICAg DQogICAgICAgIDwvYmxvY2txdW90ZT48YnI+DQogICAgPC9kaXY+ --b1=_OdLJLr8sUAwsmR6QT9puQVzBTbPy2TZJZy1FfVY-- From nobody Mon Sep 22 00:20:30 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVP0M38r9z68F8b for ; Mon, 22 Sep 2025 00:20:51 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVP0K4qcMz3yH6 for ; Mon, 22 Sep 2025 00:20:49 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=iitbombay.org header.s=google header.b=O0Q0MiDK; dmarc=pass (policy=quarantine) header.from=iitbombay.org; spf=pass (mx1.freebsd.org: domain of bakul@iitbombay.org designates 2607:f8b0:4864:20::1032 as permitted sender) smtp.mailfrom=bakul@iitbombay.org Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-33257e0fb88so163272a91.1 for ; Sun, 21 Sep 2025 17:20:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay.org; s=google; t=1758500443; x=1759105243; darn=freebsd.org; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=xp2d13DYV5iwo54E0Hj25N40DjAn/tFY6JvbakEU7Fw=; b=O0Q0MiDKX/UeirMv0hIPe3vFHRwyUchD9uVUAOF4xUABLfpy9aWlODTeTjBP8HAlNO ezxjPuxDNPaZYVd9zXFZoh06egId8gGbG1J55DHcpeF9E8as3eUNvQI60QTVMI0mbu8K 43/yhcxbTlSbFRNEStRgAl1REPsDly7VKN+mM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758500443; x=1759105243; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xp2d13DYV5iwo54E0Hj25N40DjAn/tFY6JvbakEU7Fw=; b=fshIGPT8RBfm1nX0eWXero5r5suHpehCWdurTP5Ocv2eY3F54QuX7xeCD1rNNEn6rM qC+zuaqQC9gBVGPHTRIcWPt3euPHW0zr0O/0rjaUqe2uC+8qhBSeLpRB7YJqFjucM0Cp l7Xap+T45ktoimQmwSJiQK8JnEBW4x9HYzu0J+OBTmwAcWFX5JxHxwMPNaE/lzX/NYLF UCwJ355JWqpFM5xwlDfYTkJJxFYW3mn9x5ZNawlToParUh0y04ZmKXOQsfrNAbyGhY2Y PU69XpfwU5h43V/A77VDPY/Ol0ZSv2EGdjSbtogqMnvp6d3rqniDXDOFyr4F2ZZnb51U g8hQ== X-Gm-Message-State: AOJu0Yxtw+ACNnkAH5jEXoxxVWPZ+a9NW432EEP7STwDVyR6Qq6j3o2v wgWiz/Gyv10U5bL3nJ4ENcI+cKhbcaEV7MkSbaXRwTMTmyc5ZOW0KUMDFmIEs3K4zaePZUP+bbu wFc4= X-Gm-Gg: ASbGncu2PQzaBbZunpGNMF4zryxNv8xDxwTvLZ2BD1RAKpAhrIAcuNvwUTF9prZOLgk bmiLoj65cIiCpJo0DLE5Dk+ie4dJjrM3G7nhAYKz/YBGNcoZfuR/eFYl6AOp30+vt+EI7+PMxKa udOvEEN3hkmBOrdVT2pJh3RwT75p6aXgThALbSzwHw2+Z40V8xWeKkR8C2J4EIQUjVg0Zs/RapG 6LA77CjttN6LZT6xORFQTizyTuK2CSLqAj6us7x4VqewsNeSPfzaGGrnvMXPR0tJQWjiXQq6oII FvGCqDwZfr56OwNc+AIYXc1+1tepVLxdykQasd11YhTfmJukTGd4QFRNeqjo/pV4ZA1/mFPX/5v z95+bbiaLazAvfT/1yQ7I/IGEUhggZ+edcvrB+APzytpidVm//jMERJiQNO1YjqMmH2qcoFLNsX DrA/9uhHj99tc/GlIlQl5yuE005eSFMcV9Rjo= X-Google-Smtp-Source: AGHT+IE8gaw/f6gOEb3ofxX8B+Sfbgl5wSM2e6M0W6y3xkoHXtJ5EUIq4tm2N7qpRf8A0KHaQVzcgg== X-Received: by 2002:a17:90b:4b4f:b0:330:3fb7:d874 with SMTP id 98e67ed59e1d1-330983a5927mr7599597a91.8.1758500442696; Sun, 21 Sep 2025 17:20:42 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-32ed275fb3csm14353007a91.20.2025.09.21.17.20.42 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Sep 2025 17:20:42 -0700 (PDT) From: Bakul Shah Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.100.1.1.5\)) Subject: Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. Message-Id: Date: Sun, 21 Sep 2025 17:20:30 -0700 To: Virtualisation on FreeBSD X-Mailer: Apple Mail (2.3864.100.1.1.5) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[iitbombay.org,quarantine]; R_DKIM_ALLOW(-0.20)[iitbombay.org:s=google]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[bakul]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1032:from]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; DKIM_TRACE(0.00)[iitbombay.org:+] X-Rspamd-Queue-Id: 4cVP0K4qcMz3yH6 On a 15.0-ALPHA3 host, bhyve fails when I try to start a 9front VM. Error: Assertion failed: (error =3D=3D 0), function modify_bar_registration, = file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. fbuf frame buffer base: 0x38bfae800000 [sz 33554432] I don't see any recent changes in bhyve/pci_emul.c recently. >>>This used to work fine when the same machine ran stable/14....<<< I can boot bsd and linux vms that I tried fine but not 9front. This is what I run: bhyve -c 1 \ -s 0,hostbridge \ -s 5,virtio-blk,9front.img \ -s 10,virtio-net,tap1,mac=3DXXXXXXXXXXXX \ -s 11,fbuf,tcp=3D0.0.0.0:5900,w=3D1920,h=3D1080,wait \ -s 20,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -m 4G -H -w \ 9front Any hints on how to debug this? Thanks, Bakul PS: stable/15 kernels also seem to be a bit unstable. Once in a while the machine just freezes. Happened twice in 4 days. Haven't looked into this yet. From nobody Mon Sep 22 00:20:30 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVP0L4d8qz68DhZ for ; Mon, 22 Sep 2025 00:20:50 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVP0K4L5Hz3yDp for ; Mon, 22 Sep 2025 00:20:49 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=iitbombay.org header.s=google header.b=R75aAVlf; dmarc=pass (policy=quarantine) header.from=iitbombay.org; spf=pass (mx1.freebsd.org: domain of bakul@iitbombay.org designates 2607:f8b0:4864:20::1033 as permitted sender) smtp.mailfrom=bakul@iitbombay.org Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-3307e8979f2so1146714a91.2 for ; Sun, 21 Sep 2025 17:20:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay.org; s=google; t=1758500442; x=1759105242; darn=freebsd.org; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=xp2d13DYV5iwo54E0Hj25N40DjAn/tFY6JvbakEU7Fw=; b=R75aAVlfzwryuDaCEHj4FA5YSfImrkeRG+2lPkVkIJujfnxhYhuBPP6IYECQ5Ng43z iFdtlC7jliNOp6N8vHpXonDkxfkP4T+Kk5kpFTYokZvfrrO7jGn8yQm3s7p1zF24yH02 z9KF4Pa2KnO6Df3CjzUfowcoudusV0/d8h4U0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758500442; x=1759105242; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xp2d13DYV5iwo54E0Hj25N40DjAn/tFY6JvbakEU7Fw=; b=vcMuhqwmXwO1iahWYHdIlWHUF+60wm38JGKJWl7zA5skNjB05wL5IpZZxSIr58PmRv LNiGU1Rj/W5O68wZEUzd9UymtRDBANIURkVSF5I0zGjO95PnVcauT3no9q6Drs1PyPtW MepF9U5kfDXJIeeik+OQ+ONM5110ZaWTudJQ18S0i4OuY0D7nlmoHYc1eTBnzIOVgT2z ZGhtZ6udI5O6IpKflkNMHR+d6kipv/YrL1c7Y1d4EY8tGaptWB/P/GpVMXjHgo7kFM6B 49jRXDwgjtB8MySpvz2K57/6EiUwVGTCXumsCSyVom4pkdfegv0bxr01ztsyW7nvfOni ur6w== X-Gm-Message-State: AOJu0YxAFmWxmzSx6sLPDhtJi2vtZfGYdQ109OPi3uPNFk/CVtOhCLI4 BjPfbYgmeZiGTLrchWBN9d7Xvyi/zIWWnSpy1y3J2MqADnTL/oaSFE9IfhFGUQW/z+z+ChRCn2x 4Ncc= X-Gm-Gg: ASbGncvT/YjpVHF3Ludawvn23rSA7t5NaxLj5PYgNmGTVMt8cwVVNaZE+F+wMWyrPrh cOWbC6qZM8NTYrNRcmUX0QxISpumECGSDPIuGFjT8Tb4CLTLH0D18tfSczh09ydufKv2U2ilTWD ajoh6RWePQ/6c1PKCuvDIaR3S9viFy9zCn82wT4a+rEHcO9LjK59hKo1NBbTOAS3/Ozb476rbB/ ZXs2tmsUusgtCfMNMs436uGEPG6Y1sImSPMOcN3fS39m4DqXBzj3BtZQZkkns6pHBc58iA/8w6U NO4rtfRfSHRG9vnj7xniLi568JpJr1HsckxVy/VQ6kH/9Hy5UFZMeLRdtKQ6jc/2SNNSJpXb3Jh GWEVS4WLCYRIXyYt5XGpxq8SISxMPIMV8j8zO79P6dp5lhg/4mDNA2OMiy8IrZuSzymgU2bKvtx k1O6JWvkkrUXSZlsqvmJ412wZP X-Google-Smtp-Source: AGHT+IHd8knXI8tILOqWmCcUvX+WXoWzr5XWGWOZK3XYSgHbgbszYf1HRbgtRFdnlJ9coA+QICF7cA== X-Received: by 2002:a17:902:d4d0:b0:25a:4437:dbb7 with SMTP id d9443c01a7336-269ba464124mr80109545ad.4.1758500442423; Sun, 21 Sep 2025 17:20:42 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2698018a9b7sm115759275ad.61.2025.09.21.17.20.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Sep 2025 17:20:42 -0700 (PDT) From: Bakul Shah Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.100.1.1.5\)) Subject: Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. Message-Id: Date: Sun, 21 Sep 2025 17:20:30 -0700 To: Virtualisation on FreeBSD X-Mailer: Apple Mail (2.3864.100.1.1.5) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[iitbombay.org,quarantine]; R_DKIM_ALLOW(-0.20)[iitbombay.org:s=google]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[bakul]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1033:from]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[iitbombay.org:+] X-Rspamd-Queue-Id: 4cVP0K4L5Hz3yDp On a 15.0-ALPHA3 host, bhyve fails when I try to start a 9front VM. Error: Assertion failed: (error =3D=3D 0), function modify_bar_registration, = file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. fbuf frame buffer base: 0x38bfae800000 [sz 33554432] I don't see any recent changes in bhyve/pci_emul.c recently. >>>This used to work fine when the same machine ran stable/14....<<< I can boot bsd and linux vms that I tried fine but not 9front. This is what I run: bhyve -c 1 \ -s 0,hostbridge \ -s 5,virtio-blk,9front.img \ -s 10,virtio-net,tap1,mac=3DXXXXXXXXXXXX \ -s 11,fbuf,tcp=3D0.0.0.0:5900,w=3D1920,h=3D1080,wait \ -s 20,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -m 4G -H -w \ 9front Any hints on how to debug this? Thanks, Bakul PS: stable/15 kernels also seem to be a bit unstable. Once in a while the machine just freezes. Happened twice in 4 days. Haven't looked into this yet. From nobody Mon Sep 22 06:48:17 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVYbj4XqCz68hZc for ; Mon, 22 Sep 2025 06:48:33 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yx1-xb12b.google.com (mail-yx1-xb12b.google.com [IPv6:2607:f8b0:4864:20::b12b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVYbh74sJz3gv3 for ; Mon, 22 Sep 2025 06:48:32 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yx1-xb12b.google.com with SMTP id 956f58d0204a3-60f476eab6cso2269631d50.2 for ; Sun, 21 Sep 2025 23:48:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1758523710; x=1759128510; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=oka9HBbq4UaNt3w+9LdnjsX98W21Cs1exqusr+oaikw=; b=K1JDKMXr4of/XsQ4dnTYK7qr3PGIIqWpiZtnRsnKefROHFDhy9G1OLB5pS7oFkuZu8 UwXaXLGFK7BIrT9AzZvBqteI8NGthSqZBDOekzWYOTxsnBlZuijEsVj1o4WwvqEao7Eh B8+1oLGab79rd8I98UoXvqrJQjMYbwFAAlT+nIgmgCU6UWvWQEGms89J83hnKesR4rNd xCMXjLe2O+0BW31HFl7owNUUN08kBpqdKcHF6n9mZ/t7SJGR8MfKfvMyPgZRlVJFbxD1 wIokRP2lI9+Kad8j53iLV85Hj1bZ3AsH3b5AXrCKcuRAv3dkBBoV4ZxDkEKQtswbb3WP 1c7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758523710; x=1759128510; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=oka9HBbq4UaNt3w+9LdnjsX98W21Cs1exqusr+oaikw=; b=YFztlGQbB8IGqUgvmkwdetNvPzBUhWrb1qI8THHsytDYoO6SzpCvkQ+CtP4ye/U13j U4D1uk4e0CKAg+iT6J6XrWecUDiKTIhRFqLT/hCNsGwN4C0pUMizjh+VPQR1ctyN6jkl TdV/r/gD++Id6dT0vKeOKiUISAr+uH5yMjbnhucnT/dvvuTwYFLw9wUO2UCswV2gIAW4 JBx9DQucSaDq+hyaTumXLozORrX+Uqrze/y6NTvkFCTAdciuI9ObODW4/c+hfXWeF2PY prNh8KWGYKvc7FgpRiWtGUJei5PKLoKIqnXyywn178+M6UzI1fZByqzoSjG5IPrDcu6Z wCxA== X-Gm-Message-State: AOJu0Ywa+VwWAl2+x16BYKrypyxpYDBLz4u0RX+k4h57v/40W0tVmehb htduVEINwONBbK7fCImW/Zc3eMvyFDwdt5v/xCecSBGP5e8ewO9IFRM0IUt9cj3vBAUEaUUrpdt 5FaY= X-Gm-Gg: ASbGncvsaFixVtmc9f1zWZiGmV7IkZy84T+1J5TTdMt4ijlF5nPn5Mnld3EtAYydp/f KgATKkUZGpvrngygc2teePkjNhXTuDJfvhBBj73Em4nAgEo7b18m4Ro/+/6oR/F3tYSH4myzt15 3fdm257ZZKftXa5eSAzCQ51zOlDfdUjo9Z6hSJb14tGFGJGIG8Q8nBMoRyY2AW9zoQ63oaEj5VM wSHP1wn2gK6k5+yy35IganH8YRfTpGtt8Y7utC+5+vDdfxB2ifcyS0KykKA3THbOZm4Uxz1MNMm Vz4qhop6nbVQV1FLZ1w/VI95inHNLZBGFl38ZjLJgNEGvUsGAvxcnvum05Sslkb0EUxlVwHVPI9 33PrD+7ZbtGusfgtOSP5ENtwKtJMGvLeeRh7un2D+NjN//FAcsSQ+tVE= X-Google-Smtp-Source: AGHT+IEpJ+QoBOIyFXSnfoWIc5WNZFO9Q60ZYxzFSqv+ql4eYrHAbhDRiag4jsw3UG5qxG3hCfpwRQ== X-Received: by 2002:a05:690e:164e:b0:632:ed6b:754 with SMTP id 956f58d0204a3-634779df1ffmr6239354d50.9.1758523710306; Sun, 21 Sep 2025 23:48:30 -0700 (PDT) Received: from mail-yx1-f41.google.com (mail-yx1-f41.google.com. [74.125.224.41]) by smtp.gmail.com with ESMTPSA id 00721157ae682-74b53fe4ef8sm9882617b3.61.2025.09.21.23.48.29 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 21 Sep 2025 23:48:29 -0700 (PDT) Received: by mail-yx1-f41.google.com with SMTP id 956f58d0204a3-60f476eab6cso2269609d50.2 for ; Sun, 21 Sep 2025 23:48:29 -0700 (PDT) X-Received: by 2002:a05:690e:4345:b0:634:e9b2:eedc with SMTP id 956f58d0204a3-634e9b2ef9amr5710395d50.38.1758523708772; Sun, 21 Sep 2025 23:48:28 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> In-Reply-To: From: Tomek CEDRO Date: Mon, 22 Sep 2025 08:48:17 +0200 X-Gmail-Original-Message-ID: X-Gm-Features: AS18NWABlz43JkypBKdKx0G4tSwJUBFKg9Dzug0-Tr9RamFjypLgczjt_YgzHmA Message-ID: Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) To: Jonathan Vasquez Cc: virtualization@freebsd.org, Nuno Teixeira , =?UTF-8?Q?Corvin_K=C3=B6hne?= , Desktop Mailing List Content-Type: multipart/alternative; boundary="000000000000fd962e063f5e34ad" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cVYbh74sJz3gv3 --000000000000fd962e063f5e34ad Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable what i noticed that the new pc have very complex bios setup options that may impact stability of freebsd. i encountered this recently after hw upgrade. pci-e gen 4 or 5, x16 or x8 mode, and split controller between single x16 for gpu or x8 for gpu and x8/x4 between other peripherals such as nvme controller usb additional card slots etc. in addition to that there are power modes and runtime frequency changes for cpu and all peripherals. not all new features are supported by freebsd. it took me some while to adjust optimal settings to get machine stable (at the cost of power savings). also there were irritating glitches in audio that are now considerably less noticable. long story short it is cpu, ram, and peripherals settings that may impact overall stability of the freebsd box. this work seems to be not only important for virtualization but also FreeBSD's LDWG project (Laptop Desktop Work Group) i am attaching freebsd-desktop@ mailing list :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info On Mon, Sep 22, 2025, 02:20 Jonathan Vasquez wrote: > Hey all, > > Just wanted to report that I still haven't received one crash after my > adjustments yesterday. So far over 20+ hours of uptime~. I've also reboot= ed > the VM/host a few times just to ensure that it is deterministically stabl= e > and so far zero issues. So this is pointing in a positive direction. I'm > tempted to mark this as a stable set up but I'll need to play for longer = to > make that call. > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Sunday, September 21st, 2025 at 08:23, Jonathan Vasquez > wrote: > > System has been up for about 9h40m. Most of it idle and no crashes yet. > I'm gonna reboot the host and see if the stability still continues. > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > > Sent from Proton Mail for Android. > > > -------- Original Message -------- > On Saturday, 09/20/25 at 23:45 Jonathan Vasquez wrote: > > Just finished beating the main story line in Cyberpunk 2077 in the VM. So > far no crashes and it's been running for an hour. So this is a positive > signal... we'll see what happens. I'll leave the VM running idle over nig= ht > and see if it's still running in the morning. Stay safe all! > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Saturday, September 20th, 2025 at 22:28, Jonathan Vasquez < > jon@xyinn.org> wrote: > > I did a bunch of more experiments today and I was able to switch from > using the USB controller connected to the multi-functional bus that has t= he > integrated card, and switched to a separate USB controller that only has > USB ports. I'm hoping that this increases VM stability and that there may > have been some wires being crossed. This was the previous bus I was using > (I couldn't isolate just the USB functions, it had to be all or nothing f= or > it to function): > > vgapci0@pci0:18:0:0: class=3D0x030000 rev=3D0xc1 hdr=3D0x00 vendor=3D0x10= 02 > device=3D0x164e subvendor=3D0x1043 subdevice=3D0x8877 > vendor =3D 'Advanced Micro Devices, Inc. [AMD/ATI]' > device =3D 'Raphael' > class =3D display > subclass =3D VGA > hdac0@pci0:18:0:1: class=3D0x040300 rev=3D0x00 hdr=3D0x00 vendor=3D0x1002 > device=3D0x1640 subvendor=3D0x1043 subdevice=3D0x8877 > vendor =3D 'Advanced Micro Devices, Inc. [AMD/ATI]' > device =3D 'Rembrandt Radeon High Definition Audio Controller' > class =3D multimedia > subclass =3D HDA > none1@pci0:18:0:2: class=3D0x108000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1022 > device=3D0x1649 subvendor=3D0x1043 subdevice=3D0x8877 > vendor =3D 'Advanced Micro Devices, Inc. [AMD]' > device =3D 'Family 19h PSP/CCP' > class =3D encrypt/decrypt > xhci1@pci0:18:0:3: class=3D0x0c0330 rev=3D0x00 hdr=3D0x00 vendor=3D0x1022 > device=3D0x15b6 subvendor=3D0x1043 subdevice=3D0x8877 > vendor =3D 'Advanced Micro Devices, Inc. [AMD]' > device =3D 'Raphael/Granite Ridge USB 3.1 xHCI' > class =3D serial bus > subclass =3D USB > xhci2@pci0:18:0:4: class=3D0x0c0330 rev=3D0x00 hdr=3D0x00 vendor=3D0x1022 > device=3D0x15b7 subvendor=3D0x1043 subdevice=3D0x8877 > vendor =3D 'Advanced Micro Devices, Inc. [AMD]' > device =3D 'Raphael/Granite Ridge USB 3.1 xHCI' > class =3D serial bus > subclass =3D USB > > Although now I'm longer using this at all for USB functionality in the VM= . > I've switched to: > > ppt2@pci0:13:0:0: class=3D0x0c0330 rev=3D0x01 hdr=3D0x00 vendor=3D0x1022 > device=3D0x43f7 subvendor=3D0x1b21 subdevice=3D0x1142 > vendor =3D 'Advanced Micro Devices, Inc. [AMD]' > device =3D '600 Series Chipset USB 3.2 Controller' > class =3D serial bus > subclass =3D USB > > Much cleaner and still contains a variety of high speed ports. Although > ultimately I'm just using one of them because I have a KVM connected to i= t > that allows me to plug in keyboard/mouse/3.5mm jack (audio)/ethernet all > through one USB wire. Although to remind everyone, I'm gaming in an offli= ne > capacity to I intentionally mess up the IPv4 gateway so my VM can only > communicate internally (and I can sideload apps/games) but prevent > Microsoft from reaching my machine and collecting telemetry info, and doi= ng > forced updates. Plus, I have no intention of moving to Windows 11 with al= l > of their new requirements and forced online activation (I know of the > workarounds, but I'm against on mandating a MSA on principle). People > should be able to fully use their machines even during installation/OOBE > without having to authenticate to a central server. Anyways, small rant > over lol. > > I'll update everyone on my progress with this re-configuration. > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Saturday, September 20th, 2025 at 20:02, Jonathan Vasquez < > jon@xyinn.org> wrote: > > Hey Nuno, > > > 1. I'm not. I plugged in my monitor's HDMI cable directly into the > GPU's HDMI port. Once bhyve starts up (if your passthrough is working > correctly and your GPU's drivers are installed), it will output to you= r > monitor. You would only need VNC for initial installation of Windows. > 2. I'm not sure of the NVIDIA case. I do know there is someone who > blogged about how they got theirs working: > https://dflund.se/~getz/Notes/2024/freebsd-gpu/ > > > Atm I'm currently trying to collect more info on why my bhyve VM randomly > crashes sometimes. So far I've collected that the exit reason bhyve is > showing is 0x60, which points to the following entries in the amd freebsd > kernel sources: > > 0x60 > > https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe136= 5978a282178/sys/amd64/vmm/amd/vmcb.h#L195 > > https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe136= 5978a282178/sys/amd64/vmm/amd/vmcb.h#L128C1-L128C30 > > XSETBV > > https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe136= 5978a282178/sys/amd64/vmm/amd/vmcb.h#L94 > > I also asked Proton's Lumo+ AI what is 0x60 to see what it comes up with, > it said the following: > > exit_reason[0] =3D 0x60 is the VM=E2=80=91exit code that bhyve reports wh= en the > guest triggers the XSETBV instruction. > Why it happens > > XSETBV writes to the extended control register XCR0. > In Intel=E2=80=AFVMX (the virtualization technology bhyve uses on x86= ), > execution of XSETBV is not allowed to run directly in the guest; the CPU > forces a VM=E2=80=91exit so the hypervisor can validate the operation. > bhyve records the exit reason as 0x60, which maps to the VMX exit > reason =E2=80=9CXSETBV=E2=80=9D in the Intel specification. > > What it means for you > > The guest tried to enable or change AVX=E2=80=91512 / XSAVE feature b= its via > XSETBV. > Because bhyve does not currently emulate that instruction, it aborts > the guest=E2=80=99s instruction stream and reports the exit code. > In most cases the guest will continue after bhyve handles the exit, > but if the hypervisor isn=E2=80=99t set up to emulate XSETBV you=E2=80=99= ll see the VM stop > or crash. > > Typical ways to deal with it > > Update bhyve =E2=80=93 newer releases include basic XSETBV handling f= or common > feature sets. > Adjust the guest=E2=80=99s CPU flags =E2=80=93 start the VM with a CP= U model that > doesn=E2=80=99t expose the XSAVE/XGETBV features the guest expects (e.g.,= -c host > vs. a specific model). > Patch bhyve =E2=80=93 if you=E2=80=99re comfortable with C, you can a= dd a handler for > VMX_EXIT_REASON_XSETBV that either emulates the instruction or simply > ignores it when the requested bits are already enabled. > > If you keep seeing the same exit repeatedly, it usually points to a > mismatch between the guest=E2=80=99s expectations (often a modern Linux k= ernel or a > Windows build that wants AVX=E2=80=91512) and the capabilities bhyve is p= resenting. > Aligning those expectations=E2=80=94either by tweaking the VM=E2=80=99s C= PU configuration > or by upgrading bhyve=E2=80=94generally resolves the issue. > > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Saturday, September 20th, 2025 at 19:47, Nuno Teixeira < > eduardo@freebsd.org> wrote: > > Hello Jonathan! > > Thanks for excellent quality videos on yt, I already subscribed it :) > > Just a quick questions: > > 1- Are you using freerdp3 to connect to win11pro? (Asking this because vn= c > is very slow) > 2- Do you know how is nvidia status compared to your use case? > > Thanks! > > Jonathan Vasquez escreveu (s=C3=A1bado, 20/09/2025 =C3=A0= (s) 02:42): > >> Hey all, I've spent some time today doing a gaming demo. I've uploaded >> the video in 2K so you can get the maximum effect. It's been an interest= ing >> experience so far, and I'm going to try to make it my primary gaming >> machine. We'll see how it goes. It's definitely qwirky, and sometimes I >> need to reboot the VM 4-5 times before it "stabilizes" (which means the = VM >> won't just randomly crash or have massive lag in game). I'll need to >> collect more data through play testing. >> >> You can take a look at the video here (I'll post timestamps in the video >> soon): >> >> https://youtu.be/_cz0RUAw5p8 >> >> Jonathan Vasquez >> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >> Sent with ProtonMail Secure Email >> >> >> >> On Wednesday, September 17th, 2025 at 22:33, Jonathan Vasquez < >> jon@xyinn.org> wrote: >> >> > Alright! The YouTube Video is up. Let me know what you think. I'm >> really excited about this. >> > >> > https://youtu.be/Ob4-v7dTJGs >> >> > > -- > Nuno Teixeira > FreeBSD UNIX: Web: https://FreeBSD.org > > > > > > > --000000000000fd962e063f5e34ad Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
what i noticed that the new pc have very complex bio= s setup options that may impact stability of freebsd. i encountered this re= cently after hw upgrade. pci-e gen 4 or 5, x16 or x8 mode, and split contro= ller between single x16 for gpu or x8 for gpu and x8/x4 between other perip= herals such as nvme controller usb additional card slots etc. in addition t= o that there are power modes and runtime frequency changes for cpu and all = peripherals. not all new features are supported by freebsd. it took me some= while to adjust optimal settings to get machine stable (at the cost of pow= er savings). also there were irritating glitches in audio that are now cons= iderably less noticable.

long story short it is cpu, ram, and peripherals settings that may impact = overall stability of the freebsd box.

this work seems to be not only important for virtualization b= ut also FreeBSD's LDWG project (Laptop Desktop Work Group) i am attachi= ng freebsd-desktop@ mailing list :-)

=
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
=
On Mon, Sep 22, 2025, 02:20 Jonathan Vasquez <jon@xyinn.org> wrote:
Hey all,

Just wanted to report that I still haven't received o= ne crash after my adjustments yesterday. So far over 20+ hours of uptime~. = I've also rebooted the VM/host a few times just to ensure that it is de= terministically stable and so far zero issues. So this is pointing in a pos= itive direction. I'm tempted to mark this as a stable set up but I'= ll need to play for longer to make that call.

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A = D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email

=20
=20

On Sunday, September 21st, 2025 at 08:23, Jonathan Vasquez <jon@xyinn= .org> wrote:
System has been up for about 9h40m. Most of it idle and no cras= hes yet. I'm gonna reboot the host and see if the stability still conti= nues.

Jonathan Vasquez
PGP: 34DA 858C 1447 5= 09E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secur= e Email



Sent from Proton Mail for Android.


-------- Original Mess= age --------
On Saturday, 09/20/25 at 23:45 Jonathan Vasquez <jon@xyinn.or= g> wrote:
Just finished beating the main story line in Cyberpunk 2077= in the VM. So far no crashes and it's been running for an hour. So thi= s is a positive signal... we'll see what happens. I'll leave the VM= running idle over night and see if it's still running in the morning. = Stay safe all!

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A = D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Saturday, September 20th, 2025 at 22:28, Jonathan Vasquez <jon@xyi= nn.org> wrote:
I di= d a bunch of more experiments today and I was able to switch from using the= USB controller connected to the multi-functional bus that has the integrat= ed card, and switched to a separate USB controller that only has USB ports.= I'm hoping that this increases VM stability and that there may have be= en some wires being crossed. This was the previous bus I was using (I could= n't isolate just the USB functions, it had to be all or nothing for it = to function):

vgapci0@pci0:18:0:0: class=3D0x030000 rev=3D0xc1 hdr=3D0x00 vendor=3D0x1= 002 device=3D0x164e subvendor=3D0x1043 subdevice=3D0x8877
= =C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'Advanced Micro Devices, Inc. [A= MD/ATI]'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D = 'Raphael'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 = =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D = VGA
hdac0@pci0:18:0:1: class=3D0x040300 rev=3D0x00 h= dr=3D0x00 vendor=3D0x1002 device=3D0x1640 subvendor=3D0x1043 subdevice=3D0x= 8877
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'Adv= anced Micro Devices, Inc. [AMD/ATI]'
=C2=A0 =C2= =A0 device =C2=A0 =C2=A0 =3D 'Rembrandt Radeon High Definition Audio Co= ntroller'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2= =A0=3D multimedia
=C2=A0 =C2=A0 subclass =C2=A0 =3D = HDA
none1@pci0:18:0:2: class=3D0x108000 rev=3D0x00 h= dr=3D0x00 vendor=3D0x1022 device=3D0x1649 subvendor=3D0x1043 subdevice=3D0x= 8877
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'Adv= anced Micro Devices, Inc. [AMD]'
=C2=A0 =C2=A0 d= evice =C2=A0 =C2=A0 =3D 'Family 19h PSP/CCP'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D encrypt/decrypt
xhci1@pci0:18:0:3: class=3D0x0c0330 rev=3D0x00 hdr=3D0x00 vendor= =3D0x1022 device=3D0x15b6 subvendor=3D0x1043 subdevice=3D0x8877
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'Advanced Micro Devi= ces, Inc. [AMD]'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'Raphael/Granite Ridge USB 3.1 xHCI'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D serial bus
=C2=A0 =C2=A0 subclass =C2=A0 =3D USB
xhci2@pci0= :18:0:4: class=3D0x0c0330 rev=3D0x00 hdr=3D0x00 vendor=3D0x1022 device=3D0x= 15b7 subvendor=3D0x1043 subdevice=3D0x8877
=C2=A0 = =C2=A0 vendor =C2=A0 =C2=A0 =3D 'Advanced Micro Devices, Inc. [AMD]'= ;
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'Raphae= l/Granite Ridge USB 3.1 xHCI'
=C2=A0 =C2=A0 clas= s =C2=A0 =C2=A0 =C2=A0=3D serial bus
=C2=A0 =C2=A0 subcla= ss =C2=A0 =3D USB

Although now I'm longer using this at all for USB functionality= in the VM. I've switched to:

ppt2@pci0:13:0:0: class=3D0x0c0330 rev=3D0x01 hdr=3D= 0x00 vendor=3D0x1022 device=3D0x43f7 subvendor=3D0x1b21 subdevice=3D0x1142<= /span>
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'Advanced Micro= Devices, Inc. [AMD]'
=C2=A0 =C2=A0 device =C2= =A0 =C2=A0 =3D '600 Series Chipset USB 3.2 Controller'
=
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D serial bus=C2=A0 =C2=A0 subclass =C2=A0 =3D USB

Much cleaner and still contains a vari= ety of high speed ports. Although ultimately I'm just using one of them= because I have a KVM connected to it that allows me to plug in keyboard/mo= use/3.5mm jack (audio)/ethernet all through one USB wire. Although to remin= d everyone, I'm gaming in an offline capacity to I intentionally mess u= p the IPv4 gateway so my VM can only communicate internally (and I can side= load apps/games) but prevent Microsoft from reaching my machine and collect= ing telemetry info, and doing forced updates. Plus, I have no intention of = moving to Windows 11 with all of their new requirements and forced online a= ctivation (I know of the workarounds, but I'm against on mandating a MS= A on principle). People should be able to fully use their machines even dur= ing installation/OOBE without having to authenticate to a central server. A= nyways, small rant over lol.

I'll update everyone on my progress with this re-configurat= ion.

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A = D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Saturday, September 20th, 2025 at 20:02, Jonathan Vasquez <jon@xyi= nn.org> wrote:
Hey = Nuno,

<= /div>
  1. I'm not. I plugged in my monitor's HDMI cable direct= ly into the GPU's HDMI port. Once bhyve starts up (if your passthrough = is working correctly and your GPU's drivers are installed), it will out= put to your monitor. You would only need VNC for initial installation of Wi= ndows.
  2. I'= ;m not sure of the NVIDIA case. I do know there is someone who blogged abou= t how they got theirs working:=C2=A0https://dflund.se/~getz/Notes/2024/freebsd-gpu/

Atm I'm currently trying to colle= ct more info on why my bhyve VM randomly crashes sometimes. So far I've= collected that the exit reason bhyve is showing is 0x60, which points to t= he following entries in the amd freebsd kernel sources:


I also asked Proton's Lumo+ AI what is 0x60 to see what it comes = up with, it said the following:

exit_reason[= 0] =3D 0x60 is the VM=E2=80=91exit code that bhyve reports when the guest t= riggers the XSETBV instruction.
Why it happens

=C2=A0 =C2=A0 XSETBV writes to the extended cont= rol register XCR0.
=C2=A0 =C2=A0 In Intel=E2=80=AFVM= X (the virtualization technology bhyve uses on x86), execution of XSETBV is= not allowed to run directly in the guest; the CPU forces a VM=E2=80=91exit= so the hypervisor can validate the operation.
=C2= =A0 =C2=A0 bhyve records the exit reason as 0x60, which maps to the VMX exi= t reason =E2=80=9CXSETBV=E2=80=9D in the Intel specification.
<= div>
What it means for you

<= div>=C2=A0 =C2=A0 The guest tried to enable or change AVX=E2=80=91512= / XSAVE feature bits via XSETBV.
=C2=A0 =C2=A0 Beca= use bhyve does not currently emulate that instruction, it aborts the guest= =E2=80=99s instruction stream and reports the exit code.
<= span>=C2=A0 =C2=A0 In most cases the guest will continue after bhyve handle= s the exit, but if the hypervisor isn=E2=80=99t set up to emulate XSETBV yo= u=E2=80=99ll see the VM stop or crash.

Typical ways to deal with it

=C2=A0= =C2=A0 Update bhyve =E2=80=93 newer releases include basic XSETBV handling= for common feature sets.
=C2=A0 =C2=A0 Adjust the g= uest=E2=80=99s CPU flags =E2=80=93 start the VM with a CPU model that doesn= =E2=80=99t expose the XSAVE/XGETBV features the guest expects (e.g., -c hos= t vs. a specific model).
=C2=A0 =C2=A0 Patch bhyve = =E2=80=93 if you=E2=80=99re comfortable with C, you can add a handler for V= MX_EXIT_REASON_XSETBV that either emulates the instruction or simply ignore= s it when the requested bits are already enabled.

If you keep seeing the same exit repeatedly, it usually points= to a mismatch between the guest=E2=80=99s expectations (often a modern Lin= ux kernel or a Windows build that wants AVX=E2=80=91512) and the capabiliti= es bhyve is presenting. Aligning those expectations=E2=80=94either by tweak= ing the VM=E2=80=99s CPU configuration or by upgrading bhyve=E2=80=94genera= lly resolves the issue.


Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A = D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Saturday, September 20th, 2025 at 19:47, Nuno Teixeira <edua= rdo@freebsd.org> wrote:
Hello Jonathan!

= Thanks for excellent quality videos on yt, I already subscribed it :)
=

Just a quick questions:

1- Are= you using freerdp3 to connect to win11pro? (Asking this because vnc is ve= ry slow)
2- Do you know how is nvidia status compared to your use= case?

Thanks!

Jonathan Vasquez <jon@xyinn.org> escreveu (s=C3=A1bado, 20/09/2025 = =C3=A0(s) 02:42):
https://youtu.be/_cz0RUAw5p8

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email



On Wednesday, September 17th, 2025 at 22:33, Jonathan Vasquez <jon@xyinn.org> wrote:

> Alright! The YouTube Video is up. Let me know what you think. I'm = really excited about this.
>
> https://youtu.be/Ob4-v7dTJGs



--
Nuno Teixeira
=
FreeBSD UNIX: <eduardo@FreeBSD.org> Web: = https://FreeBSD.org



=20
--000000000000fd962e063f5e34ad-- From nobody Mon Sep 22 10:25:43 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVfQT38fFz68vRg; Mon, 22 Sep 2025 10:25:53 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.protonmail.ch (mail-4317.protonmail.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVfQS68nLz476k; Mon, 22 Sep 2025 10:25:52 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758536748; x=1758795948; bh=zfLl+r9nsKaeXL4NZHodmsJwWKmR6ca8OxP1t6i/i7Y=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Kp8n58le6JbvJPc+5N+ZLKv+Y4fONIWmwNkP/2uiC8Br7OWLIbhP3FM6yjQuY82PX usvVhKLhDM0swemesiv3YIB/hBbLD7j22dnG6Z9OoV/l5JY0ykv/AV/aBL2d8xr2bL zK2GtNmwmfbZdoj3utzu9qYHUCQgMEvZm7qGUzTL/EUAHscjtdw2Ry08UvJ5IXMZIE 4C45RcOT8UCoGKcdnq2o9HbV15uecMI57ymGLfdoJ/Kvc013OizI4s/R5Q/Q74hyMG 00LHaK5mfYy4Pfxu+CLVskgkrc5uIHeO2ZPxy0IPK7Y9b2s9Ifk06Ll5mlTdE1JaYC UDg2h64scvOew== Date: Mon, 22 Sep 2025 10:25:43 +0000 To: Tomek CEDRO From: Jonathan Vasquez Cc: virtualization@freebsd.org, Nuno Teixeira , =?utf-8?Q?Corvin_K=C3=B6hne?= , Desktop Mailing List Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: =?us-ascii?Q?<6CV-OY6BcErrWRit9jSpi6fWsYBG3E=5FZ3u6eTLPcz6foPAZV1gQpZYaZTR7JA=5F1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=3D@xyinn.org>________?= Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 93b74cfa8bb88788695d316e4ae21b20c0edce50 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1=_BARyjgvDZAqqPXTTnklG1rA9YLWIPNYCRXmUllUuo" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cVfQS68nLz476k --b1=_BARyjgvDZAqqPXTTnklG1rA9YLWIPNYCRXmUllUuo Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 VGhhbmtzIFRvbWVrLiBZZWEgZm9yIHN1cmUuIFRoZXJlIGFyZSBhIGxvdCBvZiBzZXR0aW5ncyBp biB0aGVyZSBub3dhZGF5cy4gSSBsb29rZWQgb3ZlciBhbGwgb2YgdGhlbSBidXQgd2hhdCBJIHRy aWVkIHRvIGRvIGZvciBteSBzYW5pdHkgKGxvbCkgaXMgdG8gImxvYWQgb3B0aW1pemVkIGRlZmF1 bHRzIiwgYW5kIGNoYW5nZSBhcyBsaXR0bGUgc2V0dGluZ3MgYXMgcG9zc2libGUuIEx1Y2tpbHkg SSB3YXMgYWJsZSB0byBmaW5kIGEgZ29vZCBjb25maWcuIFRoZXJlIHdlcmUgc29tZSBvdGhlciB0 aGluZ3MgSSB3YW50ZWQgZGlzYWJsZWQgdGhvdWdoLCBwcmltYXJpbHkgc29tZSBleHRyYSBtb2Jv IHBlcmlwaGVyYWxzIHRoYXQgSSBkaWRuJ3QgbmVlZC4gSSBkaXNhYmxlZCB0aGUgc3lzdGVtcyB3 aWZpL0JsdWV0b290aCBjb250cm9sbGVyLCBvbmJvYXJkIGF1ZGlvLCBhbmQgb25ib2FyZCBsYW4u IEkgaGF2ZSBhIHNlcGFyYXRlIElQWCBiYXNlZCBJbnRlbCBjYXJkIGZvciBFdGhlcm5ldCwgYW5k IG15IEtWTSBwcm92aWRlcyBhIHNlcGFyYXRlIDMuNW1tIGphY2sgdGhhdCBJIHBhc3MgaW50byB0 aGUgS1ZNLiBTbyB0aGUgaG9zdCBkb2Vzbid0IG5lZWQgdG8gaGF2ZSBtdWx0aXBsZSB1bnVzZWQg cGVyaXBoZXJhbHMuIEEgZmV3IHllYXJzIGFnbyB0aGVyZSB3YXMgYSBidWcgaW4gRnJlZUJTRCB3 aGVyZSB0aGUgb25ib2FyZCBuZXR3b3JrIGNvbnRyb2xsZXIgd291bGQgY2F1c2Ugc3lzdGVtIGNy YXNoZXMsIHNvIHRoYXQncyB3aHkgSSBkaXNhYmxlZCBpdCBhIGxvbmcgdGltZSBhZ28gYW5kIGdv dCBhIGJldHRlciBzdXBwb3J0ZWQgTklDLgoKSm9uYXRoYW4gVmFzcXVlegpQR1A6IDM0REEgODU4 QyAxNDQ3IDUwOUUgQzc3QSBENDlGIEZCODUgOTBCNyBDNENBIDUyNzkKU2VudCB3aXRoIFByb3Rv bk1haWwgU2VjdXJlIEVtYWlsCgpTZW50IGZyb20gW1Byb3RvbiBNYWlsXShodHRwczovL3Byb3Rv bi5tZS9tYWlsL2hvbWUpIGZvciBBbmRyb2lkLgoKLS0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAt LS0tLS0tLQpPbiBNb25kYXksIDA5LzIyLzI1IGF0IDAyOjQ4IFRvbWVrIENFRFJPIDx0b21la0Bj ZWRyby5pbmZvPiB3cm90ZToKCj4gd2hhdCBpIG5vdGljZWQgdGhhdCB0aGUgbmV3IHBjIGhhdmUg dmVyeSBjb21wbGV4IGJpb3Mgc2V0dXAgb3B0aW9ucyB0aGF0IG1heSBpbXBhY3Qgc3RhYmlsaXR5 IG9mIGZyZWVic2QuIGkgZW5jb3VudGVyZWQgdGhpcyByZWNlbnRseSBhZnRlciBodyB1cGdyYWRl LiBwY2ktZSBnZW4gNCBvciA1LCB4MTYgb3IgeDggbW9kZSwgYW5kIHNwbGl0IGNvbnRyb2xsZXIg YmV0d2VlbiBzaW5nbGUgeDE2IGZvciBncHUgb3IgeDggZm9yIGdwdSBhbmQgeDgveDQgYmV0d2Vl biBvdGhlciBwZXJpcGhlcmFscyBzdWNoIGFzIG52bWUgY29udHJvbGxlciB1c2IgYWRkaXRpb25h bCBjYXJkIHNsb3RzIGV0Yy4gaW4gYWRkaXRpb24gdG8gdGhhdCB0aGVyZSBhcmUgcG93ZXIgbW9k ZXMgYW5kIHJ1bnRpbWUgZnJlcXVlbmN5IGNoYW5nZXMgZm9yIGNwdSBhbmQgYWxsIHBlcmlwaGVy YWxzLiBub3QgYWxsIG5ldyBmZWF0dXJlcyBhcmUgc3VwcG9ydGVkIGJ5IGZyZWVic2QuIGl0IHRv b2sgbWUgc29tZSB3aGlsZSB0byBhZGp1c3Qgb3B0aW1hbCBzZXR0aW5ncyB0byBnZXQgbWFjaGlu ZSBzdGFibGUgKGF0IHRoZSBjb3N0IG9mIHBvd2VyIHNhdmluZ3MpLiBhbHNvIHRoZXJlIHdlcmUg aXJyaXRhdGluZyBnbGl0Y2hlcyBpbiBhdWRpbyB0aGF0IGFyZSBub3cgY29uc2lkZXJhYmx5IGxl c3Mgbm90aWNhYmxlLgo+Cj4gbG9uZyBzdG9yeSBzaG9ydCBpdCBpcyBjcHUsIHJhbSwgYW5kIHBl cmlwaGVyYWxzIHNldHRpbmdzIHRoYXQgbWF5IGltcGFjdCBvdmVyYWxsIHN0YWJpbGl0eSBvZiB0 aGUgZnJlZWJzZCBib3guCj4KPiB0aGlzIHdvcmsgc2VlbXMgdG8gYmUgbm90IG9ubHkgaW1wb3J0 YW50IGZvciB2aXJ0dWFsaXphdGlvbiBidXQgYWxzbyBGcmVlQlNEJ3MgTERXRyBwcm9qZWN0IChM YXB0b3AgRGVza3RvcCBXb3JrIEdyb3VwKSBpIGFtIGF0dGFjaGluZyBmcmVlYnNkLWRlc2t0b3BA IG1haWxpbmcgbGlzdCA6LSkKPgo+IC0tCj4gQ2VEZVJPTSwgU1E3TUhaLCBbaHR0cDovL3d3dy50 b21lay5jZWRyby5pbmZvXShodHRwOi8vd3d3LnRvbWVrLmNlZHJvLmluZm8vKQo+Cj4gT24gTW9u LCBTZXAgMjIsIDIwMjUsIDAyOjIwIEpvbmF0aGFuIFZhc3F1ZXogPGpvbkB4eWlubi5vcmc+IHdy b3RlOgo+Cj4+IEhleSBhbGwsCj4+Cj4+IEp1c3Qgd2FudGVkIHRvIHJlcG9ydCB0aGF0IEkgc3Rp bGwgaGF2ZW4ndCByZWNlaXZlZCBvbmUgY3Jhc2ggYWZ0ZXIgbXkgYWRqdXN0bWVudHMgeWVzdGVy ZGF5LiBTbyBmYXIgb3ZlciAyMCsgaG91cnMgb2YgdXB0aW1lfi4gSSd2ZSBhbHNvIHJlYm9vdGVk IHRoZSBWTS9ob3N0IGEgZmV3IHRpbWVzIGp1c3QgdG8gZW5zdXJlIHRoYXQgaXQgaXMgZGV0ZXJt aW5pc3RpY2FsbHkgc3RhYmxlIGFuZCBzbyBmYXIgemVybyBpc3N1ZXMuIFNvIHRoaXMgaXMgcG9p bnRpbmcgaW4gYSBwb3NpdGl2ZSBkaXJlY3Rpb24uIEknbSB0ZW1wdGVkIHRvIG1hcmsgdGhpcyBh cyBhIHN0YWJsZSBzZXQgdXAgYnV0IEknbGwgbmVlZCB0byBwbGF5IGZvciBsb25nZXIgdG8gbWFr ZSB0aGF0IGNhbGwuCj4+Cj4+IEpvbmF0aGFuIFZhc3F1ZXoKPj4gUEdQOiAzNERBIDg1OEMgMTQ0 NyA1MDlFIEM3N0EgRDQ5RiBGQjg1IDkwQjcgQzRDQSA1Mjc5Cj4+IFNlbnQgd2l0aCBQcm90b25N YWlsIFNlY3VyZSBFbWFpbAo+Pgo+PiBPbiBTdW5kYXksIFNlcHRlbWJlciAyMXN0LCAyMDI1IGF0 IDA4OjIzLCBKb25hdGhhbiBWYXNxdWV6IDxqb25AeHlpbm4ub3JnPiB3cm90ZToKPj4KPj4+IFN5 c3RlbSBoYXMgYmVlbiB1cCBmb3IgYWJvdXQgOWg0MG0uIE1vc3Qgb2YgaXQgaWRsZSBhbmQgbm8g Y3Jhc2hlcyB5ZXQuIEknbSBnb25uYSByZWJvb3QgdGhlIGhvc3QgYW5kIHNlZSBpZiB0aGUgc3Rh YmlsaXR5IHN0aWxsIGNvbnRpbnVlcy4KPj4+Cj4+PiBKb25hdGhhbiBWYXNxdWV6Cj4+PiBQR1A6 IDM0REEgODU4QyAxNDQ3IDUwOUUgQzc3QSBENDlGIEZCODUgOTBCNyBDNENBIDUyNzkKPj4+IFNl bnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbAo+Pj4KPj4+IFNlbnQgZnJvbSBbUHJvdG9u IE1haWxdKGh0dHBzOi8vcHJvdG9uLm1lL21haWwvaG9tZSkgZm9yIEFuZHJvaWQuCj4+Pgo+Pj4g LS0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLS0tLQo+Pj4gT24gU2F0dXJkYXksIDA5LzIw LzI1IGF0IDIzOjQ1IEpvbmF0aGFuIFZhc3F1ZXogPGpvbkB4eWlubi5vcmc+IHdyb3RlOgo+Pj4K Pj4+PiBKdXN0IGZpbmlzaGVkIGJlYXRpbmcgdGhlIG1haW4gc3RvcnkgbGluZSBpbiBDeWJlcnB1 bmsgMjA3NyBpbiB0aGUgVk0uIFNvIGZhciBubyBjcmFzaGVzIGFuZCBpdCdzIGJlZW4gcnVubmlu ZyBmb3IgYW4gaG91ci4gU28gdGhpcyBpcyBhIHBvc2l0aXZlIHNpZ25hbC4uLiB3ZSdsbCBzZWUg d2hhdCBoYXBwZW5zLiBJJ2xsIGxlYXZlIHRoZSBWTSBydW5uaW5nIGlkbGUgb3ZlciBuaWdodCBh bmQgc2VlIGlmIGl0J3Mgc3RpbGwgcnVubmluZyBpbiB0aGUgbW9ybmluZy4gU3RheSBzYWZlIGFs bCEKPj4+Pgo+Pj4+IEpvbmF0aGFuIFZhc3F1ZXoKPj4+PiBQR1A6IDM0REEgODU4QyAxNDQ3IDUw OUUgQzc3QSBENDlGIEZCODUgOTBCNyBDNENBIDUyNzkKPj4+PiBTZW50IHdpdGggUHJvdG9uTWFp bCBTZWN1cmUgRW1haWwKPj4+Pgo+Pj4+IE9uIFNhdHVyZGF5LCBTZXB0ZW1iZXIgMjB0aCwgMjAy NSBhdCAyMjoyOCwgSm9uYXRoYW4gVmFzcXVleiA8am9uQHh5aW5uLm9yZz4gd3JvdGU6Cj4+Pj4K Pj4+Pj4gSSBkaWQgYSBidW5jaCBvZiBtb3JlIGV4cGVyaW1lbnRzIHRvZGF5IGFuZCBJIHdhcyBh YmxlIHRvIHN3aXRjaCBmcm9tIHVzaW5nIHRoZSBVU0IgY29udHJvbGxlciBjb25uZWN0ZWQgdG8g dGhlIG11bHRpLWZ1bmN0aW9uYWwgYnVzIHRoYXQgaGFzIHRoZSBpbnRlZ3JhdGVkIGNhcmQsIGFu ZCBzd2l0Y2hlZCB0byBhIHNlcGFyYXRlIFVTQiBjb250cm9sbGVyIHRoYXQgb25seSBoYXMgVVNC IHBvcnRzLiBJJ20gaG9waW5nIHRoYXQgdGhpcyBpbmNyZWFzZXMgVk0gc3RhYmlsaXR5IGFuZCB0 aGF0IHRoZXJlIG1heSBoYXZlIGJlZW4gc29tZSB3aXJlcyBiZWluZyBjcm9zc2VkLiBUaGlzIHdh cyB0aGUgcHJldmlvdXMgYnVzIEkgd2FzIHVzaW5nIChJIGNvdWxkbid0IGlzb2xhdGUganVzdCB0 aGUgVVNCIGZ1bmN0aW9ucywgaXQgaGFkIHRvIGJlIGFsbCBvciBub3RoaW5nIGZvciBpdCB0byBm dW5jdGlvbik6Cj4+Pj4+Cj4+Pj4+IHZnYXBjaTBAcGNpMDoxODowOjA6IGNsYXNzPTB4MDMwMDAw IHJldj0weGMxIGhkcj0weDAwIHZlbmRvcj0weDEwMDIgZGV2aWNlPTB4MTY0ZSBzdWJ2ZW5kb3I9 MHgxMDQzIHN1YmRldmljZT0weDg4NzcKPj4+Pj4gdmVuZG9yID0gJ0FkdmFuY2VkIE1pY3JvIERl dmljZXMsIEluYy4gW0FNRC9BVEldJwo+Pj4+PiBkZXZpY2UgPSAnUmFwaGFlbCcKPj4+Pj4gY2xh c3MgPSBkaXNwbGF5Cj4+Pj4+IHN1YmNsYXNzID0gVkdBCj4+Pj4+IGhkYWMwQHBjaTA6MTg6MDox OiBjbGFzcz0weDA0MDMwMCByZXY9MHgwMCBoZHI9MHgwMCB2ZW5kb3I9MHgxMDAyIGRldmljZT0w eDE2NDAgc3VidmVuZG9yPTB4MTA0MyBzdWJkZXZpY2U9MHg4ODc3Cj4+Pj4+IHZlbmRvciA9ICdB ZHZhbmNlZCBNaWNybyBEZXZpY2VzLCBJbmMuIFtBTUQvQVRJXScKPj4+Pj4gZGV2aWNlID0gJ1Jl bWJyYW5kdCBSYWRlb24gSGlnaCBEZWZpbml0aW9uIEF1ZGlvIENvbnRyb2xsZXInCj4+Pj4+IGNs YXNzID0gbXVsdGltZWRpYQo+Pj4+PiBzdWJjbGFzcyA9IEhEQQo+Pj4+PiBub25lMUBwY2kwOjE4 OjA6MjogY2xhc3M9MHgxMDgwMDAgcmV2PTB4MDAgaGRyPTB4MDAgdmVuZG9yPTB4MTAyMiBkZXZp Y2U9MHgxNjQ5IHN1YnZlbmRvcj0weDEwNDMgc3ViZGV2aWNlPTB4ODg3Nwo+Pj4+PiB2ZW5kb3Ig PSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywgSW5jLiBbQU1EXScKPj4+Pj4gZGV2aWNlID0gJ0Zh bWlseSAxOWggUFNQL0NDUCcKPj4+Pj4gY2xhc3MgPSBlbmNyeXB0L2RlY3J5cHQKPj4+Pj4geGhj aTFAcGNpMDoxODowOjM6IGNsYXNzPTB4MGMwMzMwIHJldj0weDAwIGhkcj0weDAwIHZlbmRvcj0w eDEwMjIgZGV2aWNlPTB4MTViNiBzdWJ2ZW5kb3I9MHgxMDQzIHN1YmRldmljZT0weDg4NzcKPj4+ Pj4gdmVuZG9yID0gJ0FkdmFuY2VkIE1pY3JvIERldmljZXMsIEluYy4gW0FNRF0nCj4+Pj4+IGRl dmljZSA9ICdSYXBoYWVsL0dyYW5pdGUgUmlkZ2UgVVNCIDMuMSB4SENJJwo+Pj4+PiBjbGFzcyA9 IHNlcmlhbCBidXMKPj4+Pj4gc3ViY2xhc3MgPSBVU0IKPj4+Pj4geGhjaTJAcGNpMDoxODowOjQ6 IGNsYXNzPTB4MGMwMzMwIHJldj0weDAwIGhkcj0weDAwIHZlbmRvcj0weDEwMjIgZGV2aWNlPTB4 MTViNyBzdWJ2ZW5kb3I9MHgxMDQzIHN1YmRldmljZT0weDg4NzcKPj4+Pj4gdmVuZG9yID0gJ0Fk dmFuY2VkIE1pY3JvIERldmljZXMsIEluYy4gW0FNRF0nCj4+Pj4+IGRldmljZSA9ICdSYXBoYWVs L0dyYW5pdGUgUmlkZ2UgVVNCIDMuMSB4SENJJwo+Pj4+PiBjbGFzcyA9IHNlcmlhbCBidXMgc3Vi Y2xhc3MgPSBVU0IKPj4+Pj4KPj4+Pj4gQWx0aG91Z2ggbm93IEknbSBsb25nZXIgdXNpbmcgdGhp cyBhdCBhbGwgZm9yIFVTQiBmdW5jdGlvbmFsaXR5IGluIHRoZSBWTS4gSSd2ZSBzd2l0Y2hlZCB0 bzoKPj4+Pj4KPj4+Pj4gcHB0MkBwY2kwOjEzOjA6MDogY2xhc3M9MHgwYzAzMzAgcmV2PTB4MDEg aGRyPTB4MDAgdmVuZG9yPTB4MTAyMiBkZXZpY2U9MHg0M2Y3IHN1YnZlbmRvcj0weDFiMjEgc3Vi ZGV2aWNlPTB4MTE0Mgo+Pj4+PiB2ZW5kb3IgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywgSW5j LiBbQU1EXScKPj4+Pj4gZGV2aWNlID0gJzYwMCBTZXJpZXMgQ2hpcHNldCBVU0IgMy4yIENvbnRy b2xsZXInCj4+Pj4+IGNsYXNzID0gc2VyaWFsIGJ1cyBzdWJjbGFzcyA9IFVTQgo+Pj4+Pgo+Pj4+ PiBNdWNoIGNsZWFuZXIgYW5kIHN0aWxsIGNvbnRhaW5zIGEgdmFyaWV0eSBvZiBoaWdoIHNwZWVk IHBvcnRzLiBBbHRob3VnaCB1bHRpbWF0ZWx5IEknbSBqdXN0IHVzaW5nIG9uZSBvZiB0aGVtIGJl Y2F1c2UgSSBoYXZlIGEgS1ZNIGNvbm5lY3RlZCB0byBpdCB0aGF0IGFsbG93cyBtZSB0byBwbHVn IGluIGtleWJvYXJkL21vdXNlLzMuNW1tIGphY2sgKGF1ZGlvKS9ldGhlcm5ldCBhbGwgdGhyb3Vn aCBvbmUgVVNCIHdpcmUuIEFsdGhvdWdoIHRvIHJlbWluZCBldmVyeW9uZSwgSSdtIGdhbWluZyBp biBhbiBvZmZsaW5lIGNhcGFjaXR5IHRvIEkgaW50ZW50aW9uYWxseSBtZXNzIHVwIHRoZSBJUHY0 IGdhdGV3YXkgc28gbXkgVk0gY2FuIG9ubHkgY29tbXVuaWNhdGUgaW50ZXJuYWxseSAoYW5kIEkg Y2FuIHNpZGVsb2FkIGFwcHMvZ2FtZXMpIGJ1dCBwcmV2ZW50IE1pY3Jvc29mdCBmcm9tIHJlYWNo aW5nIG15IG1hY2hpbmUgYW5kIGNvbGxlY3RpbmcgdGVsZW1ldHJ5IGluZm8sIGFuZCBkb2luZyBm b3JjZWQgdXBkYXRlcy4gUGx1cywgSSBoYXZlIG5vIGludGVudGlvbiBvZiBtb3ZpbmcgdG8gV2lu ZG93cyAxMSB3aXRoIGFsbCBvZiB0aGVpciBuZXcgcmVxdWlyZW1lbnRzIGFuZCBmb3JjZWQgb25s aW5lIGFjdGl2YXRpb24gKEkga25vdyBvZiB0aGUgd29ya2Fyb3VuZHMsIGJ1dCBJJ20gYWdhaW5z dCBvbiBtYW5kYXRpbmcgYSBNU0Egb24gcHJpbmNpcGxlKS4gUGVvcGxlIHNob3VsZCBiZSBhYmxl IHRvIGZ1bGx5IHVzZSB0aGVpciBtYWNoaW5lcyBldmVuIGR1cmluZyBpbnN0YWxsYXRpb24vT09C RSB3aXRob3V0IGhhdmluZyB0byBhdXRoZW50aWNhdGUgdG8gYSBjZW50cmFsIHNlcnZlci4gQW55 d2F5cywgc21hbGwgcmFudCBvdmVyIGxvbC4KPj4+Pj4KPj4+Pj4gSSdsbCB1cGRhdGUgZXZlcnlv bmUgb24gbXkgcHJvZ3Jlc3Mgd2l0aCB0aGlzIHJlLWNvbmZpZ3VyYXRpb24uCj4+Pj4+Cj4+Pj4+ IEpvbmF0aGFuIFZhc3F1ZXoKPj4+Pj4gUEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgRDQ5 RiBGQjg1IDkwQjcgQzRDQSA1Mjc5Cj4+Pj4+IFNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBF bWFpbAo+Pj4+Pgo+Pj4+PiBPbiBTYXR1cmRheSwgU2VwdGVtYmVyIDIwdGgsIDIwMjUgYXQgMjA6 MDIsIEpvbmF0aGFuIFZhc3F1ZXogPGpvbkB4eWlubi5vcmc+IHdyb3RlOgo+Pj4+Pgo+Pj4+Pj4g SGV5IE51bm8sCj4+Pj4+Pgo+Pj4+Pj4gLSBJJ20gbm90LiBJIHBsdWdnZWQgaW4gbXkgbW9uaXRv cidzIEhETUkgY2FibGUgZGlyZWN0bHkgaW50byB0aGUgR1BVJ3MgSERNSSBwb3J0LiBPbmNlIGJo eXZlIHN0YXJ0cyB1cCAoaWYgeW91ciBwYXNzdGhyb3VnaCBpcyB3b3JraW5nIGNvcnJlY3RseSBh bmQgeW91ciBHUFUncyBkcml2ZXJzIGFyZSBpbnN0YWxsZWQpLCBpdCB3aWxsIG91dHB1dCB0byB5 b3VyIG1vbml0b3IuIFlvdSB3b3VsZCBvbmx5IG5lZWQgVk5DIGZvciBpbml0aWFsIGluc3RhbGxh dGlvbiBvZiBXaW5kb3dzLgo+Pj4+Pj4gLSBJJ20gbm90IHN1cmUgb2YgdGhlIE5WSURJQSBjYXNl LiBJIGRvIGtub3cgdGhlcmUgaXMgc29tZW9uZSB3aG8gYmxvZ2dlZCBhYm91dCBob3cgdGhleSBn b3QgdGhlaXJzIHdvcmtpbmc6IGh0dHBzOi8vZGZsdW5kLnNlL35nZXR6L05vdGVzLzIwMjQvZnJl ZWJzZC1ncHUvCj4+Pj4+Pgo+Pj4+Pj4gQXRtIEknbSBjdXJyZW50bHkgdHJ5aW5nIHRvIGNvbGxl Y3QgbW9yZSBpbmZvIG9uIHdoeSBteSBiaHl2ZSBWTSByYW5kb21seSBjcmFzaGVzIHNvbWV0aW1l cy4gU28gZmFyIEkndmUgY29sbGVjdGVkIHRoYXQgdGhlIGV4aXQgcmVhc29uIGJoeXZlIGlzIHNo b3dpbmcgaXMgMHg2MCwgd2hpY2ggcG9pbnRzIHRvIHRoZSBmb2xsb3dpbmcgZW50cmllcyBpbiB0 aGUgYW1kIGZyZWVic2Qga2VybmVsIHNvdXJjZXM6Cj4+Pj4+Pgo+Pj4+Pj4gMHg2MAo+Pj4+Pj4g aHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMvYmxvYi80YzNhODY4ZDEzYzA1 M2VmMTczMjY4Y2RmZTEzNjU5NzhhMjgyMTc4L3N5cy9hbWQ2NC92bW0vYW1kL3ZtY2IuaCNMMTk1 Cj4+Pj4+PiBodHRwczovL2dpdGh1Yi5jb20vZnJlZWJzZC9mcmVlYnNkLXNyYy9ibG9iLzRjM2E4 NjhkMTNjMDUzZWYxNzMyNjhjZGZlMTM2NTk3OGEyODIxNzgvc3lzL2FtZDY0L3ZtbS9hbWQvdm1j Yi5oI0wxMjhDMS1MMTI4QzMwCj4+Pj4+Pgo+Pj4+Pj4gWFNFVEJWCj4+Pj4+PiBodHRwczovL2dp dGh1Yi5jb20vZnJlZWJzZC9mcmVlYnNkLXNyYy9ibG9iLzRjM2E4NjhkMTNjMDUzZWYxNzMyNjhj ZGZlMTM2NTk3OGEyODIxNzgvc3lzL2FtZDY0L3ZtbS9hbWQvdm1jYi5oI0w5NAo+Pj4+Pj4KPj4+ Pj4+IEkgYWxzbyBhc2tlZCBQcm90b24ncyBMdW1vKyBBSSB3aGF0IGlzIDB4NjAgdG8gc2VlIHdo YXQgaXQgY29tZXMgdXAgd2l0aCwgaXQgc2FpZCB0aGUgZm9sbG93aW5nOgo+Pj4+Pj4KPj4+Pj4+ IGV4aXRfcmVhc29uWzBdID0gMHg2MCBpcyB0aGUgVk3igJFleGl0IGNvZGUgdGhhdCBiaHl2ZSBy ZXBvcnRzIHdoZW4gdGhlIGd1ZXN0IHRyaWdnZXJzIHRoZSBYU0VUQlYgaW5zdHJ1Y3Rpb24uCj4+ Pj4+PiBXaHkgaXQgaGFwcGVucwo+Pj4+Pj4KPj4+Pj4+IFhTRVRCViB3cml0ZXMgdG8gdGhlIGV4 dGVuZGVkIGNvbnRyb2wgcmVnaXN0ZXIgWENSMC4KPj4+Pj4+IEluIEludGVs4oCvVk1YICh0aGUg dmlydHVhbGl6YXRpb24gdGVjaG5vbG9neSBiaHl2ZSB1c2VzIG9uIHg4NiksIGV4ZWN1dGlvbiBv ZiBYU0VUQlYgaXMgbm90IGFsbG93ZWQgdG8gcnVuIGRpcmVjdGx5IGluIHRoZSBndWVzdDsgdGhl IENQVSBmb3JjZXMgYSBWTeKAkWV4aXQgc28gdGhlIGh5cGVydmlzb3IgY2FuIHZhbGlkYXRlIHRo ZSBvcGVyYXRpb24uCj4+Pj4+PiBiaHl2ZSByZWNvcmRzIHRoZSBleGl0IHJlYXNvbiBhcyAweDYw LCB3aGljaCBtYXBzIHRvIHRoZSBWTVggZXhpdCByZWFzb24g4oCcWFNFVEJW4oCdIGluIHRoZSBJ bnRlbCBzcGVjaWZpY2F0aW9uLgo+Pj4+Pj4KPj4+Pj4+IFdoYXQgaXQgbWVhbnMgZm9yIHlvdQo+ Pj4+Pj4KPj4+Pj4+IFRoZSBndWVzdCB0cmllZCB0byBlbmFibGUgb3IgY2hhbmdlIEFWWOKAkTUx MiAvIFhTQVZFIGZlYXR1cmUgYml0cyB2aWEgWFNFVEJWLgo+Pj4+Pj4gQmVjYXVzZSBiaHl2ZSBk b2VzIG5vdCBjdXJyZW50bHkgZW11bGF0ZSB0aGF0IGluc3RydWN0aW9uLCBpdCBhYm9ydHMgdGhl IGd1ZXN04oCZcyBpbnN0cnVjdGlvbiBzdHJlYW0gYW5kIHJlcG9ydHMgdGhlIGV4aXQgY29kZS4K Pj4+Pj4+IEluIG1vc3QgY2FzZXMgdGhlIGd1ZXN0IHdpbGwgY29udGludWUgYWZ0ZXIgYmh5dmUg aGFuZGxlcyB0aGUgZXhpdCwgYnV0IGlmIHRoZSBoeXBlcnZpc29yIGlzbuKAmXQgc2V0IHVwIHRv IGVtdWxhdGUgWFNFVEJWIHlvdeKAmWxsIHNlZSB0aGUgVk0gc3RvcCBvciBjcmFzaC4KPj4+Pj4+ Cj4+Pj4+PiBUeXBpY2FsIHdheXMgdG8gZGVhbCB3aXRoIGl0Cj4+Pj4+Pgo+Pj4+Pj4gVXBkYXRl IGJoeXZlIOKAkyBuZXdlciByZWxlYXNlcyBpbmNsdWRlIGJhc2ljIFhTRVRCViBoYW5kbGluZyBm b3IgY29tbW9uIGZlYXR1cmUgc2V0cy4KPj4+Pj4+IEFkanVzdCB0aGUgZ3Vlc3TigJlzIENQVSBm bGFncyDigJMgc3RhcnQgdGhlIFZNIHdpdGggYSBDUFUgbW9kZWwgdGhhdCBkb2VzbuKAmXQgZXhw b3NlIHRoZSBYU0FWRS9YR0VUQlYgZmVhdHVyZXMgdGhlIGd1ZXN0IGV4cGVjdHMgKGUuZy4sIC1j IGhvc3QgdnMuIGEgc3BlY2lmaWMgbW9kZWwpLgo+Pj4+Pj4gUGF0Y2ggYmh5dmUg4oCTIGlmIHlv deKAmXJlIGNvbWZvcnRhYmxlIHdpdGggQywgeW91IGNhbiBhZGQgYSBoYW5kbGVyIGZvciBWTVhf RVhJVF9SRUFTT05fWFNFVEJWIHRoYXQgZWl0aGVyIGVtdWxhdGVzIHRoZSBpbnN0cnVjdGlvbiBv ciBzaW1wbHkgaWdub3JlcyBpdCB3aGVuIHRoZSByZXF1ZXN0ZWQgYml0cyBhcmUgYWxyZWFkeSBl bmFibGVkLgo+Pj4+Pj4KPj4+Pj4+IElmIHlvdSBrZWVwIHNlZWluZyB0aGUgc2FtZSBleGl0IHJl cGVhdGVkbHksIGl0IHVzdWFsbHkgcG9pbnRzIHRvIGEgbWlzbWF0Y2ggYmV0d2VlbiB0aGUgZ3Vl c3TigJlzIGV4cGVjdGF0aW9ucyAob2Z0ZW4gYSBtb2Rlcm4gTGludXgga2VybmVsIG9yIGEgV2lu ZG93cyBidWlsZCB0aGF0IHdhbnRzIEFWWOKAkTUxMikgYW5kIHRoZSBjYXBhYmlsaXRpZXMgYmh5 dmUgaXMgcHJlc2VudGluZy4gQWxpZ25pbmcgdGhvc2UgZXhwZWN0YXRpb25z4oCUZWl0aGVyIGJ5 IHR3ZWFraW5nIHRoZSBWTeKAmXMgQ1BVIGNvbmZpZ3VyYXRpb24gb3IgYnkgdXBncmFkaW5nIGJo eXZl4oCUZ2VuZXJhbGx5IHJlc29sdmVzIHRoZSBpc3N1ZS4KPj4+Pj4+Cj4+Pj4+PiBKb25hdGhh biBWYXNxdWV6Cj4+Pj4+PiBQR1A6IDM0REEgODU4QyAxNDQ3IDUwOUUgQzc3QSBENDlGIEZCODUg OTBCNyBDNENBIDUyNzkKPj4+Pj4+IFNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbAo+ Pj4+Pj4KPj4+Pj4+IE9uIFNhdHVyZGF5LCBTZXB0ZW1iZXIgMjB0aCwgMjAyNSBhdCAxOTo0Nywg TnVubyBUZWl4ZWlyYSA8ZWR1YXJkb0BmcmVlYnNkLm9yZz4gd3JvdGU6Cj4+Pj4+Pgo+Pj4+Pj4+ IEhlbGxvIEpvbmF0aGFuIQo+Pj4+Pj4+Cj4+Pj4+Pj4gVGhhbmtzIGZvciBleGNlbGxlbnQgcXVh bGl0eSB2aWRlb3Mgb24geXQsIEkgYWxyZWFkeSBzdWJzY3JpYmVkIGl0IDopCj4+Pj4+Pj4KPj4+ Pj4+PiBKdXN0IGEgcXVpY2sgcXVlc3Rpb25zOgo+Pj4+Pj4+Cj4+Pj4+Pj4gMS0gQXJlIHlvdSB1 c2luZyBmcmVlcmRwMyB0byBjb25uZWN0IHRvIHdpbjExcHJvPyAoQXNraW5nIHRoaXMgYmVjYXVz ZSB2bmMgaXMgdmVyeSBzbG93KQo+Pj4+Pj4+IDItIERvIHlvdSBrbm93IGhvdyBpcyBudmlkaWEg c3RhdHVzIGNvbXBhcmVkIHRvIHlvdXIgdXNlIGNhc2U/Cj4+Pj4+Pj4KPj4+Pj4+PiBUaGFua3Mh Cj4+Pj4+Pj4KPj4+Pj4+PiBKb25hdGhhbiBWYXNxdWV6IDxqb25AeHlpbm4ub3JnPiBlc2NyZXZl dSAoc8OhYmFkbywgMjAvMDkvMjAyNSDDoChzKSAwMjo0Mik6Cj4+Pj4+Pj4KPj4+Pj4+Pj4gSGV5 IGFsbCwgSSd2ZSBzcGVudCBzb21lIHRpbWUgdG9kYXkgZG9pbmcgYSBnYW1pbmcgZGVtby4gSSd2 ZSB1cGxvYWRlZCB0aGUgdmlkZW8gaW4gMksgc28geW91IGNhbiBnZXQgdGhlIG1heGltdW0gZWZm ZWN0LiBJdCdzIGJlZW4gYW4gaW50ZXJlc3RpbmcgZXhwZXJpZW5jZSBzbyBmYXIsIGFuZCBJJ20g Z29pbmcgdG8gdHJ5IHRvIG1ha2UgaXQgbXkgcHJpbWFyeSBnYW1pbmcgbWFjaGluZS4gV2UnbGwg c2VlIGhvdyBpdCBnb2VzLiBJdCdzIGRlZmluaXRlbHkgcXdpcmt5LCBhbmQgc29tZXRpbWVzIEkg bmVlZCB0byByZWJvb3QgdGhlIFZNIDQtNSB0aW1lcyBiZWZvcmUgaXQgInN0YWJpbGl6ZXMiICh3 aGljaCBtZWFucyB0aGUgVk0gd29uJ3QganVzdCByYW5kb21seSBjcmFzaCBvciBoYXZlIG1hc3Np dmUgbGFnIGluIGdhbWUpLiBJJ2xsIG5lZWQgdG8gY29sbGVjdCBtb3JlIGRhdGEgdGhyb3VnaCBw bGF5IHRlc3RpbmcuCj4+Pj4+Pj4+Cj4+Pj4+Pj4+IFlvdSBjYW4gdGFrZSBhIGxvb2sgYXQgdGhl IHZpZGVvIGhlcmUgKEknbGwgcG9zdCB0aW1lc3RhbXBzIGluIHRoZSB2aWRlbyBzb29uKToKPj4+ Pj4+Pj4KPj4+Pj4+Pj4gaHR0cHM6Ly95b3V0dS5iZS9fY3owUlVBdzVwOAo+Pj4+Pj4+Pgo+Pj4+ Pj4+PiBKb25hdGhhbiBWYXNxdWV6Cj4+Pj4+Pj4+IFBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBD NzdBIEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3OQo+Pj4+Pj4+PiBTZW50IHdpdGggUHJvdG9uTWFp bCBTZWN1cmUgRW1haWwKPj4+Pj4+Pj4KPj4+Pj4+Pj4gT24gV2VkbmVzZGF5LCBTZXB0ZW1iZXIg MTd0aCwgMjAyNSBhdCAyMjozMywgSm9uYXRoYW4gVmFzcXVleiA8am9uQHh5aW5uLm9yZz4gd3Jv dGU6Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+PiBBbHJpZ2h0ISBUaGUgWW91VHViZSBWaWRlbyBpcyB1cC4g TGV0IG1lIGtub3cgd2hhdCB5b3UgdGhpbmsuIEknbSByZWFsbHkgZXhjaXRlZCBhYm91dCB0aGlz Lgo+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+IGh0dHBzOi8veW91dHUuYmUvT2I0LXY3ZFRKR3MKPj4+Pj4+ Pgo+Pj4+Pj4+IC0tCj4+Pj4+Pj4KPj4+Pj4+PiBOdW5vIFRlaXhlaXJhCj4+Pj4+Pj4gRnJlZUJT RCBVTklYOiA8ZWR1YXJkb0BGcmVlQlNELm9yZz4gV2ViOiBbaHR0cHM6Ly9GcmVlQlNELm9yZ10o aHR0cHM6Ly9mcmVlYnNkLm9yZy8p --b1=_BARyjgvDZAqqPXTTnklG1rA9YLWIPNYCRXmUllUuo Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+PGhlYWQ+PC9oZWFkPjxib2R5PlRoYW5rcyBUb21lay4gWWVhIGZvciBzdXJlLiBUaGVy ZSBhcmUgYSBsb3Qgb2Ygc2V0dGluZ3MgaW4gdGhlcmUgbm93YWRheXMuIEkgbG9va2VkIG92ZXIg YWxsIG9mIHRoZW0gYnV0IHdoYXQgSSB0cmllZCB0byBkbyBmb3IgbXkgc2FuaXR5IChsb2wpIGlz IHRvICJsb2FkIG9wdGltaXplZCBkZWZhdWx0cyIsIGFuZCBjaGFuZ2UgYXMgbGl0dGxlIHNldHRp bmdzIGFzIHBvc3NpYmxlLiBMdWNraWx5IEkgd2FzIGFibGUgdG8gZmluZCBhIGdvb2QgY29uZmln LiBUaGVyZSB3ZXJlIHNvbWUgb3RoZXIgdGhpbmdzIEkgd2FudGVkIGRpc2FibGVkIHRob3VnaCwg cHJpbWFyaWx5IHNvbWUgZXh0cmEgbW9ibyBwZXJpcGhlcmFscyB0aGF0IEkgZGlkbid0IG5lZWQu IEkgZGlzYWJsZWQgdGhlIHN5c3RlbXMgd2lmaS9CbHVldG9vdGggY29udHJvbGxlciwgb25ib2Fy ZCBhdWRpbywgYW5kIG9uYm9hcmQgbGFuLiBJIGhhdmUgYSBzZXBhcmF0ZSBJUFggYmFzZWQgSW50 ZWwgY2FyZCBmb3IgRXRoZXJuZXQsIGFuZCBteSBLVk0gcHJvdmlkZXMgYSBzZXBhcmF0ZSAzLjVt bSBqYWNrIHRoYXQgSSBwYXNzIGludG8gdGhlIEtWTS4gU28gdGhlIGhvc3QgZG9lc24ndCBuZWVk IHRvIGhhdmUgbXVsdGlwbGUgdW51c2VkIHBlcmlwaGVyYWxzLiBBIGZldyB5ZWFycyBhZ28gdGhl cmUgd2FzIGEgYnVnIGluIEZyZWVCU0Qgd2hlcmUgdGhlIG9uYm9hcmQgbmV0d29yayBjb250cm9s bGVyIHdvdWxkIGNhdXNlIHN5c3RlbSBjcmFzaGVzLCBzbyB0aGF0J3Mgd2h5IEkgZGlzYWJsZWQg aXQgYSBsb25nIHRpbWUgYWdvIGFuZCBnb3QgYSBiZXR0ZXIgc3VwcG9ydGVkIE5JQy48YnI+PGJy PjxkaXYgY2xhc3M9InByb3Rvbm1haWxfc2lnbmF0dXJlX2Jsb2NrLXVzZXIiPjxkaXY+Sm9uYXRo YW4gVmFzcXVlejxicj48L2Rpdj48ZGl2PlBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBICBE NDlGIEZCODUgOTBCNyBDNENBIDUyNzk8YnI+PC9kaXY+PGRpdj5TZW50IHdpdGggUHJvdG9uTWFp bCBTZWN1cmUgRW1haWw8YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PC9kaXY+PGJyPjxicj5TZW50 IGZyb20gPGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Imh0dHBzOi8vcHJvdG9uLm1lL21haWwvaG9t ZSIgcmVsPSJub3JlZmVycmVyIj5Qcm90b24gTWFpbDwvYT4gZm9yIEFuZHJvaWQuPGRpdiBjbGFz cz0icHJvdG9ubWFpbF9xdW90ZSI+PGJyPjxicj4tLS0tLS0tLSBPcmlnaW5hbCBNZXNzYWdlIC0t LS0tLS0tPGJyPk9uIE1vbmRheSwgMDkvMjIvMjUgYXQgMDI6NDggVG9tZWsgQ0VEUk8gJmx0O3Rv bWVrQGNlZHJvLmluZm8mZ3Q7IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0icHJvdG9ubWFp bF9xdW90ZSI+PGRpdiBkaXI9ImF1dG8iPjxkaXY+d2hhdCBpIG5vdGljZWQgdGhhdCB0aGUgbmV3 IHBjIGhhdmUgdmVyeSBjb21wbGV4IGJpb3Mgc2V0dXAgb3B0aW9ucyB0aGF0IG1heSBpbXBhY3Qg c3RhYmlsaXR5IG9mIGZyZWVic2QuIGkgZW5jb3VudGVyZWQgdGhpcyByZWNlbnRseSBhZnRlciBo dyB1cGdyYWRlLiBwY2ktZSBnZW4gNCBvciA1LCB4MTYgb3IgeDggbW9kZSwgYW5kIHNwbGl0IGNv bnRyb2xsZXIgYmV0d2VlbiBzaW5nbGUgeDE2IGZvciBncHUgb3IgeDggZm9yIGdwdSBhbmQgeDgv eDQgYmV0d2VlbiBvdGhlciBwZXJpcGhlcmFscyBzdWNoIGFzIG52bWUgY29udHJvbGxlciB1c2Ig YWRkaXRpb25hbCBjYXJkIHNsb3RzIGV0Yy4gaW4gYWRkaXRpb24gdG8gdGhhdCB0aGVyZSBhcmUg cG93ZXIgbW9kZXMgYW5kIHJ1bnRpbWUgZnJlcXVlbmN5IGNoYW5nZXMgZm9yIGNwdSBhbmQgYWxs IHBlcmlwaGVyYWxzLiBub3QgYWxsIG5ldyBmZWF0dXJlcyBhcmUgc3VwcG9ydGVkIGJ5IGZyZWVi c2QuIGl0IHRvb2sgbWUgc29tZSB3aGlsZSB0byBhZGp1c3Qgb3B0aW1hbCBzZXR0aW5ncyB0byBn ZXQgbWFjaGluZSBzdGFibGUgKGF0IHRoZSBjb3N0IG9mIHBvd2VyIHNhdmluZ3MpLiBhbHNvIHRo ZXJlIHdlcmUgaXJyaXRhdGluZyBnbGl0Y2hlcyBpbiBhdWRpbyB0aGF0IGFyZSBub3cgY29uc2lk ZXJhYmx5IGxlc3Mgbm90aWNhYmxlLjwvZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRp diBkaXI9ImF1dG8iPmxvbmcgc3Rvcnkgc2hvcnQgaXQgaXMgY3B1LCByYW0sIGFuZCBwZXJpcGhl cmFscyBzZXR0aW5ncyB0aGF0IG1heSBpbXBhY3Qgb3ZlcmFsbCBzdGFiaWxpdHkgb2YgdGhlIGZy ZWVic2QgYm94LjwvZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8i PnRoaXMgd29yayBzZWVtcyB0byBiZSBub3Qgb25seSBpbXBvcnRhbnQgZm9yIHZpcnR1YWxpemF0 aW9uIGJ1dCBhbHNvIEZyZWVCU0QncyBMRFdHIHByb2plY3QgKExhcHRvcCBEZXNrdG9wIFdvcmsg R3JvdXApIGkgYW0gYXR0YWNoaW5nIGZyZWVic2QtZGVza3RvcEAgbWFpbGluZyBsaXN0IDotKTwv ZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj4tLTxicj5D ZURlUk9NLCBTUTdNSFosIDxhIGhyZWY9Imh0dHA6Ly93d3cudG9tZWsuY2Vkcm8uaW5mby8iIHJl bD0ibm9yZWZlcnJlciI+aHR0cDovL3d3dy50b21lay5jZWRyby5pbmZvPC9hPjwvZGl2PjwvZGl2 Pjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSBnbWFpbF9xdW90ZV9jb250YWluZXIiPjxkaXYg ZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9hdHRyIj5PbiBNb24sIFNlcCAyMiwgMjAyNSwgMDI6MjAg Sm9uYXRoYW4gVmFzcXVleiAmbHQ7PGEgaHJlZj0ibWFpbHRvOmpvbkB4eWlubi5vcmciIHJlbD0i bm9yZWZlcnJlciI+am9uQHh5aW5uLm9yZzwvYT4mZ3Q7IHdyb3RlOjxicj48L2Rpdj48YmxvY2tx dW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46IDAgMCAwIC44ZXg7IGJvcmRl ci1sZWZ0OiAxcHggc29saWQgI2NjYzsgcGFkZGluZy1sZWZ0OiAxZXgiPjxkaXYgc3R5bGU9ImZv bnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij5IZXkgYWxsLDwv ZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXpl OiAxNHB4Ij48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNl cmlmOyBmb250LXNpemU6IDE0cHgiPkp1c3Qgd2FudGVkIHRvIHJlcG9ydCB0aGF0IEkgc3RpbGwg aGF2ZW4ndCByZWNlaXZlZCBvbmUgY3Jhc2ggYWZ0ZXIgbXkgYWRqdXN0bWVudHMgeWVzdGVyZGF5 LiBTbyBmYXIgb3ZlciAyMCsgaG91cnMgb2YgdXB0aW1lfi4gSSd2ZSBhbHNvIHJlYm9vdGVkIHRo ZSBWTS9ob3N0IGEgZmV3IHRpbWVzIGp1c3QgdG8gZW5zdXJlIHRoYXQgaXQgaXMgZGV0ZXJtaW5p c3RpY2FsbHkgc3RhYmxlIGFuZCBzbyBmYXIgemVybyBpc3N1ZXMuIFNvIHRoaXMgaXMgcG9pbnRp bmcgaW4gYSBwb3NpdGl2ZSBkaXJlY3Rpb24uIEknbSB0ZW1wdGVkIHRvIG1hcmsgdGhpcyBhcyBh IHN0YWJsZSBzZXQgdXAgYnV0IEknbGwgbmVlZCB0byBwbGF5IGZvciBsb25nZXIgdG8gbWFrZSB0 aGF0IGNhbGwuPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlm OyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rpdj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBB cmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij4NCiAgICA8ZGl2Pg0KICAgICAgICA8 ZGl2PkpvbmF0aGFuIFZhc3F1ZXo8YnI+PC9kaXY+PGRpdj5QR1A6IDM0REEgODU4QyAxNDQ3IDUw OUUgQzc3QSAgRDQ5RiBGQjg1IDkwQjcgQzRDQSA1Mjc5PGJyPjwvZGl2PjxkaXY+U2VudCB3aXRo IFByb3Rvbk1haWwgU2VjdXJlIEVtYWlsPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2Pg0KICAgIDwv ZGl2Pg0KDQogICAgICAgICAgICA8ZGl2Pg0KDQogICAgICAgICAgICA8L2Rpdj4NCjwvZGl2Pg0K PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0 cHgiPjxicj48L2Rpdj48ZGl2Pg0KICAgICAgICBPbiBTdW5kYXksIFNlcHRlbWJlciAyMXN0LCAy MDI1IGF0IDA4OjIzLCBKb25hdGhhbiBWYXNxdWV6ICZsdDs8YSBocmVmPSJtYWlsdG86am9uQHh5 aW5uLm9yZyIgdGFyZ2V0PSJfYmxhbmsiIHJlbD0ibm9yZWZlcnJlciI+am9uQHh5aW5uLm9yZzwv YT4mZ3Q7IHdyb3RlOjxicj4NCiAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+DQogICAg ICAgICAgICBTeXN0ZW0gaGFzIGJlZW4gdXAgZm9yIGFib3V0IDloNDBtLiBNb3N0IG9mIGl0IGlk bGUgYW5kIG5vIGNyYXNoZXMgeWV0LiBJJ20gZ29ubmEgcmVib290IHRoZSBob3N0IGFuZCBzZWUg aWYgdGhlIHN0YWJpbGl0eSBzdGlsbCBjb250aW51ZXMuPGJyPjxicj48ZGl2PjxkaXY+Sm9uYXRo YW4gVmFzcXVlejxicj48L2Rpdj48ZGl2PlBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBICBE NDlGIEZCODUgOTBCNyBDNENBIDUyNzk8YnI+PC9kaXY+PGRpdj5TZW50IHdpdGggUHJvdG9uTWFp bCBTZWN1cmUgRW1haWw8YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PC9kaXY+PGJyPjxicj5TZW50 IGZyb20gPGEgaHJlZj0iaHR0cHM6Ly9wcm90b24ubWUvbWFpbC9ob21lIiByZWw9Im5vcmVmZXJy ZXIiIHRhcmdldD0iX2JsYW5rIj5Qcm90b24gTWFpbDwvYT4gZm9yIEFuZHJvaWQuPGRpdj48YnI+ PGJyPi0tLS0tLS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0tLS08YnI+T24gU2F0dXJkYXksIDA5 LzIwLzI1IGF0IDIzOjQ1IEpvbmF0aGFuIFZhc3F1ZXogJmx0OzxhIGhyZWY9Im1haWx0bzpqb25A eHlpbm4ub3JnIiB0YXJnZXQ9Il9ibGFuayIgcmVsPSJub3JlZmVycmVyIj5qb25AeHlpbm4ub3Jn PC9hPiZndDsgd3JvdGU6PGJyPjxibG9ja3F1b3RlPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBB cmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij5KdXN0IGZpbmlzaGVkIGJlYXRpbmcg dGhlIG1haW4gc3RvcnkgbGluZSBpbiBDeWJlcnB1bmsgMjA3NyBpbiB0aGUgVk0uIFNvIGZhciBu byBjcmFzaGVzIGFuZCBpdCdzIGJlZW4gcnVubmluZyBmb3IgYW4gaG91ci4gU28gdGhpcyBpcyBh IHBvc2l0aXZlIHNpZ25hbC4uLiB3ZSdsbCBzZWUgd2hhdCBoYXBwZW5zLiBJJ2xsIGxlYXZlIHRo ZSBWTSBydW5uaW5nIGlkbGUgb3ZlciBuaWdodCBhbmQgc2VlIGlmIGl0J3Mgc3RpbGwgcnVubmlu ZyBpbiB0aGUgbW9ybmluZy4gU3RheSBzYWZlIGFsbCE8L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZh bWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2Pg0KPGRp diBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgi Pg0KICAgIDxkaXY+DQogICAgICAgIDxkaXY+Sm9uYXRoYW4gVmFzcXVlejxicj48L2Rpdj48ZGl2 PlBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBICBENDlGIEZCODUgOTBCNyBDNENBIDUyNzk8 YnI+PC9kaXY+PGRpdj5TZW50IHdpdGggUHJvdG9uTWFpbCBTZWN1cmUgRW1haWw8YnI+PC9kaXY+ PGRpdj48YnI+PC9kaXY+DQogICAgPC9kaXY+DQoNCiAgICAgICAgICAgIDxkaXY+DQoNCiAgICAg ICAgICAgIDwvZGl2Pg0KPC9kaXY+DQo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNh bnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2PjxkaXY+DQogICAgICAgIE9uIFNh dHVyZGF5LCBTZXB0ZW1iZXIgMjB0aCwgMjAyNSBhdCAyMjoyOCwgSm9uYXRoYW4gVmFzcXVleiAm bHQ7PGEgaHJlZj0ibWFpbHRvOmpvbkB4eWlubi5vcmciIHRhcmdldD0iX2JsYW5rIiByZWw9Im5v cmVmZXJyZXIiPmpvbkB4eWlubi5vcmc8L2E+Jmd0OyB3cm90ZTo8YnI+DQogICAgICAgIDxibG9j a3F1b3RlIHR5cGU9ImNpdGUiPg0KICAgICAgICAgICAgPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6 IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPkkgZGlkIGEgYnVuY2ggb2YgbW9y ZSBleHBlcmltZW50cyB0b2RheSBhbmQgSSB3YXMgYWJsZSB0byBzd2l0Y2ggZnJvbSB1c2luZyB0 aGUgVVNCIGNvbnRyb2xsZXIgY29ubmVjdGVkIHRvIHRoZSBtdWx0aS1mdW5jdGlvbmFsIGJ1cyB0 aGF0IGhhcyB0aGUgaW50ZWdyYXRlZCBjYXJkLCBhbmQgc3dpdGNoZWQgdG8gYSBzZXBhcmF0ZSBV U0IgY29udHJvbGxlciB0aGF0IG9ubHkgaGFzIFVTQiBwb3J0cy4gSSdtIGhvcGluZyB0aGF0IHRo aXMgaW5jcmVhc2VzIFZNIHN0YWJpbGl0eSBhbmQgdGhhdCB0aGVyZSBtYXkgaGF2ZSBiZWVuIHNv bWUgd2lyZXMgYmVpbmcgY3Jvc3NlZC4gVGhpcyB3YXMgdGhlIHByZXZpb3VzIGJ1cyBJIHdhcyB1 c2luZyAoSSBjb3VsZG4ndCBpc29sYXRlIGp1c3QgdGhlIFVTQiBmdW5jdGlvbnMsIGl0IGhhZCB0 byBiZSBhbGwgb3Igbm90aGluZyBmb3IgaXQgdG8gZnVuY3Rpb24pOjwvZGl2PjxkaXYgc3R5bGU9 ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48YnI+PC9k aXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6 IDE0cHgiPjxzcGFuPnZnYXBjaTBAcGNpMDoxODowOjA6CWNsYXNzPTB4MDMwMDAwIHJldj0weGMx IGhkcj0weDAwIHZlbmRvcj0weDEwMDIgZGV2aWNlPTB4MTY0ZSBzdWJ2ZW5kb3I9MHgxMDQzIHN1 YmRldmljZT0weDg4Nzc8L3NwYW4+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IHZlbmRvciAmbmJz cDsgJm5ic3A7ID0gJ0FkdmFuY2VkIE1pY3JvIERldmljZXMsIEluYy4gW0FNRC9BVEldJzwvc3Bh bj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgZGV2aWNlICZuYnNwOyAmbmJzcDsgPSAn UmFwaGFlbCc8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IGNsYXNzICZuYnNw OyAmbmJzcDsgJm5ic3A7PSBkaXNwbGF5PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZu YnNwOyBzdWJjbGFzcyAmbmJzcDsgPSBWR0E8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj5oZGFjMEBw Y2kwOjE4OjA6MToJY2xhc3M9MHgwNDAzMDAgcmV2PTB4MDAgaGRyPTB4MDAgdmVuZG9yPTB4MTAw MiBkZXZpY2U9MHgxNjQwIHN1YnZlbmRvcj0weDEwNDMgc3ViZGV2aWNlPTB4ODg3Nzwvc3Bhbj48 L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgdmVuZG9yICZuYnNwOyAmbmJzcDsgPSAnQWR2 YW5jZWQgTWljcm8gRGV2aWNlcywgSW5jLiBbQU1EL0FUSV0nPC9zcGFuPjwvZGl2PjxkaXY+PHNw YW4+Jm5ic3A7ICZuYnNwOyBkZXZpY2UgJm5ic3A7ICZuYnNwOyA9ICdSZW1icmFuZHQgUmFkZW9u IEhpZ2ggRGVmaW5pdGlvbiBBdWRpbyBDb250cm9sbGVyJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFu PiZuYnNwOyAmbmJzcDsgY2xhc3MgJm5ic3A7ICZuYnNwOyAmbmJzcDs9IG11bHRpbWVkaWE8L3Nw YW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IHN1YmNsYXNzICZuYnNwOyA9IEhEQTwv c3Bhbj48L2Rpdj48ZGl2PjxzcGFuPm5vbmUxQHBjaTA6MTg6MDoyOgljbGFzcz0weDEwODAwMCBy ZXY9MHgwMCBoZHI9MHgwMCB2ZW5kb3I9MHgxMDIyIGRldmljZT0weDE2NDkgc3VidmVuZG9yPTB4 MTA0MyBzdWJkZXZpY2U9MHg4ODc3PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNw OyB2ZW5kb3IgJm5ic3A7ICZuYnNwOyA9ICdBZHZhbmNlZCBNaWNybyBEZXZpY2VzLCBJbmMuIFtB TURdJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgZGV2aWNlICZuYnNwOyAm bmJzcDsgPSAnRmFtaWx5IDE5aCBQU1AvQ0NQJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNw OyAmbmJzcDsgY2xhc3MgJm5ic3A7ICZuYnNwOyAmbmJzcDs9IGVuY3J5cHQvZGVjcnlwdDwvc3Bh bj48L2Rpdj48ZGl2PjxzcGFuPnhoY2kxQHBjaTA6MTg6MDozOgljbGFzcz0weDBjMDMzMCByZXY9 MHgwMCBoZHI9MHgwMCB2ZW5kb3I9MHgxMDIyIGRldmljZT0weDE1YjYgc3VidmVuZG9yPTB4MTA0 MyBzdWJkZXZpY2U9MHg4ODc3PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyB2 ZW5kb3IgJm5ic3A7ICZuYnNwOyA9ICdBZHZhbmNlZCBNaWNybyBEZXZpY2VzLCBJbmMuIFtBTURd Jzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgZGV2aWNlICZuYnNwOyAmbmJz cDsgPSAnUmFwaGFlbC9HcmFuaXRlIFJpZGdlIFVTQiAzLjEgeEhDSSc8L3NwYW4+PC9kaXY+PGRp dj48c3Bhbj4mbmJzcDsgJm5ic3A7IGNsYXNzICZuYnNwOyAmbmJzcDsgJm5ic3A7PSBzZXJpYWwg YnVzPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBzdWJjbGFzcyAmbmJzcDsg PSBVU0I8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj54aGNpMkBwY2kwOjE4OjA6NDoJY2xhc3M9MHgw YzAzMzAgcmV2PTB4MDAgaGRyPTB4MDAgdmVuZG9yPTB4MTAyMiBkZXZpY2U9MHgxNWI3IHN1YnZl bmRvcj0weDEwNDMgc3ViZGV2aWNlPTB4ODg3Nzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNw OyAmbmJzcDsgdmVuZG9yICZuYnNwOyAmbmJzcDsgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcywg SW5jLiBbQU1EXSc8L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IGRldmljZSAm bmJzcDsgJm5ic3A7ID0gJ1JhcGhhZWwvR3Jhbml0ZSBSaWRnZSBVU0IgMy4xIHhIQ0knPC9zcGFu PjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBjbGFzcyAmbmJzcDsgJm5ic3A7ICZuYnNw Oz0gc2VyaWFsIGJ1czwvc3Bhbj48L2Rpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IHN1YmNsYXNzICZu YnNwOyA9IFVTQjwvc3Bhbj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNh bnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFt aWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij5BbHRob3VnaCBub3cgSSdt IGxvbmdlciB1c2luZyB0aGlzIGF0IGFsbCBmb3IgVVNCIGZ1bmN0aW9uYWxpdHkgaW4gdGhlIFZN LiBJJ3ZlIHN3aXRjaGVkIHRvOjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwg c2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1m YW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxzcGFuPnBwdDJAcGNp MDoxMzowOjA6CWNsYXNzPTB4MGMwMzMwIHJldj0weDAxIGhkcj0weDAwIHZlbmRvcj0weDEwMjIg ZGV2aWNlPTB4NDNmNyBzdWJ2ZW5kb3I9MHgxYjIxIHN1YmRldmljZT0weDExNDI8L3NwYW4+PGRp dj48c3Bhbj4mbmJzcDsgJm5ic3A7IHZlbmRvciAmbmJzcDsgJm5ic3A7ID0gJ0FkdmFuY2VkIE1p Y3JvIERldmljZXMsIEluYy4gW0FNRF0nPC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4+Jm5ic3A7ICZu YnNwOyBkZXZpY2UgJm5ic3A7ICZuYnNwOyA9ICc2MDAgU2VyaWVzIENoaXBzZXQgVVNCIDMuMiBD b250cm9sbGVyJzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgY2xhc3MgJm5i c3A7ICZuYnNwOyAmbmJzcDs9IHNlcmlhbCBidXM8L3NwYW4+PC9kaXY+PHNwYW4+Jm5ic3A7ICZu YnNwOyBzdWJjbGFzcyAmbmJzcDsgPSBVU0I8L3NwYW4+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1m YW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rpdj48ZGl2 IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+ TXVjaCBjbGVhbmVyIGFuZCBzdGlsbCBjb250YWlucyBhIHZhcmlldHkgb2YgaGlnaCBzcGVlZCBw b3J0cy4gQWx0aG91Z2ggdWx0aW1hdGVseSBJJ20ganVzdCB1c2luZyBvbmUgb2YgdGhlbSBiZWNh dXNlIEkgaGF2ZSBhIEtWTSBjb25uZWN0ZWQgdG8gaXQgdGhhdCBhbGxvd3MgbWUgdG8gcGx1ZyBp biBrZXlib2FyZC9tb3VzZS8zLjVtbSBqYWNrIChhdWRpbykvZXRoZXJuZXQgYWxsIHRocm91Z2gg b25lIFVTQiB3aXJlLiBBbHRob3VnaCB0byByZW1pbmQgZXZlcnlvbmUsIEknbSBnYW1pbmcgaW4g YW4gb2ZmbGluZSBjYXBhY2l0eSB0byBJIGludGVudGlvbmFsbHkgbWVzcyB1cCB0aGUgSVB2NCBn YXRld2F5IHNvIG15IFZNIGNhbiBvbmx5IGNvbW11bmljYXRlIGludGVybmFsbHkgKGFuZCBJIGNh biBzaWRlbG9hZCBhcHBzL2dhbWVzKSBidXQgcHJldmVudCBNaWNyb3NvZnQgZnJvbSByZWFjaGlu ZyBteSBtYWNoaW5lIGFuZCBjb2xsZWN0aW5nIHRlbGVtZXRyeSBpbmZvLCBhbmQgZG9pbmcgZm9y Y2VkIHVwZGF0ZXMuIFBsdXMsIEkgaGF2ZSBubyBpbnRlbnRpb24gb2YgbW92aW5nIHRvIFdpbmRv d3MgMTEgd2l0aCBhbGwgb2YgdGhlaXIgbmV3IHJlcXVpcmVtZW50cyBhbmQgZm9yY2VkIG9ubGlu ZSBhY3RpdmF0aW9uIChJIGtub3cgb2YgdGhlIHdvcmthcm91bmRzLCBidXQgSSdtIGFnYWluc3Qg b24gbWFuZGF0aW5nIGEgTVNBIG9uIHByaW5jaXBsZSkuIFBlb3BsZSBzaG91bGQgYmUgYWJsZSB0 byBmdWxseSB1c2UgdGhlaXIgbWFjaGluZXMgZXZlbiBkdXJpbmcgaW5zdGFsbGF0aW9uL09PQkUg d2l0aG91dCBoYXZpbmcgdG8gYXV0aGVudGljYXRlIHRvIGEgY2VudHJhbCBzZXJ2ZXIuIEFueXdh eXMsIHNtYWxsIHJhbnQgb3ZlciBsb2wuPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFy aWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rpdj48ZGl2IHN0eWxlPSJm b250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweCI+SSdsbCB1cGRh dGUgZXZlcnlvbmUgb24gbXkgcHJvZ3Jlc3Mgd2l0aCB0aGlzIHJlLWNvbmZpZ3VyYXRpb24uPC9k aXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6 IDE0cHgiPjxicj48L2Rpdj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1z ZXJpZjsgZm9udC1zaXplOiAxNHB4Ij4NCiAgICA8ZGl2Pg0KICAgICAgICA8ZGl2PkpvbmF0aGFu IFZhc3F1ZXo8YnI+PC9kaXY+PGRpdj5QR1A6IDM0REEgODU4QyAxNDQ3IDUwOUUgQzc3QSAgRDQ5 RiBGQjg1IDkwQjcgQzRDQSA1Mjc5PGJyPjwvZGl2PjxkaXY+U2VudCB3aXRoIFByb3Rvbk1haWwg U2VjdXJlIEVtYWlsPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2Pg0KICAgIDwvZGl2Pg0KDQogICAg ICAgICAgICA8ZGl2Pg0KDQogICAgICAgICAgICA8L2Rpdj4NCjwvZGl2Pg0KPGRpdiBzdHlsZT0i Zm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHgiPjxicj48L2Rp dj48ZGl2Pg0KICAgICAgICBPbiBTYXR1cmRheSwgU2VwdGVtYmVyIDIwdGgsIDIwMjUgYXQgMjA6 MDIsIEpvbmF0aGFuIFZhc3F1ZXogJmx0OzxhIGhyZWY9Im1haWx0bzpqb25AeHlpbm4ub3JnIiB0 YXJnZXQ9Il9ibGFuayIgcmVsPSJub3JlZmVycmVyIj5qb25AeHlpbm4ub3JnPC9hPiZndDsgd3Jv dGU6PGJyPg0KICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4NCiAgICAgICAgICAgIDxk aXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4 Ij5IZXkgTnVubyw8L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2Vy aWY7IGZvbnQtc2l6ZTogMTRweCI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBB cmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48b2wgc3R5bGU9Im1hcmdpbi10b3A6 IDA7IG1hcmdpbi1ib3R0b206IDAiPjxsaSBzdHlsZT0ibGlzdC1zdHlsZS10eXBlOiAmcXVvdDsx LiAmcXVvdDsiPjxzcGFuPkknbSBub3QuIEkgcGx1Z2dlZCBpbiBteSBtb25pdG9yJ3MgSERNSSBj YWJsZSBkaXJlY3RseSBpbnRvIHRoZSBHUFUncyBIRE1JIHBvcnQuIE9uY2UgYmh5dmUgc3RhcnRz IHVwIChpZiB5b3VyIHBhc3N0aHJvdWdoIGlzIHdvcmtpbmcgY29ycmVjdGx5IGFuZCB5b3VyIEdQ VSdzIGRyaXZlcnMgYXJlIGluc3RhbGxlZCksIGl0IHdpbGwgb3V0cHV0IHRvIHlvdXIgbW9uaXRv ci4gWW91IHdvdWxkIG9ubHkgbmVlZCBWTkMgZm9yIGluaXRpYWwgaW5zdGFsbGF0aW9uIG9mIFdp bmRvd3MuPC9zcGFuPjwvbGk+PGxpIHN0eWxlPSJsaXN0LXN0eWxlLXR5cGU6ICZxdW90OzIuICZx dW90OyI+PHNwYW4+SSdtIG5vdCBzdXJlIG9mIHRoZSBOVklESUEgY2FzZS4gSSBkbyBrbm93IHRo ZXJlIGlzIHNvbWVvbmUgd2hvIGJsb2dnZWQgYWJvdXQgaG93IHRoZXkgZ290IHRoZWlycyB3b3Jr aW5nOiZuYnNwOzxzcGFuPjxhIHJlbD0ibm9yZWZlcnJlciIgaHJlZj0iaHR0cHM6Ly9kZmx1bmQu c2UvfmdldHovTm90ZXMvMjAyNC9mcmVlYnNkLWdwdS8iIHRhcmdldD0iX2JsYW5rIj5odHRwczov L2RmbHVuZC5zZS9+Z2V0ei9Ob3Rlcy8yMDI0L2ZyZWVic2QtZ3B1LzwvYT48L3NwYW4+PC9zcGFu PjwvbGk+PC9vbD48ZGl2Pjxicj48L2Rpdj48ZGl2PkF0bSBJJ20gY3VycmVudGx5IHRyeWluZyB0 byBjb2xsZWN0IG1vcmUgaW5mbyBvbiB3aHkgbXkgYmh5dmUgVk0gcmFuZG9tbHkgY3Jhc2hlcyBz b21ldGltZXMuIFNvIGZhciBJJ3ZlIGNvbGxlY3RlZCB0aGF0IHRoZSBleGl0IHJlYXNvbiBiaHl2 ZSBpcyBzaG93aW5nIGlzIDB4NjAsIHdoaWNoIHBvaW50cyB0byB0aGUgZm9sbG93aW5nIGVudHJp ZXMgaW4gdGhlIGFtZCBmcmVlYnNkIGtlcm5lbCBzb3VyY2VzOjwvZGl2PjxkaXY+PGJyPjwvZGl2 PjxkaXY+PHNwYW4+MHg2MDwvc3Bhbj48ZGl2PjxzcGFuPjxhIHJlbD0ibm9yZWZlcnJlciIgaHJl Zj0iaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMvYmxvYi80YzNhODY4ZDEz YzA1M2VmMTczMjY4Y2RmZTEzNjU5NzhhMjgyMTc4L3N5cy9hbWQ2NC92bW0vYW1kL3ZtY2IuaCNM MTk1IiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1z cmMvYmxvYi80YzNhODY4ZDEzYzA1M2VmMTczMjY4Y2RmZTEzNjU5NzhhMjgyMTc4L3N5cy9hbWQ2 NC92bW0vYW1kL3ZtY2IuaCNMMTk1PC9hPjwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuPjxhIHJlbD0i bm9yZWZlcnJlciIgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMv YmxvYi80YzNhODY4ZDEzYzA1M2VmMTczMjY4Y2RmZTEzNjU5NzhhMjgyMTc4L3N5cy9hbWQ2NC92 bW0vYW1kL3ZtY2IuaCNMMTI4QzEtTDEyOEMzMCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vZ2l0 aHViLmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL2Jsb2IvNGMzYTg2OGQxM2MwNTNlZjE3MzI2OGNk ZmUxMzY1OTc4YTI4MjE3OC9zeXMvYW1kNjQvdm1tL2FtZC92bWNiLmgjTDEyOEMxLUwxMjhDMzA8 L2E+PC9zcGFuPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PHNwYW4+WFNFVEJWPC9zcGFuPjwv ZGl2PjxkaXY+PHNwYW4+PGEgcmVsPSJub3JlZmVycmVyIiBocmVmPSJodHRwczovL2dpdGh1Yi5j b20vZnJlZWJzZC9mcmVlYnNkLXNyYy9ibG9iLzRjM2E4NjhkMTNjMDUzZWYxNzMyNjhjZGZlMTM2 NTk3OGEyODIxNzgvc3lzL2FtZDY0L3ZtbS9hbWQvdm1jYi5oI0w5NCIgdGFyZ2V0PSJfYmxhbmsi Pmh0dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL2Jsb2IvNGMzYTg2OGQxM2Mw NTNlZjE3MzI2OGNkZmUxMzY1OTc4YTI4MjE3OC9zeXMvYW1kNjQvdm1tL2FtZC92bWNiLmgjTDk0 PC9hPjwvc3Bhbj48L2Rpdj48c3Bhbj48L3NwYW4+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5J IGFsc28gYXNrZWQgUHJvdG9uJ3MgTHVtbysgQUkgd2hhdCBpcyAweDYwIHRvIHNlZSB3aGF0IGl0 IGNvbWVzIHVwIHdpdGgsIGl0IHNhaWQgdGhlIGZvbGxvd2luZzo8L2Rpdj48ZGl2Pjxicj48L2Rp dj48ZGl2PjxzcGFuPmV4aXRfcmVhc29uWzBdID0gMHg2MCBpcyB0aGUgVk3igJFleGl0IGNvZGUg dGhhdCBiaHl2ZSByZXBvcnRzIHdoZW4gdGhlIGd1ZXN0IHRyaWdnZXJzIHRoZSBYU0VUQlYgaW5z dHJ1Y3Rpb24uPC9zcGFuPjxkaXY+PHNwYW4+V2h5IGl0IGhhcHBlbnM8L3NwYW4+PC9kaXY+PGRp dj48YnI+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7IFhTRVRCViB3cml0ZXMgdG8gdGhl IGV4dGVuZGVkIGNvbnRyb2wgcmVnaXN0ZXIgWENSMC48L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4m bmJzcDsgJm5ic3A7IEluIEludGVs4oCvVk1YICh0aGUgdmlydHVhbGl6YXRpb24gdGVjaG5vbG9n eSBiaHl2ZSB1c2VzIG9uIHg4NiksIGV4ZWN1dGlvbiBvZiBYU0VUQlYgaXMgbm90IGFsbG93ZWQg dG8gcnVuIGRpcmVjdGx5IGluIHRoZSBndWVzdDsgdGhlIENQVSBmb3JjZXMgYSBWTeKAkWV4aXQg c28gdGhlIGh5cGVydmlzb3IgY2FuIHZhbGlkYXRlIHRoZSBvcGVyYXRpb24uPC9zcGFuPjwvZGl2 PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBiaHl2ZSByZWNvcmRzIHRoZSBleGl0IHJlYXNvbiBh cyAweDYwLCB3aGljaCBtYXBzIHRvIHRoZSBWTVggZXhpdCByZWFzb24g4oCcWFNFVEJW4oCdIGlu IHRoZSBJbnRlbCBzcGVjaWZpY2F0aW9uLjwvc3Bhbj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2 PjxzcGFuPldoYXQgaXQgbWVhbnMgZm9yIHlvdTwvc3Bhbj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48 ZGl2PjxzcGFuPiZuYnNwOyAmbmJzcDsgVGhlIGd1ZXN0IHRyaWVkIHRvIGVuYWJsZSBvciBjaGFu Z2UgQVZY4oCRNTEyIC8gWFNBVkUgZmVhdHVyZSBiaXRzIHZpYSBYU0VUQlYuPC9zcGFuPjwvZGl2 PjxkaXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBCZWNhdXNlIGJoeXZlIGRvZXMgbm90IGN1cnJlbnRs eSBlbXVsYXRlIHRoYXQgaW5zdHJ1Y3Rpb24sIGl0IGFib3J0cyB0aGUgZ3Vlc3TigJlzIGluc3Ry dWN0aW9uIHN0cmVhbSBhbmQgcmVwb3J0cyB0aGUgZXhpdCBjb2RlLjwvc3Bhbj48L2Rpdj48ZGl2 PjxzcGFuPiZuYnNwOyAmbmJzcDsgSW4gbW9zdCBjYXNlcyB0aGUgZ3Vlc3Qgd2lsbCBjb250aW51 ZSBhZnRlciBiaHl2ZSBoYW5kbGVzIHRoZSBleGl0LCBidXQgaWYgdGhlIGh5cGVydmlzb3IgaXNu 4oCZdCBzZXQgdXAgdG8gZW11bGF0ZSBYU0VUQlYgeW914oCZbGwgc2VlIHRoZSBWTSBzdG9wIG9y IGNyYXNoLjwvc3Bhbj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxzcGFuPlR5cGljYWwgd2F5 cyB0byBkZWFsIHdpdGggaXQ8L3NwYW4+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48c3Bhbj4m bmJzcDsgJm5ic3A7IFVwZGF0ZSBiaHl2ZSDigJMgbmV3ZXIgcmVsZWFzZXMgaW5jbHVkZSBiYXNp YyBYU0VUQlYgaGFuZGxpbmcgZm9yIGNvbW1vbiBmZWF0dXJlIHNldHMuPC9zcGFuPjwvZGl2Pjxk aXY+PHNwYW4+Jm5ic3A7ICZuYnNwOyBBZGp1c3QgdGhlIGd1ZXN04oCZcyBDUFUgZmxhZ3Mg4oCT IHN0YXJ0IHRoZSBWTSB3aXRoIGEgQ1BVIG1vZGVsIHRoYXQgZG9lc27igJl0IGV4cG9zZSB0aGUg WFNBVkUvWEdFVEJWIGZlYXR1cmVzIHRoZSBndWVzdCBleHBlY3RzIChlLmcuLCAtYyBob3N0IHZz LiBhIHNwZWNpZmljIG1vZGVsKS48L3NwYW4+PC9kaXY+PGRpdj48c3Bhbj4mbmJzcDsgJm5ic3A7 IFBhdGNoIGJoeXZlIOKAkyBpZiB5b3XigJlyZSBjb21mb3J0YWJsZSB3aXRoIEMsIHlvdSBjYW4g YWRkIGEgaGFuZGxlciBmb3IgVk1YX0VYSVRfUkVBU09OX1hTRVRCViB0aGF0IGVpdGhlciBlbXVs YXRlcyB0aGUgaW5zdHJ1Y3Rpb24gb3Igc2ltcGx5IGlnbm9yZXMgaXQgd2hlbiB0aGUgcmVxdWVz dGVkIGJpdHMgYXJlIGFscmVhZHkgZW5hYmxlZC48L3NwYW4+PC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj48c3Bhbj5JZiB5b3Uga2VlcCBzZWVpbmcgdGhlIHNhbWUgZXhpdCByZXBlYXRlZGx5LCBp dCB1c3VhbGx5IHBvaW50cyB0byBhIG1pc21hdGNoIGJldHdlZW4gdGhlIGd1ZXN04oCZcyBleHBl Y3RhdGlvbnMgKG9mdGVuIGEgbW9kZXJuIExpbnV4IGtlcm5lbCBvciBhIFdpbmRvd3MgYnVpbGQg dGhhdCB3YW50cyBBVljigJE1MTIpIGFuZCB0aGUgY2FwYWJpbGl0aWVzIGJoeXZlIGlzIHByZXNl bnRpbmcuIEFsaWduaW5nIHRob3NlIGV4cGVjdGF0aW9uc+KAlGVpdGhlciBieSB0d2Vha2luZyB0 aGUgVk3igJlzIENQVSBjb25maWd1cmF0aW9uIG9yIGJ5IHVwZ3JhZGluZyBiaHl2ZeKAlGdlbmVy YWxseSByZXNvbHZlcyB0aGUgaXNzdWUuPC9zcGFuPjwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxicj48 L2Rpdj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZv bnQtc2l6ZTogMTRweCI+DQogICAgPGRpdj4NCiAgICAgICAgPGRpdj5Kb25hdGhhbiBWYXNxdWV6 PGJyPjwvZGl2PjxkaXY+UEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgIEQ0OUYgRkI4NSA5 MEI3IEM0Q0EgNTI3OTxicj48L2Rpdj48ZGl2PlNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBF bWFpbDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj4NCiAgICA8L2Rpdj4NCg0KICAgICAgICAgICAg PGRpdj4NCg0KICAgICAgICAgICAgPC9kaXY+DQo8L2Rpdj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFt aWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4Ij48YnI+PC9kaXY+PGRpdj4N CiAgICAgICAgT24gU2F0dXJkYXksIFNlcHRlbWJlciAyMHRoLCAyMDI1IGF0IDE5OjQ3LCBOdW5v IFRlaXhlaXJhICZsdDs8YSBocmVmPSJtYWlsdG86ZWR1YXJkb0BmcmVlYnNkLm9yZyIgdGFyZ2V0 PSJfYmxhbmsiIHJlbD0ibm9yZWZlcnJlciI+ZWR1YXJkb0BmcmVlYnNkLm9yZzwvYT4mZ3Q7IHdy b3RlOjxicj4NCiAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+DQogICAgICAgICAgICA8 ZGl2IGRpcj0ibHRyIj48ZGl2PkhlbGxvIEpvbmF0aGFuITwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxk aXY+VGhhbmtzIGZvciBleGNlbGxlbnQgcXVhbGl0eSB2aWRlb3Mgb24geXQsIEkgYWxyZWFkeSBz dWJzY3JpYmVkIGl0IDopPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5KdXN0IGEgcXVpY2sgcXVl c3Rpb25zOjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+MS0gQXJlIHlvdSB1c2luZyBmcmVlcmRw MyB0byBjb25uZWN0IHRvIHdpbjExcHJvPyAgKEFza2luZyB0aGlzIGJlY2F1c2Ugdm5jIGlzIHZl cnkgc2xvdyk8L2Rpdj48ZGl2PjItIERvIHlvdSBrbm93IGhvdyBpcyBudmlkaWEgc3RhdHVzIGNv bXBhcmVkIHRvIHlvdXIgdXNlIGNhc2U/PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGFua3Mh PC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IGRpcj0ibHRyIiBj bGFzcz0iZ21haWxfYXR0ciI+Sm9uYXRoYW4gVmFzcXVleiAmbHQ7PGEgaHJlZj0ibWFpbHRvOmpv bkB4eWlubi5vcmciIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmpvbkB4eWlubi5v cmc8L2E+Jmd0OyBlc2NyZXZldSAoc8OhYmFkbywgMjAvMDkvMjAyNSDDoChzKSAwMjo0Mik6PGJy PjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjogMCAw IDAgLjhleDsgYm9yZGVyLWxlZnQ6IDFweCBzb2xpZCAjY2NjOyBwYWRkaW5nLWxlZnQ6IDFleCI+ SGV5IGFsbCwgSSd2ZSBzcGVudCBzb21lIHRpbWUgdG9kYXkgZG9pbmcgYSBnYW1pbmcgZGVtby4g SSd2ZSB1cGxvYWRlZCB0aGUgdmlkZW8gaW4gMksgc28geW91IGNhbiBnZXQgdGhlIG1heGltdW0g ZWZmZWN0LiBJdCdzIGJlZW4gYW4gaW50ZXJlc3RpbmcgZXhwZXJpZW5jZSBzbyBmYXIsIGFuZCBJ J20gZ29pbmcgdG8gdHJ5IHRvIG1ha2UgaXQgbXkgcHJpbWFyeSBnYW1pbmcgbWFjaGluZS4gV2Un bGwgc2VlIGhvdyBpdCBnb2VzLiBJdCdzIGRlZmluaXRlbHkgcXdpcmt5LCBhbmQgc29tZXRpbWVz IEkgbmVlZCB0byByZWJvb3QgdGhlIFZNIDQtNSB0aW1lcyBiZWZvcmUgaXQgInN0YWJpbGl6ZXMi ICh3aGljaCBtZWFucyB0aGUgVk0gd29uJ3QganVzdCByYW5kb21seSBjcmFzaCBvciBoYXZlIG1h c3NpdmUgbGFnIGluIGdhbWUpLiBJJ2xsIG5lZWQgdG8gY29sbGVjdCBtb3JlIGRhdGEgdGhyb3Vn aCBwbGF5IHRlc3RpbmcuPGJyPg0KPGJyPg0KWW91IGNhbiB0YWtlIGEgbG9vayBhdCB0aGUgdmlk ZW8gaGVyZSAoSSdsbCBwb3N0IHRpbWVzdGFtcHMgaW4gdGhlIHZpZGVvIHNvb24pOjxicj4NCjxi cj4NCjxhIGhyZWY9Imh0dHBzOi8veW91dHUuYmUvX2N6MFJVQXc1cDgiIHJlbD0ibm9yZWZlcnJl ciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8veW91dHUuYmUvX2N6MFJVQXc1cDg8L2E+PGJyPg0K PGJyPg0KSm9uYXRoYW4gVmFzcXVlejxicj4NClBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdB IEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3OTxicj4NClNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3Vy ZSBFbWFpbDxicj4NCjxicj4NCjxicj4NCjxicj4NCk9uIFdlZG5lc2RheSwgU2VwdGVtYmVyIDE3 dGgsIDIwMjUgYXQgMjI6MzMsIEpvbmF0aGFuIFZhc3F1ZXogJmx0OzxhIGhyZWY9Im1haWx0bzpq b25AeHlpbm4ub3JnIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5qb25AeHlpbm4u b3JnPC9hPiZndDsgd3JvdGU6PGJyPg0KPGJyPg0KJmd0OyBBbHJpZ2h0ISBUaGUgWW91VHViZSBW aWRlbyBpcyB1cC4gTGV0IG1lIGtub3cgd2hhdCB5b3UgdGhpbmsuIEknbSByZWFsbHkgZXhjaXRl ZCBhYm91dCB0aGlzLjxicj4NCiZndDsgPGJyPg0KJmd0OyA8YSBocmVmPSJodHRwczovL3lvdXR1 LmJlL09iNC12N2RUSkdzIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwczov L3lvdXR1LmJlL09iNC12N2RUSkdzPC9hPjxicj4NCjxicj4NCjwvYmxvY2txdW90ZT48L2Rpdj48 ZGl2PjxiciBjbGVhcj0iYWxsIj48L2Rpdj48YnI+PHNwYW4gY2xhc3M9ImdtYWlsX3NpZ25hdHVy ZV9wcmVmaXgiPi0tIDwvc3Bhbj48YnI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25h dHVyZSI+PGRpdiBkaXI9Imx0ciI+PGRpdj48Zm9udCBjb2xvcj0iIzg4ODg4OCI+TnVubyBUZWl4 ZWlyYTwvZm9udD48L2Rpdj48ZGl2PjxkaXY+PGZvbnQgY29sb3I9IiM4ODg4ODgiPg0KRnJlZUJT RCBVTklYOiAgJmx0O2VkdWFyZG9ARnJlZUJTRC5vcmcmZ3Q7ICAgV2ViOiAgPGEgaHJlZj0iaHR0 cHM6Ly9mcmVlYnNkLm9yZy8iIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBz Oi8vRnJlZUJTRC5vcmc8L2E+PGJyPjwvZm9udD48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj4NCg0K ICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwvZGl2Pg0KICAgICAgICA8L2Jsb2NrcXVv dGU+PGJyPg0KICAgIDwvZGl2Pg0KICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwvZGl2 PjwvYmxvY2txdW90ZT48L2Rpdj4NCg0KICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwv ZGl2PjwvYmxvY2txdW90ZT48L2Rpdj4NCjwvYmxvY2txdW90ZT48L2Rpdj4NCiAgICAgICAgICAg ICAgICAgICAgPC9ib2R5PjwvaHRtbD4= --b1=_BARyjgvDZAqqPXTTnklG1rA9YLWIPNYCRXmUllUuo-- From nobody Mon Sep 22 11:25:44 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVglh2KQMz67lXn for ; Mon, 22 Sep 2025 11:25:52 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVglh1m7Dz4NLj; Mon, 22 Sep 2025 11:25:52 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758540352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=LOvGC33F2bxTjYvQTrxaeOaKuMiaZX9+4eAvVjzu+sg=; b=k8e7K4wQzRXu4bQKMzwy1wrTr7zvPIzR67NMmbtYTARt+nejVLRjkFNe6fIEEnbGHxujZH kNEuyuoSZ/znT0bVKsVxDZJlRmrk5FGKKITlwEcM79tWb3b+G6eEWMj3h+jbwFgy34URYb j5JcTYkchgNuf/Rnvg9sLDbtVD+QdzU7YKxTTxDtKwvl7XcXn/WoyLg5gdJKOq3vl9shs4 kiZWfcknU39KM21n4hwXM5U133Mz1jKrtx2S0J4xjD8Kx28pdXGDGtsQ+dpdZuwiz/GYc6 o6S4YsF/nZWbSdlcMSeVr55UIqweLAr44l5GG7L+/VIfA5tdTRX1oIiEPzrI7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758540352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=LOvGC33F2bxTjYvQTrxaeOaKuMiaZX9+4eAvVjzu+sg=; b=Xj6655Ugcbd2grQJc8G8JalJnU8DCPOgwX7KvHuvpj8/8p4PEwAPz+gf1xveh8T6UM1aaV wC2M+dVx/wjBnZLVZnHTPt54rhKdWUghSXQcVPpFeWfDcc+fiWKORshGD92UU1SwVBS1e+ NU0BriROMqEaiUQOQmfM9edGNd4Do0Y+IRP3QhSEoWwMq6R3R4hOVnJuaaO6M/lqGBbTR7 EZPVUc9SB1QN+PaeyEbGa5QtTKq5/FgivpKwm+hEsRV5p6ed5YzbQqyH2oICJ9Gr7caJZg L06U5E80qCt4vfYdeeLdRJRl0NzJSM0IYe9EHqHXd1zBfF/DDKhNtuaH3WomGw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758540352; a=rsa-sha256; cv=none; b=NJfv50DXiGdmYXHJEINvEE0JB1gXzSvxen8ATikF7NmXZO2H245eiitVCCS3Brjh0y2RxW TURSGUPrK8tdbJBEdCZiWVYY6ziSdLjl7YxlX3Vw2lwO6jUcm/xaJty3KWXIGEZi+JkRkc 2G7YB4NpIK+y5uIykIoNnMSTqI3U9qKLOOgNB5EK9B+ENCQchKrFG+JhTYkEn+DTOgkaou kccmeUxokfaXh1wgPO61oyEHcNrpl6Nggy5+ioDmlCHstxhV6evD3EcEJBHjc/yGdWoKlW Ltsyc4fXYdmrSfEmmhw/kvIKDqunK5E4tGE6swXXax+wDxBaxX4hNT3izvF4JA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from corvink-nb.fritz.box (unknown [IPv6:2001:9e8:da59:c00:f0e:1405:9a09:7f9e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4cVglg4Mshzpkl; Mon, 22 Sep 2025 11:25:51 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Nuno Teixeira , Jonathan Vasquez Cc: "virtualization@freebsd.org" , Tomek CEDRO Date: Mon, 22 Sep 2025 13:25:44 +0200 In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <6QpHOGTzqnqxLPCDMcDbChlobQw-mVGWb2BjQk-K7kyE3OjuLCCFLGn5nVpgEvlr3xaipUzLMWz1JaESXx9W6w7aXNyAXN5wbuoS8eaWkcU=@xyinn.org> <7YJWddCC_SUuB_mwDmHL3xecft8_rMou1xosTzBIK1UP_Fw-B786LWZX6CQ8XG6smJQRlxbfJbCq8fmTI15RouBn8GN73IfJvPOg7k6jr-s=@xyinn.org> Autocrypt: addr=corvink@FreeBSD.org; prefer-encrypt=mutual; keydata=mQINBGNjZaIBEADDTrDNf+0pwiuRPBdClcnZW83dH1UhuOi0u+A1J2SatEBbNaFVtXXAa vewCTuyV/ZbNidjlhq3R/pWyiKjFKvs5dj7PMCw+3z2D5OWpMdHg7TrB+fbdFPOEsu0zQVKNaO+pS KCfN0Re0m7bL3wuvl7PXvBufRwA3Guo1P4j3TXWaEkuso7VupTvE25zVGg9ONHrGOjA9RUy+Yg4Se 3NLgtUdjBgA21SBQTDvRQV4fDmVenlwvWeE0Xm8FcDcpQb6sJTihaDku78mi3Ux1HCk7rTcepVEB0 xIB6qmFxv0sLlDmVv6Z6qg1y/Q5m23Pgz60o3TulMPV4F+3Itm8ifU+wgVSzBZbD29GYkd7LKqMkF bhvfSBk+5db3vbYY5OD//+LTM5AV7e2AhXuXMvG1UNBqXqSJTTSy6KZz+qmPQO0zos0dq46p8o82l KiBEGD2Hu0p+u0OyV+MmRYo1NIBFVbOPXp2MvUVl5II0UIJ3+N9gLBmfGA+HEpVO8PnvdoT/5NQ7m 8JK1rQHzjiDub/iDPAYMqKH4C0eZ/7zO0fuY5FeRNtuNtpH1Bw/+7/5RJH7bcKkfGHHEp15FJUrGH gWNydoDLB9QBprwQc8FEldDXBjzOMXIgh6FGKLNu6DswvIPGy6M3u7DXwDakCXz+c9Ym0oFihLzZx WntrsxdswD/CwARAQABtCdDb3J2aW4gS8O2aG5lIDxjb3J2aW4ua29laG5lQGdtYWlsLmNvbT6JAl QEEwEIAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCZArUIwIbAwUJCWYBgAULCQgHAgYVCgkICwI EFgIDAQIeAQIXgAAKCRDYVNpWMV4CalcID/44k2i/mqSSi4W6FAobSF1nFLtP/pfcRNJriWKx2UF7 cfFMKyg7Nilg7FhLb5FDB1umUW2nFfchFPTUp4FfKzgRvPzIMg4RIRcVtTpYbl3z7zs9ZXD8qS//i ibbiUG3quncm6tO2x1jLZD3ORC+8MuLGXhYQIa4O5vVF2SBHdb/U6P+wsrF+U+OpRdEdQ/4Xu9S02 kltzBGgArjcexdhUqEqW01KCCSH0+qgfN1NE+9L934ZOB+cai2b9apPbCOGuV6KcUKMj4z0RWInhl XIyMqtmhdix/P/GjrQ1REVNdp74JeweSSedM15wwc6YLMXPrtOnnExyZ1gyNFYaub+Mdo3ZQ+8386 3B0C9IlpzEW0K8bYlZrl4WBNdcMOyByETAxgQmBgP6ZFErTtaeUOH1nX2FyR6o2GLSahRgngQmnRE zScTpPbBEkMwJMDAO+rbjjGxjeKSUwD1WOfbI6QZj+MS/uBk2p08kgN7fQaCEwj6jqML/IIE+FZ9I In6TNG0hChD384VHO+YioLBno1Atgi4Q7JUWSRIHQXZW+StQajFrWPPyKQwCe1MwqpKoMpX/q8IZB lzwJgZS8ShLeFZjtzOt1jgM99TD91Neonf9OzjTSbfo7sJviSWoICMhB/MvDZjj+naMVF86uGFxnI EsjVKyBxlJd4TRHnLYPTIHFKBLQjQ29ydmluIEvDtmhuZSA8Y29ydmlua0BGcmVlQlNELm9yZz6JA lQEEwEKAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCY2NlogIbAwUJCWYBgAULCQgHAwUVCgkICw UWAwIBAAIeAQIXgAAKCRDYVNpWMV4CavfDEACCFnXpR7H9eOgP+GJMNPtK6i9/xnqdyXi8uCZIN0h YwjN4Xzo9SMLOf4UUlQEveOB+bGqbRfHd/fGKnrlXiPd0SGpKWJC21gqL/DsIH0J8I3Whth+O8tfP WeFy0oCsvBaaGFLIrDfoIgHF9i/gqEe48xhN42weB02Z3mdR1L0d7ME/BLwS0mCXe9Zh3uHw63S6x YB3Wsjptxe/ph6TpQDUKWtRJkjC6BqXPBdThpbbfIRWmjZbp2fKEJPvtRXS14+gbUqWeJ4xCvprA3 +ae7vtrp91X775yngyW3XTw5cmDiJIjykH8+zhEIoNQXNBpFrehkQDYrcM+WoE6NGSJo+3VJvSRWh UGWDVrxdTYNkbIjmTNlkI12NINC007DiuV7OF9XHWgrbbylvuZvbODmbJRdhTFy9upAUygX1/xUAQ EIMqMiJmyTdv8i0IbZ611WElQx9XHgGeZgM6+39/laN8FwspM6gE/4NzZHIZN0LEBOerZqoF+Il6e ccQpoEWx7nb/RilJp3dUUyvkBnJWg+AJByosg857kvvmDnZ3UB+bejWpcfFvnbkiKPUBPDO7tWPb5 r2yFDpDe/Vg5x+sRbkkXGUFD6Rx0p/ZiRIneVVg8emTzhDR0IL/BZkq/uVJkdaphZli1F/31cNgf6 ZJYnjxlk86uiYSySZQR0dKLDqq7QlQ29ydmluIEvDtmhuZSA8Yy5rb2VobmVAYmVja2hvZmYuY29t PokCVAQTAQgAPhYhBIL0UpWt5trfx9lPRthU2lYxXgJqBQJkCtScAhsDBQkJZgGABQsJCAcCBhUKC QgLAgQWAgMBAh4BAheAAAoJENhU2lYxXgJqI+QQAML5PTR7KpUFV3SLG60LQJGEOHUfDmJYczxBFb IAq1U4hIbivopu1AdLty7oDDrIjCVoa2/Cy34dd99O7lhLvUmZFB/zDSUtbUg2zhDkU0YSZ11Fdrl Wzky2tFaQRgxpDvWlUP0baa3Pd4dPDRiIUI6AOSR2SL6XANk6sJh56gLVM6G8yyafGsxSyDYg6Z78 EEMFejHwB+KP2DdsahupzM+F97HeC1+bOHYxtqN+2hEkPLtQWizyumPqNg5FvZhwe7yO8V95hF3Rh uDO+9aJT+WLLvcZEb/L1bI04IvZ5FWgCLI7Levd/DuOtZI8gWapHhqGZRbXB2fuJkCoKCl6V67h/7 aWhU3LjFTsC5siJyrxPjapKcIk8a7PqZDswNCKR+24LJ5D59mPgEOnsiCCVpik1WE/kgD+rOu9dQx jpjKwuKowf4EJP1KYNkYtoy1HthzyTOqPwqXC3IUl0GVPO3xw6MhUM3irCVFruC+ecVrv85Rd37vr duT6JvgW92xjWegMsamtkDZH8Ik/cmYhH0K/qEc6OZVNea/4PTEZe0uxODJ6pbMd80AJyGqDPPVeA gWJtEIG4k6IS8XyD5v1QJtlpDVpPwP/bbFnVc3h3Oatfn8Etm1KAqYvNwyO+om2PkF1p732uapDZd LwksVmgc9s79+9pSpeP5DbIeMzhrIK Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-QgmUzbD5SL7mZzKo6kcg" User-Agent: Evolution 3.56.2 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 --=-QgmUzbD5SL7mZzKo6kcg Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, 2025-09-21 at 00:47 +0100, Nuno Teixeira wrote: > 2- Do you know how is nvidia status compared to your use case? Hi Nuno, NVIDIA should to be in the same shape as AMD on CURRENT since [1]. I'm goin= g to MFC it soon to land it in 15.0. This means: some users are reporting a work= ing setup for Windows and Linux guests, however it's not tested very well yet. = So, it's an experimental setup which might fail sporadically, on heavy load or = long time testing. Note that QEMU does implement some additional quirks for NVIDIA cards [2]. = None of those are implemented in bhyve yet. However, most of them seem to be rel= ated to older cards. [1] https://reviews.freebsd.org/D51892 [2] https://elixir.bootlin.com/qemu/v10.1.0/source/hw/vfio/pci-quirks.c#L481-L4= 95 --=20 Kind regards, Corvin --=-QgmUzbD5SL7mZzKo6kcg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmjRMjgACgkQ2FTaVjFe AmoXfQ//RutZmKwtt8+C1O+yAbC4ZKXjZitkdjVHTywODyzhANTuLQ1HTdy7XnZ5 /oHiOMen8LkqfHekvUx8Y95RKtv+KGcLkSuAr0kvM6xTbXtg7LQmnNA56a/Z+f60 3BwVyrmqMNvVHFGA4Et4RmOlc8WK5NTQA0NnR8sTGlpa1mtHHLPTY2BjLQ3SZuS0 uQfQHLqJGJrEqu92QRwziSPC09ADk1Whd2GJBVBZ/exy3I1qKlaoYLqbNH56Atcl qADJXPVSCTKoQQnoeF3BpC1GNCAZ/qxi/0dTHzJWIvtLA52xAR965X9bTG3psG33 D6IZmnFJDEpspaP+tr+sXfcQZXpWMETVBwDkQv4WYi62lFzC2S1SPiA+pPFmgIfs yv6SvVAmrLY8YCVzSU14lQehycqwGb+r5lt7MBMOIKs7lVSa4spHds/I4i6Zpsp8 t2NkEkRqZfMUqz+U0oIWX+7Q+Y6N8/J2D3xnNSgE9LATnEyM+0BAUbtprbTRUIji JxUCfyjp++F78BMZYoQc8Umx/QkTxBb2y4zPqsHlBdJuIcTJgNhYI1geefA96y37 /ZTyOdmUWBKXysdCT5Eq2Gh9rAP2qEYqmS4sft/b7JFF50lbuMpDIZFCS8GxDjtP sH0ACZhfE+P4xFOM2034nN0I7VtM+O/ZIIr9MUybvJBwTHFQpAI= =3Uf2 -----END PGP SIGNATURE----- --=-QgmUzbD5SL7mZzKo6kcg-- From nobody Mon Sep 22 12:17:32 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVhvd0nJZz67pP8 for ; Mon, 22 Sep 2025 12:17:49 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVhvc0hw1z3Gh9 for ; Mon, 22 Sep 2025 12:17:47 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua Received: from [91.215.147.205] (helo=[192.168.88.227]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.98.2 (FreeBSD)) (envelope-from ) id 1v0fUC-00000000EN3-0twU for virtualization@freebsd.org; Mon, 22 Sep 2025 15:17:40 +0300 Message-ID: <5009e35f-e937-4106-b1ed-2ff6b0d790d5@shurik.kiev.ua> Date: Mon, 22 Sep 2025 15:17:32 +0300 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Bhyve vm image location To: virtualization@freebsd.org References: Content-Language: uk From: Oleksandr Kryvulia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ACL-Warn: SPF failed. 91.215.147.205 is not allowed to send mail from shurik.kiev.ua. X-Spamd-Bar: - X-Spamd-Result: default: False [-1.47 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.97)[-0.971]; NEURAL_SPAM_LONG(0.80)[0.797]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[shurik.kiev.ua]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4cVhvc0hw1z3Gh9 22.09.2025 1:11, David Pasek: > Hi, > I'm studying bhyve virtualization and have a question. > > I've created my first image from running VM with the command ... > vm image create -d fbsd-test fbsd-13 > > The image was stored in the directory $vm_dir/images/ > I would expect to be stored at $vm_dir/.img/ > > Is it expected behavior? > If so, what is the purpose of $vm_dir/.img/ directory? From vm-bhyve(8):  img [-d datastore] [url] List all the cloud-              init images currently stored in the $vm_dir/.img directory.  This              is often useful during guest installation, allowing you to copy              and paste the image filename. For example # vm img https://download.freebsd.org/releases/VM-IMAGES/14.3-RELEASE/amd64/Latest/FreeBSD-14.3-RELEASE-amd64-BASIC-CLOUDINIT-zfs.raw.xz # vm img # vm create -s 20G -i FreeBSD-14.3-RELEASE-amd64-BASIC-CLOUDINIT-zfs.raw testvm > Thanks. From nobody Mon Sep 22 15:23:34 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVn214lwlz682gT for ; Mon, 22 Sep 2025 15:23:37 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVn212Q8gz3YmG for ; Mon, 22 Sep 2025 15:23:37 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758554617; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n0lTpAGwcyHonEEW62GuJ8SqsWFRvu96+gx6TdI7OIY=; b=JgypntfK/RZ76JX9Wb6HkIIoO0JbJtYfXl/S/OhcQBUt4wOd5OFIfT3/wEBRx/zF8CN6sW 2/P5fb6w78bTFoaMqSGPYL4AT03uduq7Sew6nUNeEUEHbUZhyEmYFBWKA/+w6U2GOXtqwE wDF0XsY1fToZiOM874ijy2h+2/+I+zAW+P0r/G47iqZFtK4vlULz1xjXCwOB4MWZdgNG9E ySy03RDRG0Sqi/qxMSxnqPiaiR+o+wv3k3qgN8lIHS1TvX9O8ilQ9+HhxHB3FEmVedS1Bi qr/VA5cgnukQlTyJc5Ko51M+q9G6b7ma1otXiflGaSOl35CjAyCqENgKwj8Kcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758554617; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n0lTpAGwcyHonEEW62GuJ8SqsWFRvu96+gx6TdI7OIY=; b=LvMBz2iOVrGOZF4DPMzvRnxTgDmj8Z9dBcATi968Ra/F5DDQ3ClOCesU/nLiS0CAZmpb/5 yPPYr7ueB4C5JQJUcxerFaNdabs0KXHX+t64Wkij7g/aGMNIKNh5B3z+bDuWCN8QvXRrYt GuyD86Oy5j/T8hNxufPHZl59Ho/DiJ/9/THT0IkO0YJBNTCE7R7bNHMvLTIPqZ4B6rSbL+ dUJyLKqATZp8k9R5Q5HKBv0biKZrX5HKLKVcH9xalZxLU1p01DZQhECLSOPpoGXH1T906p EP9grF8GtS39WXovZxLL6Wuy6X07IrF0CAyVEBspSzLx0qcGYulYc/+Kufiu8A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758554617; a=rsa-sha256; cv=none; b=Z42KObvrlW9fILCYM10bvcgsyVn7+Etf2p656CcMeXO74ktINuX94WLsvLhycqd42l2cLu JaRm7OZf3oHJ4yEYOrpkFw83KQl39gHuEq2nz93rjyBs4vYvqOdD7KnRBQqmySQY9UNx0g HsbHr9YzcXLUOdkTX7E03PBUYgw79yzz3KbCqjeFxA2WywU+2Dtxe/CSBCx0d5AJ6Yy0l1 VWB+N+XDU8MXSRoUttqgStd83f1oZLRm+wNErNuNr3vJiHXYN7ZNUpO/jqQh58kxIJjVhz 6gU2SfBMT2J/ZIM4qU996rVPciGSucI2TLahJQPXAM689eh8ITlTO3IvFqocwQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [192.168.0.88] (unknown [93.188.39.137]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4cVn2103l1ztx8 for ; Mon, 22 Sep 2025 15:23:36 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> Date: Mon, 22 Sep 2025 18:23:34 +0300 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) To: virtualization@freebsd.org References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <7YJWddCC_SUuB_mwDmHL3xecft8_rMou1xosTzBIK1UP_Fw-B786LWZX6CQ8XG6smJQRlxbfJbCq8fmTI15RouBn8GN73IfJvPOg7k6jr-s=@xyinn.org> Content-Language: en-US From: Andriy Gapon In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 22/09/2025 14:25, Corvin Köhne wrote: > NVIDIA should to be in the same shape as AMD on CURRENT since [1]. I'm going to > MFC it soon to land it in 15.0. This means: some users are reporting a working > setup for Windows and Linux guests, however it's not tested very well yet. I wonder if FreeBSD-on-bhyve-on-FreeBSD works with NVIDIA pass-through. I know that it's unusual to pass-through a GPU to FreeBSD, because what a FreeBSD VM can do that a FreeBSD host cannot. But there can be reasons like hosting a FreeBSD desktop on a FreeBSD "server". -- Andriy Gapon From nobody Mon Sep 22 16:12:06 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVp6926nLz685xJ for ; Mon, 22 Sep 2025 16:12:17 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.protonmail.ch (mail-4317.protonmail.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVp666Ny0z3h3x for ; Mon, 22 Sep 2025 16:12:14 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=Sj4i3uh6; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 185.70.43.17 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758557531; x=1758816731; bh=LauRQ0rexgjcqo1b1F+shTsjYGSzAEIIAzEsa8kCuos=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Sj4i3uh6Z/8xtvm6YsdmEgwuej9WjySGdN0JwbRrGbsh10Sy3eDHybhqZkmZHjbVx IZ72mWXWyr6sdX3nCXHbh3/qgK7hLQFuXf/7mXpnyNdW0bZASEjj8n4GmmVihc7piH N1x2OnSoXsS9ZBtL39K/YCldJCFconsm+fxDf5iqMjuiia2o2CtRm0T4tUb/tVgZcf FPBMD3PWBEYhajjiynszgX6sETw8yT9mQ3nwLYnNlelsSm1wkAfcmhcwtWT7dN5pw/ fFg7Dnkg6KWPSma/EMZKH2vwM4QhQSaVSG5fQ4hFv5zXRfjsD3CXxOKIjKUE0N2JZP jxXI1CeEFYWqA== Date: Mon, 22 Sep 2025 16:12:06 +0000 To: "virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <7YJWddCC_SUuB_mwDmHL3xecft8_rMou1xosTzBIK1UP_Fw-B786LWZX6CQ8XG6smJQRlxbfJbCq8fmTI15RouBn8GN73IfJvPOg7k6jr-s=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 0208653425ad4b23e510c61479afb79b264e1b0d List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[185.70.43.17:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[jon]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[185.70.43.17:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cVp666Ny0z3h3x The machine ran for 22+ hours with no issues. I decided today to do a fresh restart of the host and the vm to continue testing the "deterministic stability" of the setup. A few minutes after it started, the= VM exited with a 0x88. This is a new error code that I haven't seen before. I then rebooted the host/vm again and so far it's be= en running fine for the past 1h27m on idle. So the stability still seems to be MUCH better than when I was using the multi-functional de= vice on 18/0/0 for the USB controller. Using the 13/0/0 USB controller (th= at isn't multi-functional) has massively increased stability. I'm not expecting the VM to be perfectly stable for 365 days 24/= 7 given that even regular "real" machines also act weird / have issues ever= y once in a while. I've uploaded some of the crashes I got back when I had = the 18/0/0 setup which was giving me more frequent crashes with error 0x60,= and I've also posted the one I got today after the reboot with 0x88. I'm n= ot sure if this is enough because it's hard to get debug information from t= he console output that bhyve is displaying. Usually when the VM crashes I j= ust see my console say: exited with NDELAY mode or something like that. Which is the same message I= see when I do a regular shut down. So not very helpful. You can find the output of "bhyvectl --vm=3D --get-all" (which includ= es --get-exit-reason) here: https://xyinn.org/freebsd/files/gpu_pass/bhyve_= crashes/ also, does anyone know how I can send the ACPI shutdown signal to my bhyve = VM? I looked at the source code for vm-bhyve, and I just see it does a regu= lar 'kill'. I also saw some people online recommending to do a "pkill bhyve= ", but regardless of what I do, doing a kill/pkill on the PID running my bh= yve vm doesn't actually do anything. The VM still continues functioning per= fectly fine. I would like to be able to shutdown the machine from the host = so I can do a graceful shutdown. I can easily run my start up script by add= ing: @reboot root /path/to/gaming/script.sh in /etc/crontab, and that works fine, but I can't get it a signal from the = host to the vm so I can automate the shutdown counterpart. Thank you! From nobody Mon Sep 22 16:28:32 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVpT40zxmz686nG for ; Mon, 22 Sep 2025 16:28:40 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10624.protonmail.ch (mail-10624.protonmail.ch [79.135.106.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVpT32SFdz3jN1 for ; Mon, 22 Sep 2025 16:28:39 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b="Hv8E/+sD"; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.24 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758558516; x=1758817716; bh=GWsGoLgsTPjBvhM9rXSRxhCQBzv1mkSHIYVYKPanNiA=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Hv8E/+sDQIaKMr3sCzElJ9tlcKAPKMKGHZiG62jk0ByHVkHv9LTZ+0klVgBrhp4bk gbY86dmhkmNclSGjtGK3NYCxvxRhTEjAD9DiwMYENPPaj5HYtZ6b15f1tf0MyMupRm cKYHg0Sc9qZ+51FQOuxHZemJOa9M/FOoSeaPl27cNkov7LMLrzWv0tfFVTmP2C9rJq kCSJGtj/mFSlcQ+U2lhhDJF+eVC12ygPN5Hpqef2xN4fHu/1iH91P4ISO4B72MAo6e SvPL9bTP5+AAf6ErhhuWJieVHofjCSRZcsUD6/F3ey1QYzbJvHv7+dqSpZFs7t7XHt K0jtlkKZws07w== Date: Mon, 22 Sep 2025 16:28:32 +0000 To: "virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 5bf75b701fbbb47f224ea147ad6f744019c809a2 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.24:from]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[jon]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.24:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cVpT32SFdz3jN1 I've also started to upload some of these crash files (like the 0x88.0.txt = file) to Lumo for analysis. The AI info is interesting but it tends to have= errors on invalid bhyve command line flags or sometimes even what code it = thinks 0x88 belongs to. I'll post some of it below. This was its response t= o the 0x88.0.txt data. If we look at what it says for the 0x88 code, it doe= sn't match what I see in the freebsd code (#define=09VMCB_EXIT_ICEBP=09=09= =090x88): https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef17326= 8cdfe1365978a282178/sys/amd64/vmm/amd/vmcb.h#L94 is the AI wrong about this? ---- Lumo Response: I can see you=E2=80=99ve uploaded a text dump named 0x88.0.txt that looks l= ike a snapshot of a bhyve/AMD=E2=80=91SVM virtual=E2=80=91machine (VM) stat= e and statistics. Here=E2=80=99s a concise walkthrough of what the file con= tains and what it tells us about the VM=E2=80=99s execution. 1. High=E2=80=91level picture Item=09Value / Meaning VM exit reason=090x88 =E2=80=93 this is the exit code for VM_EXIT_REP_INSB = (a repeated string=E2=80=91input operation, i.e., rep insb/insw/insd). Current exception vector=096 =E2=80=93 a #UD (invalid opcode) or #GP depend= ing on context; the dump shows the VM is presently handling an exception. Number of CPUs=0916 vCPUs (IDs=E2=80=AF0=E2=80=9115) are active and suspend= ed; the host has a 16=E2=80=91core topology. Total VM exits=093=E2=80=AF259=E2=80=AF037 =E2=80=93 a fairly large number,= indicating heavy interaction between guest and hypervisor. Instruction=E2=80=91emulation exits=091=E2=80=AF465=E2=80=AF913 =E2=80= =93 most of the exits are for instructions that bhyve had to emulate (e.g.,= I/O, privileged ops). Pause=E2=80=91instruction exits=091=E2=80=AF138=E2=80=AF635 =E2=80=93 the g= uest frequently executed pause, which bhyve intercepts for power=E2=80= =91saving. HLT exits=0975=E2=80=AF539 =E2=80=93 the guest entered the halt state many = times (often seen in idle loops). In/out exits=09492=E2=80=AF316 =E2=80=93 the guest performed port=E2=80= =91I/O operations that required hypervisor handling. CPUID exits=093=E2=80=AF735 =E2=80=93 the guest queried CPU features. MSR reads/writes=091=E2=80=AF783 reads, 95 writes =E2=80=93 typical for a m= odern OS that accesses model=E2=80=91specific registers. Nested page=E2=80=91fault exits=09610 =E2=80=93 indicates the guest= =E2=80=99s second=E2=80=91level address translation (NPT/EPT) missed some p= ages. Memory layout=09Two huge sysmem regions (=E2=89=8833=E2=80=AFGB and = =E2=89=8829=E2=80=AFGB) plus a 16=E2=80=AFKB boot ROM region. Most of the R= AM is =E2=80=9Cwired=E2=80=9D (pinned) for the VM. RTC time=09Mon Sep 22 10:35:00 2025 =E2=80=93 matches today=E2=80=99s date,= confirming the snapshot is recent. 2. What the 0x88 exit really means Opcode involved: rep insb/insw/insd (string input from an I/O port). Why bhyve exits: AMD=E2=80=91SVM (and Intel VT=E2=80=91x) cannot direct= ly execute this instruction in guest mode; they force a VM exit so the hype= rvisor can either emulate the I/O or inject a #DB (debug) exception. Typical causes: Guest reading from a legacy serial port, keyboard controller, or ot= her device that uses bulk input. Anti=E2=80=91debugging tricks where malware repeatedly executes rep= insb to detect a hypervisor. Impact: Each such exit costs a few microseconds of latency. A high freq= uency can degrade performance, especially if the guest is in a tight loop d= oing bulk reads. 3. Other notable fields Field=09Interpretation cr_intercept =3D 0xfee2fee2=09The hypervisor is set to intercept many contr= ol=E2=80=91register accesses (CR0, CR3, CR4, etc.). exc_intercept =3D 0x00040000=09Exception interception is enabled for the #P= F (page=E2=80=91fault) vector. inst1_intercept =3D 0xddc4000f=09Intercepts a specific instruction pattern = (likely hlt and pause). exception_bitmap =3D 0x40000=09Only the page=E2=80=91fault exception (bit= =E2=80=AF18) is being intercepted. io_bitmap =3D 0x2b053b000=09Defines which I/O ports cause exits; the presen= ce of many in/out exits suggests a fairly wide bitmap. msr_bitmap =3D 0x2b0539000=09Controls which Model=E2=80=91Specific Register= accesses trigger exits. capability flags=09The VM supports hlt_exit, pause_exit, unrestricted_guest= , ipi_exit; lacks mtrap_exit, enable_invpcid, etc. guest_pat =3D 0x0007010600070106=09PAT (Page Attribute Table) settings = =E2=80=93 standard write=E2=80=91combining / write=E2=80=91through configur= ation. asid =3D 0x07fd=09Address=E2=80=91Space Identifier used for NPT/EPT = =E2=80=93 helps reduce TLB flushes. rflags =3D 0x40193=09Typical user=E2=80=91mode flag state (IF=3D1, ZF=3D0, = etc.). rip =3D 0xffffd001b2ce151d=09Current instruction pointer; the address falls= within the first sysmem region, suggesting the guest is executing normal c= ode when the dump was taken. rsp =3D 0xffffe3018da8e7b8=09Stack pointer =E2=80=93 also in the wired sysm= em area. 4. Performance clues High pause=E2=80=91exit count (>1=E2=80=AFM) + high hlt=E2=80=91exit co= unt (~75=E2=80=AFk) indicate the guest spends a lot of time in idle loops. = That=E2=80=99s typical for a server VM that sleeps when idle. Large number of instruction=E2=80=91emulation exits (~1.46=E2=80=AFM) s= uggests the guest is doing a fair amount of privileged I/O (e.g., disk, net= work, or device drivers) that the hypervisor must emulate. Relatively low NMI and ExtINT counts (0) imply the VM isn=E2=80=99t rec= eiving asynchronous hardware interrupts, which aligns with a mostly self= =E2=80=91contained workload. 5. What you might want to investigate next Why so many rep insb exits? Look at the guest=E2=80=99s I/O port map (e.g., serial console, leg= acy floppy). If the guest is repeatedly reading from a port that isn= =E2=80=99t needed, you could disable that device in the bhyve config. If the exits are caused by malicious anti=E2=80=91debug code, consi= der adding a pass=E2=80=91through for that port or using a more permissive = I/O bitmap. Tune the I/O bitmap If you know which ports the guest actually needs, narrow the bitmap= to reduce unnecessary exits. Check pause=E2=80=91exit handling Modern kernels already use pause for spin=E2=80=91waits; the high c= ount is normal, but you can enable the pause_exit capability (already set) = to let the hypervisor handle them efficiently. Memory wiring Almost all RAM is marked =E2=80=9Cwired/iommu=E2=80=9D. If you don= =E2=80=99t need that much pinned memory, you could relax the wiring to allo= w the host to reclaim pages under pressure. Look at the instruction pointer (rip) Mapping 0xffffd001b2ce151d back to the guest=E2=80=99s binary (via = objdump/gdb on the guest image) could reveal exactly which instruction caus= ed the 0x88 exit. That can confirm whether it=E2=80=99s a legitimate driver= read or something suspicious. TL;DR Your dump shows a bhyve VM that is currently handling a VM_EXIT_REP_INSB (0= x88) =E2=80=93 a bulk input operation from an I/O port. The VM has been qui= te busy: over three million exits, many of which are for instruction emulat= ion, pause, and I/O. Memory is largely wired, and the guest is idle= =E2=80=91heavy (lots of hlt/pause). If you want to improve performance or d= iagnose a possible anti=E2=80=91debugging pattern, focus on the ports gener= ating the rep insb exits and consider tightening the I/O bitmap or providin= g a proper device backend. From nobody Mon Sep 22 16:30:31 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVpWf5Lmsz6877r for ; Mon, 22 Sep 2025 16:30:54 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from fhigh-b8-smtp.messagingengine.com (fhigh-b8-smtp.messagingengine.com [202.12.124.159]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVpWf2tvbz3k5g for ; Mon, 22 Sep 2025 16:30:54 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.stl.internal (Postfix) with ESMTP id 4B3827A0136; Mon, 22 Sep 2025 12:30:53 -0400 (EDT) Received: from phl-imap-17 ([10.202.2.105]) by phl-compute-04.internal (MEProxy); Mon, 22 Sep 2025 12:30:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1758558653; x=1758645053; bh=O9Yax50YPD IDxC7421uiZXRzsGf8hw7sqSX+xVi47uw=; b=RAyFYpf4EM4x68OU9x9yTXnQ9c Fk2vbIDt6hcJUl0/bF6Jz8gzONJGLxPYXqGfZ3uvYs8+iF05Ax1wkGjZD+ZGel7+ QUEGPDEodm35BEQCoE7Qe5ij2k9SZVuHVnHFLw5N/BeVQ6D5DG46p303kUIMqmTX Sf1dMU0qPYF3ugyyjcviv/IqD/OHoZhn+gB3FCF7WK7p1F0FvNYasApL8THUf5iS Ml5ksUOnq38Lrk+zTptnK/JFAO84/GFYV/BDLOpRC5psbcXcRxLZD3cS4ZBS7a2F GoMwyWMGE2hqtQm7AZpOEc/i3lsZU3giJsxaJXnvBb/1BcfRLRIATKJNA88A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1758558653; x=1758645053; bh=O9Yax50YPDIDxC7421uiZXRzsGf8hw7sqSX +xVi47uw=; b=G4oukUlV1j/47Sq4zP0aDSRMhOa9/lEwl7wsZrMWlucmCUoAr/g gBXax6lv3qru75HjhluEK0LFq3U+1zU7yehA5c5nSKudtHaXLF5higeME18TBydT NGlXPMWfO8hF+z6i4ReauJcWhmDULADUnHIvj8nMKHS700463gfo25jbelNKHddQ ozB6CRQypM51aodSHTCddVhQHoRvEfJZMshegRYaanHh4sPX+OAfXlnp+Q4OZhfZ Lx+66TCPdLIMYTEJICIvrpBSJyH8aIScgHNGnUOe//uN6opE9laBJTb3eUOJuE5X zGx4zwVdKNkDVUagjKzMqQSYWP6Zu1uSXjg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdehkeeffecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefoggffhffvkfgjfhfutgesrgdtreerredttdenucfhrhhomhepfdevhhhutghkucfv uhhffhhlihdfuceotghhuhgtkhesthhufhhflhhirdhnvghtqeenucggtffrrghtthgvrh hnpeejvefgveffleffheevveeivefghfeujeeikeehfffhgfffteeugfeghfettdetvden ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegthhhutg hksehtuhhffhhlihdrnhgvthdpnhgspghrtghpthhtohepvddpmhhouggvpehsmhhtphho uhhtpdhrtghpthhtohepfhhrvggvsghsugdqvhhirhhtuhgrlhhiiigrthhiohhnsehfrh gvvggsshgurdhorhhgpdhrtghpthhtohepsggrkhhulhesihhithgsohhmsggrhidrohhr gh X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 1266EC40072; Mon, 22 Sep 2025 12:30:52 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 X-ThreadId: Aevyxa_LX6Wc Date: Mon, 22 Sep 2025 09:30:31 -0700 From: "Chuck Tuffli" To: "Bakul Shah" , "FreeBSD virtualization" Message-Id: In-Reply-To: References: Subject: Re: Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. Content-Type: multipart/alternative; boundary=4b53f30f18b7492ab14758e439a3c192 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cVpWf2tvbz3k5g --4b53f30f18b7492ab14758e439a3c192 Content-Type: text/plain Content-Transfer-Encoding: 7bit On Sun, Sep 21, 2025, at 5:20 PM, Bakul Shah wrote: > On a 15.0-ALPHA3 host, bhyve fails when I try to start a 9front VM. > Error: > > Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 706. > fbuf frame buffer base: 0x38bfae800000 [sz 33554432] There doesn't appear to be an "easy button" to debug this, but this assert trips on an error in any of the BAR registrations. It would be instructive to understand the type of BAR (IO, memory, ROM, etc.) causing the error as that would indicate where to look next. If you run bhyve via a debugger (with symbols), it should stop on the assert, allowing you to investigate. But if the debugger approach is problematic, I'd sprinkle this function and others with EPRINTLN() (assuming that is something you are comfortable doing). --chuck --4b53f30f18b7492ab14758e439a3c192 Content-Type: text/html Content-Transfer-Encoding: quoted-printable
On Sun, Sep = 21, 2025, at 5:20 PM, Bakul Shah wrote:
On a 15.0-ALPHA3 host, bhyve fails when I try t= o start a 9front VM.
Error:

Assertion= failed: (error =3D=3D 0), function modify_bar_registration, file /usr/s= rc/usr.sbin/bhyve/pci_emul.= c, line 706.
fbuf frame buffer base: 0x38bfae800000 [sz 33= 554432]

There doesn't appear t= o be an "easy button" to debug this, but this assert trips on an error i= n any of the BAR registrations. It would be instructive to understand th= e type of BAR (IO, memory, ROM, etc.) causing the error as that would in= dicate where to look next. If you run bhyve via a debugger (with symbols= ), it should stop on the assert, allowing you to investigate. But if the= debugger approach is problematic, I'd sprinkle this function and others= with EPRINTLN() (assuming that is something you are comfortable doing).=

--chuck
--4b53f30f18b7492ab14758e439a3c192-- From nobody Mon Sep 22 17:36:53 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVr013dW7z68Bft for ; Mon, 22 Sep 2025 17:37:05 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVr0138Zwz3rFb for ; Mon, 22 Sep 2025 17:37:05 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758562625; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H2Eqh+EaYWFWtH5BHukHr1KUwREBuKJe4zFm+xYnoIM=; b=Pveqa5ZMURH8GNrAwc8DCdpbxs34BoMiROHG5e0Y3h6vJ+kJeqxF8l2FDxw0jTtT4Ry8f1 cROecPHCMtGZUZWdzluAzgFVtCzNOj1YcuikTkmFQ2kP+6fX+ion20gqqgc56/7sqmFwBG 7q3jtD4+/s7Vbeoi75d4Jrmksswey1sKCae9iPxr2a0EQc6veI6UQNMvtF1m/ESuqGi9pj Gc7kx1LKcSbcjT7MIOcVFVx94XXSVVm8F3nVXlSTZWdfnOLPO7nfP45eZMcnLJSZPuo2Pi ZLgkgP3g5bcDPVnSqoHDVMMBLIh0JexAJ5c60EhpaUBYa4z4WXOkURQ/kE6Org== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758562625; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H2Eqh+EaYWFWtH5BHukHr1KUwREBuKJe4zFm+xYnoIM=; b=KfG09KB5++95u+q5eu+U9C7+ulDyFfjfBhUoD7R8TPji6VfFAnxUG3XLQHpM6wOTc9j0DS tPLcLG5ATeNnkcsv+VkOzlCEP4CzgLAG6c6B54BSxc4ChEYOkZ1OgkcUWUaxr8ApOK+9uF lFF8LEKjeNnn1bS6A48n383jn+8vsDK+MJ6tiJeyxs3ZMvQKqzcZVttvH2f85O9JCEfW5B 2Lc+WxKcMCJqR9KV8BJjDyE7LDRe4eOoDuCLsKML8v1bK2HgYugc3Tx059PKc2fJpDp3DO TTZmzo1U0D8s1KlK1+DoBDHd0+ruh58mlvWVubwJDaLYW952z115i11dzgcuVg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758562625; a=rsa-sha256; cv=none; b=UCyjYpiCW4UEIDSCceQ2mDzdEG4TZ9oIKNsZ8vskPIp6zeYVAPZRPYoAe4tYnoO9J5Cn+p fegMoZF6hJ3KC3tYE7CgsRUtR7dctjjBlSKTbC3/p6Z4k5LbwpGGv3ijA/hPs5DJ69URY9 MKFiCY43wYVXyenfrHibHL6ZIVtQr/i8UyKAFyul5NqYvwrIuH3r3iICDl9drNI+of+mCS ozWY0MsoOAE3o2JwdX/TxNdEzm29fZrjcHIqxaOSM48xS3vTGm7yqB6XOTXLsvPzgu1jap IEC1Dr4OqE6vjSXy3hoFpvmtyFPpDsuV2hMg9caqiBJoINfZyork0d2VEKDzJg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4cVr011xx3zxkm for ; Mon, 22 Sep 2025 17:37:05 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-84a4b995839so6956485a.1 for ; Mon, 22 Sep 2025 10:37:05 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCUjKpgB3I6aHFP7917DzCgr+eBVt+wZbRPGzuwuzWkqQpsLOIq/InGpVxMJHJizKnZfSnlgQICtB3vPPZDYeA==@freebsd.org X-Gm-Message-State: AOJu0YxNDiynAvsHhDHQlrgEOKQud90jRgIj9P5u1HUveOnkXE8plNot 6PjOXDVqI5KEAZePlfIu6gBa7hBw1ssrmXO+XkS3XXaIxffXI4f9A3twhSsN/Iln1Tw5a00z0Pp 1vZZi2dUMgnO8ClE09g3bcz3fKSSd/PM= X-Google-Smtp-Source: AGHT+IES2/qn0RPPAB+DsoKE8pWr1yBrClQ7hRoP5iUjUq9jFm/syaYa9/GZB910g01QD/LU5R6USqwhFgBehXdJrOw= X-Received: by 2002:a05:620a:458d:b0:827:e9b0:2732 with SMTP id af79cd13be357-83bab285919mr1067520085a.13.1758562624612; Mon, 22 Sep 2025 10:37:04 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <6QpHOGTzqnqxLPCDMcDbChlobQw-mVGWb2BjQk-K7kyE3OjuLCCFLGn5nVpgEvlr3xaipUzLMWz1JaESXx9W6w7aXNyAXN5wbuoS8eaWkcU=@xyinn.org> <7YJWddCC_SUuB_mwDmHL3xecft8_rMou1xosTzBIK1UP_Fw-B786LWZX6CQ8XG6smJQRlxbfJbCq8fmTI15RouBn8GN73IfJvPOg7k6jr-s=@xyinn.org> In-Reply-To: From: Nuno Teixeira Date: Mon, 22 Sep 2025 18:36:53 +0100 X-Gmail-Original-Message-ID: X-Gm-Features: AS18NWBjUhM5dDNUHJ1i1m_w7HiF277PjBuA6EsqLABq_b3SJaG7JNq4tKjt2Ws Message-ID: Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) To: =?UTF-8?Q?Corvin_K=C3=B6hne?= Cc: Jonathan Vasquez , "virtualization@freebsd.org" , Tomek CEDRO Content-Type: multipart/alternative; boundary="0000000000008e3155063f674493" --0000000000008e3155063f674493 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Corvin and all on this thread, Last time I've searched for docs and manuals on how to properly setup graphics passtru in bhyve, documentation was very fragmented. Is there a a wiki page or a place where docs are placed so we can access? If not, we could start a new thread on virtualization@ to start gathering all we can get, so a wiki or a complete manual could be made and incorporated in our handbook. Thanks all! Corvin K=C3=B6hne escreveu (segunda, 22/09/2025 =C3= =A0(s) 12:25): > On Sun, 2025-09-21 at 00:47 +0100, Nuno Teixeira wrote: > > 2- Do you know how is nvidia status compared to your use case? > > Hi Nuno, > > NVIDIA should to be in the same shape as AMD on CURRENT since [1]. I'm > going to > MFC it soon to land it in 15.0. This means: some users are reporting a > working > setup for Windows and Linux guests, however it's not tested very well yet= . > So, > it's an experimental setup which might fail sporadically, on heavy load o= r > long > time testing. > > Note that QEMU does implement some additional quirks for NVIDIA cards [2]= . > None > of those are implemented in bhyve yet. However, most of them seem to be > related > to older cards. > > [1] https://reviews.freebsd.org/D51892 > [2] > > https://elixir.bootlin.com/qemu/v10.1.0/source/hw/vfio/pci-quirks.c#L481-= L495 > > > -- > Kind regards, > Corvin > --=20 Nuno Teixeira FreeBSD UNIX: Web: https://FreeBSD.org --0000000000008e3155063f674493 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Corvin and all on this thread,
Last time I've searched for docs and manuals on how to properly = setup graphics passtru in bhyve, documentation was very fragmented.

=
Is there a a wiki page or a place where docs are placed so we can acc= ess?
If not, we could start a new thread on virtualization@ t= o start gathering all we can get, so a wiki or a complete manual could be m= ade and incorporated in our handbook.

Thanks all!
=
Corvin K=C3=B6hne <corvink@freebsd.org> escreveu (segunda, 22/09/2025 =C3=A0(s) 12:25= ):
On Sun, 2025-= 09-21 at 00:47 +0100, Nuno Teixeira wrote:
> 2- Do you know how is nvidia status compared to your use case?

Hi Nuno,

NVIDIA should to be in the same shape as AMD on CURRENT since [1]. I'm = going to
MFC it soon to land it in 15.0. This means: some users are reporting a work= ing
setup for Windows and Linux guests, however it's not tested very well y= et. So,
it's an experimental setup which might fail sporadically, on heavy load= or long
time testing.

Note that QEMU does implement some additional quirks for NVIDIA cards [2]. = None
of those are implemented in bhyve yet. However, most of them seem to be rel= ated
to older cards.

[1] https://reviews.freebsd.org/D51892
[2]
https://elixir.bootlin.= com/qemu/v10.1.0/source/hw/vfio/pci-quirks.c#L481-L495


--
Kind regards,
Corvin


--
Nuno Teixeira
=
FreeBSD UNIX:=C2=A0 <eduardo@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 https://Fr= eeBSD.org
--0000000000008e3155063f674493-- From nobody Tue Sep 23 08:59:01 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWDRt0Rhhz68KB9 for ; Tue, 23 Sep 2025 08:59:06 +0000 (UTC) (envelope-from matt@userve.net) Received: from smtp.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo Public Server Authentication CA DV R36" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWDRs4Hm6z3wlm for ; Tue, 23 Sep 2025 08:59:05 +0000 (UTC) (envelope-from matt@userve.net) Authentication-Results: mx1.freebsd.org; none Received: from dc.ad.unitron.net (owa.usd-group.com [217.196.1.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.userve.net (Postfix) with ESMTPS id 58A4F2383EE; Tue, 23 Sep 2025 09:59:03 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1758617943; bh=8t6puSwS13fDIlu0yhWtkE6/nOlqV9LkyaBJBh456g8=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=msCtZyLHsi8Ih+axfbX9Yiy27JeCgOeRSa++Sl/LrV7EtHpajOiICQNDrTNiV4Nib C87On8RAItK63qhSV3AdAkUfDI+yNbrwW4iO31eKrtUxrdzdtVTXdXxQWk6qGzP5Ke +InneWA/DwxRHEhJJrGbQHpKuVbB230RPjdk0cdU= Received: from dc.ad.unitron.net (192.168.0.100) by dc.ad.unitron.net (192.168.0.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.25; Tue, 23 Sep 2025 09:59:02 +0100 Received: from dc.ad.unitron.net ([fe80::d116:6a94:195f:a9ba]) by dc.ad.unitron.net ([fe80::d116:6a94:195f:a9ba%6]) with mapi id 15.02.1258.025; Tue, 23 Sep 2025 09:59:02 +0100 From: Matt Churchyard To: Jonathan Vasquez CC: "freebsd-virtualization@freebsd.org" Subject: RE: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Thread-Topic: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Thread-Index: AQHcFUJVicasA63gi06IgdQLXyaJZ7RzYw+AgAABWwCAAAFEAIAAEzmQgAAIXICAACHdgIAABA4AgAg034CAAMZKgIAAJpOAgBmM9YCAAKJ5gIAAY9wAgADIWQCAAAlrgIAABrgAgAALToCAAARegIAACZQAgAMWbwCAAXI3gIACVWYAgABCdACAAA2PAIABJjxA Date: Tue, 23 Sep 2025 08:59:01 +0000 Message-ID: <77221679b528434788d667441d1a32a2@userve.net> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <7YJWddCC_SUuB_mwDmHL3xecft8_rMou1xosTzBIK1UP_Fw-B786LWZX6CQ8XG6smJQRlxbfJbCq8fmTI15RouBn8GN73IfJvPOg7k6jr-s=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.8] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cWDRs4Hm6z3wlm Qmh5dmUgaXMgZGVzaWduZWQgdG8gdHJhcCBhIGtpbGwgKHRlcm0pIHNpZ25hbCBhbmQgdHJpZ2dl ciBhY3BpIHBvd2Vyb2ZmLCBzbyB0aGlzIGlzIHRoZSBkb2N1bWVudGVkIGFuZCBpbnRlbmRlZCB3 YXkgdG8gcG93ZXIgb2ZmIGEgZ3Vlc3QNCg0KRnJvbSB0aGUgbWFuIHBhZ2U6DQoNClNJR05BTCBI QU5ETElORw0KICAgICAgIGJoeXZlIGRlYWxzIHdpdGgJdGhlIGZvbGxvd2luZyBzaWduYWxzOg0K ICAgICAgIFNJR1RFUk0JVHJpZ2dlcglBQ1BJIHBvd2Vyb2ZmIGZvciBhIFZNDQoNClRoaXMgZGVm aW5pdGVseSB3b3JrcyBhbmQgaXMgY29tcGxldGVseSByZWxpYWJsZSB3aXRoIG1vc3QgZ3Vlc3Rz LiBWbS1iaHl2ZSBzZW5kcyBpdCB0d2ljZSBhcyB0aGlzIHNlZW1lZCB0byBnZXQgV2luZG93cyB0 byByZXNwb25kIHRvIHRoZSByZXF1ZXN0IG11Y2ggbW9yZSBjb25zaXN0ZW50bHkgYW5kIGl0IHdv dWxkIG9mdGVuIGFwcGVhciB0byBjb21wbGV0ZWx5IGlnbm9yZSBhIHNpbmdsZSBraWxsIGNvbW1h bmQuIEkgZG9uJ3Qga25vdyB3aHkgdGhpcyBpcyB0aGUgY2FzZSB0aG91Z2ggb3Igd2hldGhlciBv dGhlciBoeXBlcnZpc29ycyBkbyBzb21ldGhpbmcgZGlmZmVyZW50IHdoZW4gcmVxdWVzdGluZyBh IHNodXRkb3duIGNvbXBhcmVkIHRvIHdoYXQgYmh5dmUgZG9lcy4gU2ltcGx5IHRyaWdnZXJpbmcg YWNwaSBwb3dlcm9mZiAic2hvdWxkIiBiZSBlbm91Z2guDQoNCk1hdHQNCg0KDQotLS0tLU9yaWdp bmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogb3duZXItZnJlZWJzZC12aXJ0dWFsaXphdGlvbkBGcmVl QlNELm9yZyA8b3duZXItZnJlZWJzZC12aXJ0dWFsaXphdGlvbkBGcmVlQlNELm9yZz4gT24gQmVo YWxmIE9mIEpvbmF0aGFuIFZhc3F1ZXoNClNlbnQ6IDIyIFNlcHRlbWJlciAyMDI1IDE3OjEyDQpU bzogdmlydHVhbGl6YXRpb25AZnJlZWJzZC5vcmcNClN1YmplY3Q6IFJlOiBHUFUgUGFzc3Rocm91 Z2ggb24gRnJlZUJTRCAxNC4zIChBTUQgUmFkZW9uIFJYIDY5MDAgWFQgYW5kIFdpbmRvd3MgMTAg UHJvKQ0KDQpUaGUgbWFjaGluZSByYW4gZm9yIDIyKyBob3VycyB3aXRoIG5vIGlzc3Vlcy4gSSBk ZWNpZGVkIHRvZGF5IHRvIGRvIGEgZnJlc2ggcmVzdGFydCBvZiB0aGUgaG9zdCBhbmQgdGhlIHZt IHRvIGNvbnRpbnVlIHRlc3RpbmcgdGhlICJkZXRlcm1pbmlzdGljIHN0YWJpbGl0eSIgb2YgdGhl IHNldHVwLiBBIGZldyBtaW51dGVzIGFmdGVyIGl0IHN0YXJ0ZWQsIHRoZSBWTSBleGl0ZWQgd2l0 aCBhIDB4ODguIFRoaXMgaXMgYSBuZXcgZXJyb3IgY29kZSB0aGF0IEkgaGF2ZW4ndCBzZWVuIGJl Zm9yZS4gSSB0aGVuIHJlYm9vdGVkIHRoZSBob3N0L3ZtIGFnYWluIGFuZCBzbyBmYXIgaXQncyBi ZWVuIHJ1bm5pbmcgZmluZSBmb3IgdGhlIHBhc3QgMWgyN20gb24gaWRsZS4gU28gdGhlIHN0YWJp bGl0eSBzdGlsbCBzZWVtcyB0byBiZSBNVUNIIGJldHRlciB0aGFuIHdoZW4gSSB3YXMgdXNpbmcg dGhlIG11bHRpLWZ1bmN0aW9uYWwgZGV2aWNlIG9uIDE4LzAvMCBmb3IgdGhlIFVTQiBjb250cm9s bGVyLiBVc2luZyB0aGUgMTMvMC8wICBVU0IgY29udHJvbGxlciAodGhhdCBpc24ndCBtdWx0aS1m dW5jdGlvbmFsKSBoYXMgbWFzc2l2ZWx5IGluY3JlYXNlZCBzdGFiaWxpdHkuIEknbSBub3QgZXhw ZWN0aW5nIHRoZSBWTSB0byBiZSBwZXJmZWN0bHkgc3RhYmxlIGZvciAzNjUgZGF5cyAyNC83IGdp dmVuIHRoYXQgZXZlbiByZWd1bGFyICJyZWFsIiBtYWNoaW5lcyBhbHNvIGFjdCB3ZWlyZCAvIGhh dmUgaXNzdWVzIGV2ZXJ5IG9uY2UgaW4gYSB3aGlsZS4gSSd2ZSB1cGxvYWRlZCBzb21lIG9mIHRo ZSBjcmFzaGVzIEkgZ290IGJhY2sgd2hlbiBJIGhhZCB0aGUgMTgvMC8wIHNldHVwIHdoaWNoIHdh cyBnaXZpbmcgbWUgbW9yZSBmcmVxdWVudCBjcmFzaGVzIHdpdGggZXJyb3IgMHg2MCwgYW5kIEkn dmUgYWxzbyBwb3N0ZWQgdGhlIG9uZSBJIGdvdCB0b2RheSBhZnRlciB0aGUgcmVib290IHdpdGgg MHg4OC4gSSdtIG5vdCBzdXJlIGlmIHRoaXMgaXMgZW5vdWdoIGJlY2F1c2UgaXQncyBoYXJkIHRv IGdldCBkZWJ1ZyBpbmZvcm1hdGlvbiBmcm9tIHRoZSBjb25zb2xlIG91dHB1dCB0aGF0IGJoeXZl IGlzIGRpc3BsYXlpbmcuIFVzdWFsbHkgd2hlbiB0aGUgVk0gY3Jhc2hlcyBJIGp1c3Qgc2VlIG15 IGNvbnNvbGUgc2F5Og0KDQpleGl0ZWQgd2l0aCBOREVMQVkgbW9kZSBvciBzb21ldGhpbmcgbGlr ZSB0aGF0LiBXaGljaCBpcyB0aGUgc2FtZSBtZXNzYWdlIEkgc2VlIHdoZW4gSSBkbyBhIHJlZ3Vs YXIgc2h1dCBkb3duLiBTbyBub3QgdmVyeSBoZWxwZnVsLg0KDQpZb3UgY2FuIGZpbmQgdGhlIG91 dHB1dCBvZiAiYmh5dmVjdGwgLS12bT08bmFtZT4gLS1nZXQtYWxsIiAod2hpY2ggaW5jbHVkZXMg LS1nZXQtZXhpdC1yZWFzb24pIGhlcmU6IGh0dHBzOi8veHlpbm4ub3JnL2ZyZWVic2QvZmlsZXMv Z3B1X3Bhc3MvYmh5dmVfY3Jhc2hlcy8NCg0KYWxzbywgZG9lcyBhbnlvbmUga25vdyBob3cgSSBj YW4gc2VuZCB0aGUgQUNQSSBzaHV0ZG93biBzaWduYWwgdG8gbXkgYmh5dmUgVk0/IEkgbG9va2Vk IGF0IHRoZSBzb3VyY2UgY29kZSBmb3Igdm0tYmh5dmUsIGFuZCBJIGp1c3Qgc2VlIGl0IGRvZXMg YSByZWd1bGFyICdraWxsJy4gSSBhbHNvIHNhdyBzb21lIHBlb3BsZSBvbmxpbmUgcmVjb21tZW5k aW5nIHRvIGRvIGEgInBraWxsIGJoeXZlIiwgYnV0IHJlZ2FyZGxlc3Mgb2Ygd2hhdCBJIGRvLCBk b2luZyBhIGtpbGwvcGtpbGwgb24gdGhlIFBJRCBydW5uaW5nIG15IGJoeXZlIHZtIGRvZXNuJ3Qg YWN0dWFsbHkgZG8gYW55dGhpbmcuIFRoZSBWTSBzdGlsbCBjb250aW51ZXMgZnVuY3Rpb25pbmcg cGVyZmVjdGx5IGZpbmUuIEkgd291bGQgbGlrZSB0byBiZSBhYmxlIHRvIHNodXRkb3duIHRoZSBt YWNoaW5lIGZyb20gdGhlIGhvc3Qgc28gSSBjYW4gZG8gYSBncmFjZWZ1bCBzaHV0ZG93bi4gSSBj YW4gZWFzaWx5IHJ1biBteSBzdGFydCB1cCBzY3JpcHQgYnkgYWRkaW5nOg0KDQpAcmVib290IHJv b3QgL3BhdGgvdG8vZ2FtaW5nL3NjcmlwdC5zaA0KDQppbiAvZXRjL2Nyb250YWIsIGFuZCB0aGF0 IHdvcmtzIGZpbmUsIGJ1dCBJIGNhbid0IGdldCBpdCBhIHNpZ25hbCBmcm9tIHRoZSBob3N0IHRv IHRoZSB2bSBzbyBJIGNhbiBhdXRvbWF0ZSB0aGUgc2h1dGRvd24gY291bnRlcnBhcnQuDQoNClRo YW5rIHlvdSENCg0K From nobody Tue Sep 23 10:09:30 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWG1J0SdDz68QVv for ; Tue, 23 Sep 2025 10:09:40 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10624.protonmail.ch (mail-10624.protonmail.ch [79.135.106.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWG1G2jyhz46xF for ; Tue, 23 Sep 2025 10:09:38 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=WGjFyCaF; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.24 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758622175; x=1758881375; bh=tmEVN1UbgrIc5bmTsF/JhbRfltcCUvEg+udQ0L68+P8=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=WGjFyCaF7Y26KpzcFDKndz/irLIAWESEx/K6q9lduA7QtwnVI/D2cuyUYTEkLhF2w od4kXOQHKyiXXp4tsQYLigaE6GDQD+x2GbWkXsz9+WdXvVZSnt9F74UzzxIht2+xaI 7YpHukXG/+0b0AH0HdDzv9rx/uFujQvP1wnZBa+rVOPFbq7lZN17gGZDRpwjNHbY0q Yi+GUpqSU8APdYkxZUr0fJOoC6XR1WEL/ffiXTwXqCK7wCK5+3yecc5Rb7SUZYMxXB yolh42VhAcWFBIRGq/0W8yzkdu/L+hZu8+AVcBUEm3bcWrW3AkzODWdIlYoP818357 r83PQTaZBThDA== Date: Tue, 23 Sep 2025 10:09:30 +0000 To: freebsd-virtualization@freebsd.org From: Jonathan Vasquez Subject: Re: RE: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: <77221679b528434788d667441d1a32a2@userve.net> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: cdca416886639c4eee02a2fccc5393341ffa219d List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.24:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jon]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.24:from]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cWG1G2jyhz46xF Thanks Matt. I'm going to give this a shot! Does anyone have any tips on how I would go about debugging bhyve, specific= ally I want to collect more info about why sometimes something is causing i= t to exit the VM? From nobody Tue Sep 23 10:16:20 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWG992k2Hz68QPD for ; Tue, 23 Sep 2025 10:16:29 +0000 (UTC) (envelope-from meka@tilda.center) Received: from mail.nsd1.sysit.solutions (meka.rs [109.93.255.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWG990HlZz498t for ; Tue, 23 Sep 2025 10:16:28 +0000 (UTC) (envelope-from meka@tilda.center) Authentication-Results: mx1.freebsd.org; none Received: from [192.168.111.162] (hal9000.nsd.sysit.solutions [192.168.111.162]) by mail.nsd1.sysit.solutions (Postfix) with ESMTPSA id 9844869619; Tue, 23 Sep 2025 10:16:20 +0000 (UTC) Message-ID: <7581e630-51c8-45fd-9b2e-238fd3f2a613@tilda.center> Date: Tue, 23 Sep 2025 12:16:20 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) To: Jonathan Vasquez , freebsd-virtualization@freebsd.org References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> Content-Language: en-US From: =?UTF-8?Q?Goran_Meki=C4=87?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8400, ipnet:109.92.0.0/15, country:RS] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cWG990HlZz498t On 9/23/25 12:09, Jonathan Vasquez wrote: > Thanks Matt. I'm going to give this a shot! > > Does anyone have any tips on how I would go about debugging bhyve, specifically I want to collect more info about why sometimes something is causing it to exit the VM? > If something is really sending the signal, you could try dtrace. When I was chasing signals for something totally unrelated, dwatch was the easiest way to do it. I hope it helps! Regards, meka From nobody Tue Sep 23 11:03:31 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWHCf0kckz68Tp7 for ; Tue, 23 Sep 2025 11:03:42 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-24420.protonmail.ch (mail-24420.protonmail.ch [109.224.244.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWHCd5YGDz4GCf for ; Tue, 23 Sep 2025 11:03:41 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758625417; x=1758884617; bh=n74q2c6PZl8q4GFWjNHNsWLlTKRZBxZfsnv67IzDB3Y=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=NTAZx/ipvS2a/Z4MzEtoRFYpvpVdUpNh9eQPxzGiAxnyS10mBy6RomBWJHDIUMu/j ZJTIwnQ/xEo+EXHCkK0GLKQH7O1590lK53wH2ipQIyJIJwKow+hUdEyvipBv3Y1M9D 3DgcG05xHOSCPzdPVadMvfkTZfUmVhqEa9Evs6O5t0ik1KWfENYLvDEE71cJimvGJV zkr/bd14xz6L3C+TxYo4LAxqKMx38+/4Hqhsdo6Bi4nkYj7j6/jNXi7tcJQUTHNEG4 WB0N/yXmfE7a4iPei/036EtOz9l97fQ8qxQqSch2gNNpTPmRRG0M20qHEqMFW3Py+5 FjS8P79/4z2Ow== Date: Tue, 23 Sep 2025 11:03:31 +0000 To: =?utf-8?Q?Goran_Meki=C4=87?= From: Jonathan Vasquez Cc: freebsd-virtualization@freebsd.org Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: <7581e630-51c8-45fd-9b2e-238fd3f2a613@tilda.center> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <7581e630-51c8-45fd-9b2e-238fd3f2a613@tilda.center> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 45d4e070771cf1b39dc5db8b5222a7276fc8ae86 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:109.224.244.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cWHCd5YGDz4GCf Thanks Goran, I'll give that a try. I tried to send a "kill " pid and send it mutiple times and nothing.= the Windows 10 VM completely ignores it and stays strong lol. I also tried= "pkill bhyve" and same thing. "pgrep bhyve" returns the correct PID. I onl= y have one bhyve VM running so there shouldn't be any confusion. Not sure w= hy SIGTERM is getting ignored. From nobody Tue Sep 23 11:10:07 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWHM640Zmz68V9f for ; Tue, 23 Sep 2025 11:10:10 +0000 (UTC) (envelope-from meka@tilda.center) Received: from mail.nsd1.sysit.solutions (meka.rs [109.93.255.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWHM56b68z4JYf for ; Tue, 23 Sep 2025 11:10:09 +0000 (UTC) (envelope-from meka@tilda.center) Authentication-Results: mx1.freebsd.org; none Received: from [192.168.111.162] (hal9000.nsd.sysit.solutions [192.168.111.162]) by mail.nsd1.sysit.solutions (Postfix) with ESMTPSA id 6D39B69520; Tue, 23 Sep 2025 11:10:07 +0000 (UTC) Message-ID: <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> Date: Tue, 23 Sep 2025 13:10:07 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) To: Matt Churchyard , Jonathan Vasquez Cc: "freebsd-virtualization@freebsd.org" References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> Content-Language: en-US From: =?UTF-8?Q?Goran_Meki=C4=87?= In-Reply-To: <77221679b528434788d667441d1a32a2@userve.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8400, ipnet:109.92.0.0/15, country:RS] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cWHM56b68z4JYf On 9/23/25 10:59, Matt Churchyard wrote: > Bhyve is designed to trap a kill (term) signal and trigger acpi poweroff, so this is the documented and intended way to power off a guest > > From the man page: > > SIGNAL HANDLING > bhyve deals with the following signals: > SIGTERM Trigger ACPI poweroff for a VM > > This definitely works and is completely reliable with most guests. Vm-bhyve sends it twice as this seemed to get Windows to respond to the request much more consistently and it would often appear to completely ignore a single kill command. I don't know why this is the case though or whether other hypervisors do something different when requesting a shutdown compared to what bhyve does. Simply triggering acpi poweroff "should" be enough. I can just guess about this. When I press a power button on a desktop with Windows 10, it just ends screensaver (well, blank screen, to be precise), so I have to press it another time to actually turn it off. It does sound like a similar "issue". Regards, meka From nobody Tue Sep 23 11:27:24 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWHl725xMz68VpN for ; Tue, 23 Sep 2025 11:27:31 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.protonmail.ch (mail-4317.protonmail.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWHl65jYHz4Myc for ; Tue, 23 Sep 2025 11:27:30 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758626846; x=1758886046; bh=xMTQxNqZHPauLns26BCGc5W69YIBzJFzMO5hhT0UYTo=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=DoXuGYcyKWcsuNs5hod9mfEM23qs8rbL+is2IaW7qYVstmSKDHN+YUK4Wo+hNUZDZ +G0/JcsnAZ/hYB+Hdsrty/saJueT2EWR7H4w2dg7yxgSgjdv6W5/mUv2qqw3b+riPo Fh8UcH/pqZx975LtnEyWWptPmX1qC71IMnK9oJttO1JWwMZPU40JVieiG1De55x2Xu MNxvVPJlM67dWN1usqMxMxaZIQqNTGEUE5gcG7EgE0SDhPt7XcYibm6szlGmgfPqIz 9mRsZdM16+dbOdpKYO4O4mB2IEKYcTAYJ0QjY8KglKklD8DKfaUpY//+9k7Cgytk/v DxrD3YKv4bThQ== Date: Tue, 23 Sep 2025 11:27:24 +0000 To: =?utf-8?Q?Goran_Meki=C4=87?= From: Jonathan Vasquez Cc: Matt Churchyard , "freebsd-virtualization@freebsd.org" Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 342bb73d28e6bfe941cdbeb99f14befbf6e7520a List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cWHl65jYHz4Myc That was it Meka! Thank you! I actually normally change my power settings in Windows 10 for the followin= g: Choose what the power buttons do: - When I press the power button? Do Nothing - When I press the sleep button? Do nothing Haha. I changed the "power button" option to "Shut down" and now it works. = I also first tested it by changing it to "Turn off the display", and a sing= le pkill made my monitor turn off. Then I moved the mouse and the screen tu= rned back on. So that makes completely sense why it was ignoring me. From nobody Tue Sep 23 12:50:17 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWKZq4T2fz68c8l for ; Tue, 23 Sep 2025 12:50:27 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.protonmail.ch (mail-4317.protonmail.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWKZl1Gkgz3SLS for ; Tue, 23 Sep 2025 12:50:23 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=I3OT4Szj; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 185.70.43.17 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758631820; x=1758891020; bh=KC4qe9TeDWbGUBQlqi7s30rVueNu65y8QJI2GwdHzSM=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=I3OT4Szj/M+bBpSKrle27cjUkB3w0fZTwG4E/MJ7uj+HYSgO5Gap/6bXs5L93nqtb tvxVpNh5+KGY5JApYexCG11FXeOIIOIJt6UE/HKqdvTsw+xtyDk1/vrVyKjwQkVCSo ZvYk9fC+iVb9/CS6d0OvgKnu5f9mi2HApfZCIFwjOtUdZzV4GIcSwm1PmvwKE10n8C hnmAGke7TCH/fv6SUhjq/BbDKDc82Q9NtIiKS+e+xnOeejFSyQaKra3dE9Wau5fyNk VSStUUR0rdGSxrljZBsSodk9/GNFcCngvTkVKV6IikdoYmNT+1ZFF0OBhLFR6w+3Cz ZO0RLwJFK4RQA== Date: Tue, 23 Sep 2025 12:50:17 +0000 To: =?utf-8?Q?Goran_Meki=C4=87?= From: Jonathan Vasquez Cc: Matt Churchyard , "freebsd-virtualization@freebsd.org" Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: e6bc22d34daea356800598f4574d8012def05c2d List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[185.70.43.17:from]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[185.70.43.17:from]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[jon]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MISSING_XM_UA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_COUNT_ZERO(0.00)[0]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cWKZl1Gkgz3SLS Alright all. So a few updates: After the 22+ hour uptime two days ago, I did a manual reboot of host/guest= to continue testing the "deterministic stability" of the system (as I ment= ioned yesterday). Within a few minutes, the VM exited with a new error code= (0x88) (previously it was 0x60). After I did the PCI passthrough adjustmen= ts, I haven't gotten 0x60 anymore and the VM has become a lot more stable. = For whatever reason I do sometimes still get a 0x88, but it hasn't happened= too many times. After the initial 0x88 exit code, I rebooted the host/vm a= gain, then the VM worked for about 3-4 hours before getting another 0x88. A= fter another host/vm reboot, the VM became stable and ran for another 19 ho= urs (which is this morning). I then turned it off since I wanted to experim= ent with the pkill situation we were discussing recently. So besides figuri= ng out what's causing it to sometimes do a 0x88, we are close I think to a = stable setup. I've noticed that in situations where it stays stable, I usua= lly have my Task Manager up since I want to see the resource utilization, t= emperatures, and uptime of the system quickly, while I monitor it. Could th= ere be something with the task manager always polling different hardware fo= r their state that may cause the VM to stay alive (vs not having task manag= er and just idling the machine causing it to die?). There is no sleep suppo= rt in the VM so we don't need to worry about the VM going to sleep/hiberati= on. I've also worked on a start/stop script to integrate this into the rc.d sys= tem a little bit better. Before I was using /etc/crontab and @reboot to sta= rt the VM at boot time after everything has started up, which works fine, b= ut I couldn't find a @shutdown equivalent. So I wrote a basic rc.d script. = The script is working but the problem is that for w/e reason it blocks the = rest of the services on my machine from starting. I investigated it a bit a= nd it has to do with "rcorder". I also analyzed the order by doing "rcorder= /etc/rc.d/* /usr/local/etc/rc.d/*" which revealed where in the chain it fa= lls. Originally there was no "REQUIRE:" line so it was being thrown into th= e beginning of the chain. I then switched it to "REQUIRE: LOGIN" which thre= w it close to the end of the chain (which is good), but it still has some s= ervices that follow it, and I noticed that even though I can ping the box a= fter reboot, for w/e reason even though sshd falls before LOGIN, I still co= uldn't ssh into the box until I shutdown the Windows VM, which allows execu= tion to continue. The start script I have for bhyve is a blocking call sinc= e it runs "bhyve ..." which would end up capturing the terminal. When I do = it through /etc/crontab @reboot, this doesn't seem to be an issue, but it i= s when doing it through an rc.d script. Anyone have any recommendations abo= ut this? I'll post the three scripts I'm using at the end of this message. Also, I'm more open now to writing a new section for the handbook regarding= everything I've learned for GPU passthrough for gaming purposes. Although = I have to preface that my experience is limited and it's only optimized for= this flow, so if I write it, I would only be able to write about this part= icular path and things/caveats around it. Normally the handbook usually has= a more general knowledge approach as well which I'm not qualified to write= about at the moment. Would it be fine for me to start writing this section= and get it into the handbook, and other people more experienced than me ca= n expand that section with more general knowledge about bhyve/passthrough (= and of course Intel/NVIDIA stuff can come after)? I can start with GPU pass= through / gaming / AMD Host + AMD Dedicated Card / and I can also add the s= tart/stop scripts + rc.d script. The paths, names, and values will need to be adjusted for each person's ind= ividual setup: VM START SCRIPT: #!/bin/sh =20 VM_DIR=3D"$(dirname $(realpath $0))" =20 VM_NAME=3D"$(basename $VM_DIR)" =20 cd "$VM_DIR" =20 bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu \ -l com1,stdio \ $VM_NAME # Exit the script here and leave some options for debugging more =20 # ergonomically after the exit line. =20 exit =20 # Only use this when installing the VM for the first time: =20 # VNC. Once you install Windows and enable RDP, you can turn this off. = =20 -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1024,h=3D768,wait \ =20 # The Windows ISO and the VirtIO Drivers ISO. =20 # You can download the latest stable virtio drivers at: =20 # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-v= irtio/virtio-win.iso =20 -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ =20 # If you want a network device without using virtio, you can use e1000. = =20 # You should install the virtio drivers and use virtio-net instead for = =20 # better performance. =20 -s 2,e1000,tap0 \=20 -s 2,virtio-net,tap0 \ =20 VM STOP SCRIPT: #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=3Dgaming --destroy RC.D SCRIPT: #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable=20 start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" vm_gaming_start() { =09/atlantis/vms/gaming/start.sh } vm_gaming_stop() { =09/atlantis/vms/gaming/stop.sh } load_rc_config $name=20 : ${vm_gaming_enable:=3D"NO"}=20 run_rc_command "$1" From nobody Tue Sep 23 12:56:01 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWKjM3bSTz68cTr for ; Tue, 23 Sep 2025 12:56:07 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4323.protonmail.ch (mail-4323.protonmail.ch [185.70.43.23]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWKjM1M5Bz3T1J for ; Tue, 23 Sep 2025 12:56:07 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758632164; x=1758891364; bh=9Sb2BVueHfLwdmjAG9knh7WWKf+lmvwZQlk5ea7ldAU=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=ZLokLRfs6VVCm1hNc0TpIswmG+tw4p00Ha3y0uMHfp3K5wmcI7lqXIwZ4Vmnr+zCE xGAYF4/LLdCU/6jRTH0d6hHBl6qUTAOooIPz2ZcDO2lZwDdhgEHvcEXlgVEXQkcHYe JJSZoTt8SPjm601eUXIP+PsjFiKmdA3lKl1Csw/G/Kmt7CoP+Cdknq960E9tWZB5gM Xk5tb5uy3Y6KqkgCLlFX3EozIl8SwY0g84pzbVFLD8pZqRFIE+hKZd8eLV3fBIeXQP CCcOjJg1vKe+/lZn9alpxOcsmSofj+eMeUDcBtk/Bbo0dkyuYqUdxwE9pNaAskSCVj EimTxsQXcwz/Q== Date: Tue, 23 Sep 2025 12:56:01 +0000 To: Jonathan Vasquez From: Jonathan Vasquez Cc: =?utf-8?Q?Goran_Meki=C4=87?= , Matt Churchyard , "freebsd-virtualization@freebsd.org" Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> In-Reply-To: <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 243c4b8dd37980ccddaf745f55ecd49f73172c3a List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cWKjM1M5Bz3T1J I can also confirm that doing a: service vm_gaming onestart is a blocking call since the console I run that on gets hijacked by the bhy= ve call in my start script. So I'll need to change something there so it co= ntinues. I've tried doing a & after the bhyve call but that through errors.= So there is a probably another way to make the bhyve call non-blocking wit= hout it failing (and still allowing the -l com1,stdio stuff to work). From nobody Tue Sep 23 13:25:53 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWLMn2Nfzz68f4y for ; Tue, 23 Sep 2025 13:25:57 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWLMn1w1Cz3cLC; Tue, 23 Sep 2025 13:25:57 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758633957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=RcPesz33cYwmNUV42dOunzDs7NerlbbfDR3w7ks08DM=; b=RwpBCZp5LAt1Pg65crsNS80c/1QAAwEwry1K0FDdl5ZgEaDBjfH6ghWs+zFbs/ZlGuUxF+ C6pCEFXpazoAGRlYsaCvMQhTOmxeJ4310+EuryTP/84Oyh7n2vvjz31TO/GcGRwYQNWXke eduUF0pGOz3FA2MW/a+SXh1SawG8WVdXy6HtIwUo0Lvar0eGfTMLdJLxV0J2p9HSGOp12u cxe5tYJCQ40oLfg7N0NNvREP8RlDeKRrKI0Qav4ZvQlFZLOkVZkVWTw0POGh2QOYjg7R7f Kyvy+D85f2ORA924wqRuldIB+gbltet2KMM7MB5llrm35SYcAQO+5FPfvEvMbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758633957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=RcPesz33cYwmNUV42dOunzDs7NerlbbfDR3w7ks08DM=; b=VbGog5kb2lWOrQWYXR5jiQCRnFOnCNmrp91HrkGfwI76NxnvFVdHeDWEh4az/GLqEizfmJ 5tSoXLtZu+SmL1PyaDYqRxdB1Z2XFpzEzBnJ6bZ1NfZbVl2XHYuriyElf2FwP7zXmpeMfX OYuZsnGqhy0uJiHe7FVveBRClrr5vI30mWT3cDLboklLhxJnBMFXOXh92ur7b/wcFvzqaR DiKxpg2JxFnemB/uoQknBbM1CAQVlKXeZ3VmaMunX5c9yEV4W54mIs1siXGCb58iltQdMd KBuBcZd405b9EfT4fXT+L8tVeOUHzXktCZTVcBmucvj5P7tln7FIY0WXHtkGhw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758633957; a=rsa-sha256; cv=none; b=rvf6doEeoRKIrPHuALmrXPAD4VBbo/3yXM+IkaiLHK2eYBGudcnByHAcggkzRaizzZvUjO nInTwU4NWF+XzTaxoW2gMaDLVgZo+C2sxk4SrfpX7gH+Gdi+kz/LbYVaqirgtThaCg4gZs axvQUmRSEejnlXYFoKF2EJlBVMui8IEK08WjDiItz1W/Z8Z1PHCi2Mq3XhKyzUkP2Xg2VJ 4q+o9wuRD0HKMubucdBgpQ0sfiAg5MKA4IFBJqUoXyKSY3xs5TPP/gW3nEhfXRzcwqkBU2 0NurX9wvQZvr5lWXdV66FlKl7bDr/e6z+pglI5Xjm/6h3PJWe6lKV+S+oyacvA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from corvink-nb.fritz.box (unknown [IPv6:2001:9e8:dc37:c300:923f:2186:5c58:997f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4cWLMm496jz1MTM; Tue, 23 Sep 2025 13:25:56 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Jonathan Vasquez Cc: Goran =?iso-8859-2?Q?Meki=E6?= , Matt Churchyard , "freebsd-virtualization@freebsd.org" Date: Tue, 23 Sep 2025 15:25:53 +0200 In-Reply-To: <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> Autocrypt: addr=corvink@FreeBSD.org; prefer-encrypt=mutual; keydata=mQINBGNjZaIBEADDTrDNf+0pwiuRPBdClcnZW83dH1UhuOi0u+A1J2SatEBbNaFVtXXAa vewCTuyV/ZbNidjlhq3R/pWyiKjFKvs5dj7PMCw+3z2D5OWpMdHg7TrB+fbdFPOEsu0zQVKNaO+pS KCfN0Re0m7bL3wuvl7PXvBufRwA3Guo1P4j3TXWaEkuso7VupTvE25zVGg9ONHrGOjA9RUy+Yg4Se 3NLgtUdjBgA21SBQTDvRQV4fDmVenlwvWeE0Xm8FcDcpQb6sJTihaDku78mi3Ux1HCk7rTcepVEB0 xIB6qmFxv0sLlDmVv6Z6qg1y/Q5m23Pgz60o3TulMPV4F+3Itm8ifU+wgVSzBZbD29GYkd7LKqMkF bhvfSBk+5db3vbYY5OD//+LTM5AV7e2AhXuXMvG1UNBqXqSJTTSy6KZz+qmPQO0zos0dq46p8o82l KiBEGD2Hu0p+u0OyV+MmRYo1NIBFVbOPXp2MvUVl5II0UIJ3+N9gLBmfGA+HEpVO8PnvdoT/5NQ7m 8JK1rQHzjiDub/iDPAYMqKH4C0eZ/7zO0fuY5FeRNtuNtpH1Bw/+7/5RJH7bcKkfGHHEp15FJUrGH gWNydoDLB9QBprwQc8FEldDXBjzOMXIgh6FGKLNu6DswvIPGy6M3u7DXwDakCXz+c9Ym0oFihLzZx WntrsxdswD/CwARAQABtCdDb3J2aW4gS8O2aG5lIDxjb3J2aW4ua29laG5lQGdtYWlsLmNvbT6JAl QEEwEIAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCZArUIwIbAwUJCWYBgAULCQgHAgYVCgkICwI EFgIDAQIeAQIXgAAKCRDYVNpWMV4CalcID/44k2i/mqSSi4W6FAobSF1nFLtP/pfcRNJriWKx2UF7 cfFMKyg7Nilg7FhLb5FDB1umUW2nFfchFPTUp4FfKzgRvPzIMg4RIRcVtTpYbl3z7zs9ZXD8qS//i ibbiUG3quncm6tO2x1jLZD3ORC+8MuLGXhYQIa4O5vVF2SBHdb/U6P+wsrF+U+OpRdEdQ/4Xu9S02 kltzBGgArjcexdhUqEqW01KCCSH0+qgfN1NE+9L934ZOB+cai2b9apPbCOGuV6KcUKMj4z0RWInhl XIyMqtmhdix/P/GjrQ1REVNdp74JeweSSedM15wwc6YLMXPrtOnnExyZ1gyNFYaub+Mdo3ZQ+8386 3B0C9IlpzEW0K8bYlZrl4WBNdcMOyByETAxgQmBgP6ZFErTtaeUOH1nX2FyR6o2GLSahRgngQmnRE zScTpPbBEkMwJMDAO+rbjjGxjeKSUwD1WOfbI6QZj+MS/uBk2p08kgN7fQaCEwj6jqML/IIE+FZ9I In6TNG0hChD384VHO+YioLBno1Atgi4Q7JUWSRIHQXZW+StQajFrWPPyKQwCe1MwqpKoMpX/q8IZB lzwJgZS8ShLeFZjtzOt1jgM99TD91Neonf9OzjTSbfo7sJviSWoICMhB/MvDZjj+naMVF86uGFxnI EsjVKyBxlJd4TRHnLYPTIHFKBLQjQ29ydmluIEvDtmhuZSA8Y29ydmlua0BGcmVlQlNELm9yZz6JA lQEEwEKAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCY2NlogIbAwUJCWYBgAULCQgHAwUVCgkICw UWAwIBAAIeAQIXgAAKCRDYVNpWMV4CavfDEACCFnXpR7H9eOgP+GJMNPtK6i9/xnqdyXi8uCZIN0h YwjN4Xzo9SMLOf4UUlQEveOB+bGqbRfHd/fGKnrlXiPd0SGpKWJC21gqL/DsIH0J8I3Whth+O8tfP WeFy0oCsvBaaGFLIrDfoIgHF9i/gqEe48xhN42weB02Z3mdR1L0d7ME/BLwS0mCXe9Zh3uHw63S6x YB3Wsjptxe/ph6TpQDUKWtRJkjC6BqXPBdThpbbfIRWmjZbp2fKEJPvtRXS14+gbUqWeJ4xCvprA3 +ae7vtrp91X775yngyW3XTw5cmDiJIjykH8+zhEIoNQXNBpFrehkQDYrcM+WoE6NGSJo+3VJvSRWh UGWDVrxdTYNkbIjmTNlkI12NINC007DiuV7OF9XHWgrbbylvuZvbODmbJRdhTFy9upAUygX1/xUAQ EIMqMiJmyTdv8i0IbZ611WElQx9XHgGeZgM6+39/laN8FwspM6gE/4NzZHIZN0LEBOerZqoF+Il6e ccQpoEWx7nb/RilJp3dUUyvkBnJWg+AJByosg857kvvmDnZ3UB+bejWpcfFvnbkiKPUBPDO7tWPb5 r2yFDpDe/Vg5x+sRbkkXGUFD6Rx0p/ZiRIneVVg8emTzhDR0IL/BZkq/uVJkdaphZli1F/31cNgf6 ZJYnjxlk86uiYSySZQR0dKLDqq7QlQ29ydmluIEvDtmhuZSA8Yy5rb2VobmVAYmVja2hvZmYuY29t PokCVAQTAQgAPhYhBIL0UpWt5trfx9lPRthU2lYxXgJqBQJkCtScAhsDBQkJZgGABQsJCAcCBhUKC QgLAgQWAgMBAh4BAheAAAoJENhU2lYxXgJqI+QQAML5PTR7KpUFV3SLG60LQJGEOHUfDmJYczxBFb IAq1U4hIbivopu1AdLty7oDDrIjCVoa2/Cy34dd99O7lhLvUmZFB/zDSUtbUg2zhDkU0YSZ11Fdrl Wzky2tFaQRgxpDvWlUP0baa3Pd4dPDRiIUI6AOSR2SL6XANk6sJh56gLVM6G8yyafGsxSyDYg6Z78 EEMFejHwB+KP2DdsahupzM+F97HeC1+bOHYxtqN+2hEkPLtQWizyumPqNg5FvZhwe7yO8V95hF3Rh uDO+9aJT+WLLvcZEb/L1bI04IvZ5FWgCLI7Levd/DuOtZI8gWapHhqGZRbXB2fuJkCoKCl6V67h/7 aWhU3LjFTsC5siJyrxPjapKcIk8a7PqZDswNCKR+24LJ5D59mPgEOnsiCCVpik1WE/kgD+rOu9dQx jpjKwuKowf4EJP1KYNkYtoy1HthzyTOqPwqXC3IUl0GVPO3xw6MhUM3irCVFruC+ecVrv85Rd37vr duT6JvgW92xjWegMsamtkDZH8Ik/cmYhH0K/qEc6OZVNea/4PTEZe0uxODJ6pbMd80AJyGqDPPVeA gWJtEIG4k6IS8XyD5v1QJtlpDVpPwP/bbFnVc3h3Oatfn8Etm1KAqYvNwyO+om2PkF1p732uapDZd LwksVmgc9s79+9pSpeP5DbIeMzhrIK Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-7mtupIuMH/vwuGvtDaNA" User-Agent: Evolution 3.56.2 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 --=-7mtupIuMH/vwuGvtDaNA Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2025-09-23 at 12:56 +0000, Jonathan Vasquez wrote: > I can also confirm that doing a: >=20 > service vm_gaming onestart >=20 > is a blocking call since the console I run that on gets hijacked by the b= hyve > call in my start script. So I'll need to change something there so it > continues. I've tried doing a & after the bhyve call but that through err= ors. > So there is a probably another way to make the bhyve call non-blocking wi= thout > it failing (and still allowing the -l com1,stdio stuff to work). You can take a look at daemon(8) [1]. [1] https://man.freebsd.org/cgi/man.cgi?daemon(8) --=20 Kind regards, Corvin --=-7mtupIuMH/vwuGvtDaNA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmjSn+EACgkQ2FTaVjFe AmrJmw//T2Hdfnu0HyOfW7SYY+cpdJbX9+898+v2PJrufsfbiDLL9zmuU7oulwMg f8iiDSHI2qCpNA0XbLVctbmGtUdMOWCXpTJ4Sb5pRCX8PxdKgNsZ9e6x1wlecoWu SwnLjaNox85jUf/zZKXzxDe2l4dRHjX7x1Bte73cx5QW8DaYWHBZ7wvPGnQYIaiL 2EKyhNfcQ7gCL8KOrrGgXw/ZTG7ZmH5i8T3JCaN1PoO3kio0JIK6RuAubWI5fnZe 5fvnTS8dTkSTx+9SAtU65dMobstL1hXfJglN3g4sCH7KKB/Dxp7Rsm9qPKymNwbU uGlXJoOTZq2hPFDf8KF1zdRefpJSbr6BvlKJ+8jHMfBG9hgyOuCCrwVfZn0yqPnT S0RQyohcIjGv5agUIWzZ9HhE3SeBeiwXUURpuOWlwo45UibPvadbxUjX6JKwckiR Yb3c/xhRsVSFkuWhOBTGB5DPqcF4qvZASAj7Ec3iazdhq3V3+vPn/WGYJ+pNL+6N l+xB/N64Y/ogNf3whljlJUPabGZnR2g0b85gpN8j8lOrhpobBqvEATvmcQtd6RU/ o0TJDA7xGd+MBUoydhZ91EBFRfP1EF26kWHOLxMkMk/QMqQHrIf+uIThdD9MKCXI ft21jT24GV/61EdDHKSqNLtHmZrp8tNnYtmtBk7Q7LzDIvG1x5s= =ykfc -----END PGP SIGNATURE----- --=-7mtupIuMH/vwuGvtDaNA-- From nobody Wed Sep 24 13:05:00 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWxsP05Whz67q7m for ; Wed, 24 Sep 2025 13:05:13 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10624.protonmail.ch (mail-10624.protonmail.ch [79.135.106.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWxsL1LqPz3WsK for ; Wed, 24 Sep 2025 13:05:09 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=NyRN6Vs7; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.24 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758719106; x=1758978306; bh=jWE7r/W7hQdL4vbI1QyMFk9wzDuE9a3Y38pBlsyRV1Y=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=NyRN6Vs7Pqz0UTXaOjB+HglSLuN440doWKhJoO+CSAveWW9r/VIf6USf22u3oFrp/ NSSR1guowYOlTXwrp95U8Qyubaf4F4QsYcG61j263n6La16fMBrkcJr0RI1kFDo9sF YmzOHibdwYMdBj5kuiQ5sNoz8xVejCZXQAkm7JEvPBWjmE90DShbYOcZFgvChvJuIs wQEFU1llTb/paMDlnHqRbUxQORPzPgFaotDcENRgxWQcn7wjzQN2d9C4l0AUuWT9Xn 3IKsP2CR7WPXUg2uW3qolxfXmEUZxupzlHZ2nZ5mozIy5QBw/KGSwIJZhDe6TvC4M6 XbB/f5Kec6gLA== Date: Wed, 24 Sep 2025 13:05:00 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> In-Reply-To: <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 99973b1cf5039c86cd1b4f3f71e4ae9cbe9f14f9 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.37 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.970]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.24:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[jon]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.24:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cWxsL1LqPz3WsK Hello all, I was able to get another stable vm on idle (this time without leaving Task= Manager opened) for another 22 hours. I also spent some time today improving my sta= rt/stop scripts. Using Corvin's suggestion about the "daemon" process, I was able t= o now successfully start the VM at boot time without blocking. I had to make = some adjustments to my bhyve script so it doesn't use "-l com1,stdio", because d= uring boot time, even with me making the script require "LOGIN syscons", it still would throw the following error: Unable to initialize backend 'stdio' for LPC device com1 100 Device emulation initialization error: Inappropriate ioctl for device Which prevented the VM from starting. I've switched to using "-o console=3D= stdio" and that worked. I'm able to get regular console output, which allows the V= M to always start up successfully, and also allows us to log to a file when w= e run our vm start script through daemon. I've also observed that there were a few times when I restarted the VM with= out restarting the host, 2-3 times, and each time the video output worked (mean= ing I wasn't affected by the AMD Hardware Reset Bug). However, after that I was= n't able to get that working anymore. So I would still recommend restarting the= host. I added the following to the script: - Daemonized support so the VM starts up non-blocking. This also includes logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since= the pid addresses looked off. We also don't seem to need it at the moment. - Added the restart command (with a note about the AMD issue). - Added some quality of life stuff like having some variables at the top to easily set your start/stop paths. Some other things are that if I do "service vm_gaming ", the current console in tmux seems to get really laggy and the input gets weird.= If I open another tmux window, then input is fine again. There are no other consequences other than laggy/wrong input in the current console. This does= n't seem to negatively affect anything when starting it automatically at boot t= ime through rc.d. One last thing is that I noticed that when I do a "shutdown -r now", I don'= t see windows getting the ACPI shutdown signal, so I don't see Windows gracef= ully shutting down. I have read if you do "reboot" it will just send the SIGTERM= signal, and kill everything afterwards, so the "shutdown" command should be used to give advance notice to processes. Given that my rc.d script has "shutdown" = in the keywords list, I was expecting the "vm_gaming_stop" command to be ran automatically in order to gracefully shutdown the VM, and clean itself up. = Is this not working because I'm not using -P / -p? I've updated my blog post to include all of these new scripts and modificat= ions. I'll post the current scripts below. Thanks for reading. Jonathan ## Bhyve Scripts ### start script This is the main script that will start your vm. Save this as **`start.sh`*= * in a folder called **`gaming`** somewhere on your system: ``` #!/bin/sh VM_DIR=3D"$(dirname $(realpath $0))" VM_NAME=3D"$(basename $VM_DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu \ -o console=3Dstdio \ $VM_NAME # Exit the script here and leave some options for debugging more # ergonomically after the exit line. exit # Only use this when installing the VM for the first time: # VNC. Once you install Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1024,h=3D768,wait \ # The Windows ISO and the VirtIO Drivers ISO. # You can download the latest stable virtio drivers at: # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-v= irtio/virtio-win.iso -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ # If you want a network device without using virtio, you can use e1000. # You should install the virtio drivers and use virtio-net instead for # better performance. -s 2,e1000,tap0 \ -s 2,virtio-net,tap0 \ ``` ### stop script This is your stop script. Save this as **`stop.sh`** somewhere. **`NOTE:`** This is currently set up to kill any process with **`bhyve`** i= n the name. On this system I'm only running one bhyve VM, which is the gaming= one. Adjust accordingly to restrict its scope. ``` #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=3Dgaming --destroy ``` ### rc.d script This script will allow you to start the VM automatically at boot time. Save= this file in your **`/usr/local/etc/rc.d/`** directory with the name **`vm_gamin= g`**. Make sure to adjust any paths to where you saved your start/stop scripts: **`NOTE:`** At the moment I noticed that if I do a **`shutdown -r now`**, I= don't actually see Windows gracefully shutting down (like when I send the ACPI sh= utdown signal). Currently looking into this. ``` #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" path_to_start_script=3D"/atlantis/vms/gaming/start.sh" path_to_stop_script=3D"/atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \ -o /var/log/${name}.log \ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_script}" } vm_gaming_restart() { # NOTE: AMD users will most likely experience the famous # AMD Hardware Reset Bug. This means that after you reboot # the guest, you most likely won't have video out. If this # happens, you'll need to restart the host. Sometimes this # command will work for AMD users though. So you can try a # restart and see if it works. vm_gaming_stop vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_command "$1" ``` From nobody Thu Sep 25 12:01:43 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cXXPz29CPz68Pdv for ; Thu, 25 Sep 2025 12:01:59 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10626.protonmail.ch (mail-10626.protonmail.ch [79.135.106.26]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cXXPs6MpPz3xld for ; Thu, 25 Sep 2025 12:01:52 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=B9mvWExO; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.26 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758801708; x=1759060908; bh=ot6TObKxNERBnysgye/p4IHr6UkFppW6yCcbdNJcDTY=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=B9mvWExO/hhjHMO/VEy/8RxJMBJkzuQXRxtuGhaV8g1/TnB1yKZI4TGyPYcg6S9Tc C0jcrnvSa0RP/ONPEM5vGx2pCW8Jz+6gQY+mItwwukKPCUM2ngjvaSR+BnqSO4hBnb ZHjOW9Xl0jonrtTAgUyJYtTVqxTW/nDfbpe/VsrHIc2N0nZVCL00hnIOx37bBvw/MF jfvMLfVMVffUmN2qNz9U4Iup748n7R/suP88eyw7ZQ4No/tJ6fMziYvIm0COaV/7va Hc3Jt72VEdOhFJJ0FSzRHaafBwl3jNweri2Zy2+59qf+2bB+QfXw4rZBDcaivNjDxf BEWHwMMJ5i39w== Date: Thu, 25 Sep 2025 12:01:43 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: a4562f7de2440b6a055cecd5e12ab9070c1cf1bd List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.26:from]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; FREEFALL_USER(0.00)[jon]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:62371, ipnet:79.135.106.0/24, country:CH]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_EQ_ADDR_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.26:from]; RCVD_COUNT_ZERO(0.00)[0]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cXXPs6MpPz3xld Hey all, I was able to get another 21+ hours of uptime on idle with no issues, and I= did some more gaming last night and it's stable. I haven't experienced any= of the weird vm exits recently so that's good. There are probably cases wh= ere the vm may exit but I'm not sure what those situations are given the re= cent stability. I'll keep monitoring but I'm pretty happy with how everythi= ng turned out so far. I've also debug the script a bit more to see why it wasn't shutting down. T= he docs aren't explicit about it (or at least I haven't found a place yet f= or it), but I think it may be related to the pid stuff. It doesn't seem tha= t rc.shutdown calls the stop command if it's simply defined given there are= performance optimizations to skip certain things even if KEYWORD: shutdown= is defined. So after doing a "ps -aux | grep gaming" and seeing what showe= d up, I saw that there are 3-4 pids that exist with my current set up. daem= on pid, main bhyve pid, start.sh pid, etc. The pid that was getting written= by daemon previously was the pid of my start.sh script, but the pid of the= bhyve process that gets started by start.sh, so that would be a "grandchil= d of daemon", so I think that's why originally it wasn't working for me. I = now merged my start/stop script logic into the main rc.d script, so daemon = is directly executing bhyve, and thus the child process recorded in the pid= file is correct. The stop command can now load up the child pidfile, and pr= operly call kill on that pid directly. My script now works on shutdown prop= erly. The only thing I noticed was that even though I do now see Windows di= splaying the "shutting down" message, it kinda seems like the system isn't = necessarily obeying my "sleep 10" call afterwards, which means it probably = still isn't 100% a graceful shutdown, but much better than before. There is= also the bhyvectl --destroy ... command after and I'm not sure if that's r= unning. Below is the improved and centralized rc.d script: #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable vm_path=3D"/atlantis/vms/gaming" vm_name=3D"gaming" start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" pidfile=3D"/var/run/${name}.child.pid" vm_gaming_start() { =09daemon \ =09=09-o /var/log/${name}.log \ =09=09-p "${pidfile}" \ =09=09bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ =09=09-s 0,hostbridge \ =09=09-s 1,nvme,${vm_path}/disk0.img \ =09=09-s 3:0,passthru,3/0/0 \ =09=09-s 3:1,passthru,3/0/1 \ =09=09-s 13:0,passthru,13/0/0 \ =09=09-s 30,xhci,tablet \ =09=09-s 31,lpc \ =09=09-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu = \ =09=09-o console=3Dstdio \ =09=09${vm_name} } vm_gaming_stop() { =09# Send SIGTERM twice to make sure Windows listens to =09# the ACPI shutdown signal. =09pid=3D"$(cat ${pidfile})" =09kill ${pid} =09kill ${pid} =09# Wait a bit for the guest to shutdown properly before =09# we continue shutting down the host. =09sleep 10 =09bhyvectl --vm=3D${vm_name} --destroy } vm_gaming_restart() { =09# NOTE: AMD users will most likely experience the famous =09# AMD Hardware Reset Bug. This means that after you reboot =09# the guest, you most likely won't have video out. If this =09# happens, you'll need to restart the host. Sometimes this =09# command will work for AMD users though. So you can try a =09# restart and see if it works. =09vm_gaming_stop =09vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_command "$1" Jonathan Vasquez PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 Sent with ProtonMail Secure Email On Wednesday, September 24th, 2025 at 09:05, Jonathan Vasquez wrote: > Hello all, >=20 > I was able to get another stable vm on idle (this time without leaving Ta= sk Manager > opened) for another 22 hours. I also spent some time today improving my s= tart/stop > scripts. Using Corvin's suggestion about the "daemon" process, I was able= to > now successfully start the VM at boot time without blocking. I had to mak= e some > adjustments to my bhyve script so it doesn't use "-l com1,stdio", because= during > boot time, even with me making the script require "LOGIN syscons", it sti= ll > would throw the following error: >=20 > Unable to initialize backend 'stdio' for LPC device com1 > 100 Device emulation initialization error: Inappropriate ioctl for device >=20 > Which prevented the VM from starting. I've switched to using "-o console= =3Dstdio" > and that worked. I'm able to get regular console output, which allows the= VM > to always start up successfully, and also allows us to log to a file when= we run > our vm start script through daemon. >=20 > I've also observed that there were a few times when I restarted the VM wi= thout > restarting the host, 2-3 times, and each time the video output worked (me= aning > I wasn't affected by the AMD Hardware Reset Bug). However, after that I w= asn't > able to get that working anymore. So I would still recommend restarting t= he host. >=20 > I added the following to the script: >=20 > - Daemonized support so the VM starts up non-blocking. This also includes > logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since= the > pid addresses looked off. We also don't seem to need it at the moment. > - Added the restart command (with a note about the AMD issue). > - Added some quality of life stuff like having some variables at the top > to easily set your start/stop paths. >=20 > Some other things are that if I do "service vm_gaming ", th= e >=20 > current console in tmux seems to get really laggy and the input gets weir= d. If > I open another tmux window, then input is fine again. There are no other > consequences other than laggy/wrong input in the current console. This do= esn't > seem to negatively affect anything when starting it automatically at boot= time > through rc.d. >=20 > One last thing is that I noticed that when I do a "shutdown -r now", I do= n't > see windows getting the ACPI shutdown signal, so I don't see Windows grac= efully > shutting down. I have read if you do "reboot" it will just send the SIGTE= RM signal, > and kill everything afterwards, so the "shutdown" command should be used = to > give advance notice to processes. Given that my rc.d script has "shutdown= " in > the keywords list, I was expecting the "vm_gaming_stop" command to be ran > automatically in order to gracefully shutdown the VM, and clean itself up= . Is this > not working because I'm not using -P / -p? >=20 > I've updated my blog post to include all of these new scripts and modific= ations. >=20 > I'll post the current scripts below. Thanks for reading. >=20 > Jonathan >=20 >=20 > ## Bhyve Scripts >=20 > ### start script >=20 > This is the main script that will start your vm. Save this as `start.sh` > in a folder called `gaming` somewhere on your system: >=20 > `#!/bin/sh VM_DIR=3D"$(dirname $(realpath $0))" VM_NAME=3D"$(basename $VM= _DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 3= 2G \\ -s 0,hostbridge \\ -s 1,nvme,disk0.img \\ -s 3:0,passthru,3/0/0 \\ -s= 3:1,passthru,3/0/1 \\ -s 13:0,passthru,13/0/0 \\ -s 30,xhci,tablet \\ -s 3= 1,lpc \\ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqe= mu \\ -o console=3Dstdio \\ $VM_NAME # Exit the script here and leave some = options for debugging more # ergonomically after the exit line. exit # Only= use this when installing the VM for the first time: # VNC. Once you instal= l Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=3D0.0.0.0:5= 900,w=3D1024,h=3D768,wait \\ # The Windows ISO and the VirtIO Drivers ISO. = # You can download the latest stable virtio drivers at: # https://fedorapeo= ple.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.is= o -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \\ -s 5,ahci-cd,.= ./files/virtio-win-0.1.271.iso \\ # If you want a network device without us= ing virtio, you can use e1000. # You should install the virtio drivers and = use virtio-net instead for # better performance. -s 2,e1000,tap0 \\ -s 2,vi= rtio-net,tap0 \\` >=20 > ### stop script >=20 > This is your stop script. Save this as `stop.sh` somewhere. >=20 > `NOTE:` This is currently set up to kill any process with `bhyve` in > the name. On this system I'm only running one bhyve VM, which is the gami= ng one. > Adjust accordingly to restrict its scope. >=20 > `#!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACP= I shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to sh= utdown properly before # we continue shutting down the host. sleep 10 bhyve= ctl --vm=3Dgaming --destroy` >=20 > ### rc.d script >=20 > This script will allow you to start the VM automatically at boot time. Sa= ve this > file in your `/usr/local/etc/rc.d/` directory with the name `vm_gaming`. > Make sure to adjust any paths to where you saved your start/stop scripts: >=20 > `NOTE:` At the moment I noticed that if I do a `shutdown -r now`, I don't > actually see Windows gracefully shutting down (like when I send the ACPI = shutdown > signal). Currently looking into this. >=20 > `#!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdo= wn . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable start_cmd=3D"${= name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" path= _to_start_script=3D"/atlantis/vms/gaming/start.sh" path_to_stop_script=3D"/= atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \\ -o /var/log/${na= me}.log \\ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_s= cript}" } vm_gaming_restart() { # NOTE: AMD users will most likely experien= ce the famous # AMD Hardware Reset Bug. This means that after you reboot # = the guest, you most likely won't have video out. If this # happens, you'll = need to restart the host. Sometimes this # command will work for AMD users = though. So you can try a # restart and see if it works. vm_gaming_stop vm_g= aming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_com= mand "$1"` From nobody Thu Sep 25 12:32:16 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cXY564g2Wz68Rbs for ; Thu, 25 Sep 2025 12:32:26 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.protonmail.ch (mail-4317.protonmail.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cXY550MJtz41Wb for ; Thu, 25 Sep 2025 12:32:25 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=dFqRfFmm; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 185.70.43.17 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758803540; x=1759062740; bh=/y2NS1Yju5VZScNddkYGHz+RQhtT0z/vt8DlpaEoLxE=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=dFqRfFmm3Cd6s774HFZ3P6Eic5jkYJV4c8FT2RkBsEtVp3E7ChyHSaq5TJrO6a0kJ yGlET13NxxIRhi038MRMwxUVbg1N6l7aCHXnHd/lmv69JONY1Ds9GLOohlLCmYqBgn ClKhtw0t4bHWOFdfnjM2xG6HrlBMxlGgbfmPvGcNvmWkYF+YHgNs6KKWXuzl5HEqNo 8fBCz2NPzaNueLk8B3rIvl0j13SOpbkAMWWI9EtR+MVn2ADHyJ1VYYr1a6HHGqUTcH J6EAn3vZ4MRZmsZQJSl0MYYvP3jnaUgGOtJCRGVO8hjsCLQaBUNIgLN5M7WBPpbJRk 4W/qzAW7bgGhg== Date: Thu, 25 Sep 2025 12:32:16 +0000 To: freebsd-virtualization@freebsd.org From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: e508338f7d14aa46ce6b3d09ce7fdaf533570600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.39 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[185.70.43.17:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH]; FREEFALL_USER(0.00)[jon]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[185.70.43.17:from]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cXY550MJtz41Wb I did another test with shutdown -r now and for whatever reason I don't see= Windows saying that it's shutting down. So I'm still not sure what's missi= ng. Everything seems to be properly linked and the start/stop commands to w= ork correctly when I do: service vm_gaming start/stop/restart. From nobody Fri Sep 26 00:00:00 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cXrLS3bt0z68rL3 for ; Fri, 26 Sep 2025 00:00:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cXrLS2qL4z3L2y for ; Fri, 26 Sep 2025 00:00:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758844800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=sH83wCedWIMi2pZSzQCPqmnjBIUvpvGoYGPZfNQGI3k=; b=b3xqXeMyYdSiGuGaKX2HsaW2+4cRybHtepYvPxGRb6mr5/pm5iVeydBnPUa6DNFv2WPI1i bPbfebPol7EcAfR4Df8OBZNL61fJMAoyBf2RKRFJzdA5mjz8wrWbZWOS42BD7jzMtbYzDK QSBCkY2VKyznNpiyu0T4Dr24z0DXEeUH8UQek6Jir8z4QwMeY0zsU6McGk0y0AIIcbC6tk MTrmRrlGFRILK10GDutsx0xaoreyC//makdqDD2C8fAUxWPp3ehjEwoDg9ciLoPCbI72Lu HtpcCXIYLCo+wMJABPWfTzWd8A1Yu5OlcJ/dLF/TKvFSCsDXz94g43e7yKu7SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758844800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=sH83wCedWIMi2pZSzQCPqmnjBIUvpvGoYGPZfNQGI3k=; b=gwoECUscjwSaMv3sR4ZE9NzAd19LFIvJI5e9/Otz4Sc9Tc8ww94C2gHAxzkiKgjW00vlTy BzUeiRAyobIfs4UbKFGoj4DzzQe0SFGTdHCUONRwskxpWw2PG2YsCRYQVyC+vPm2V4OUNW Wy3Vm1V4NkZmV7UfjeKLwzpJRJ5I+4BnoSUzMMrdkpZGav0BOvj6lyRQqSAYy2jk6ZcE09 1YgWRa5NUXyqi6HjtwnpIV7XOg2+n0sK3RsK1xGxS4df9ayJ5iH4zdYVCu3eD7GUNV9OG7 hh/+ZIX2wvZWzIdwRV7Lhbd70WTOFB6VGrKrKJqU1uu5S12bF9+V/Z3DCS4BDQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758844800; a=rsa-sha256; cv=none; b=G5QFZ5f/DClJcB9mb/zzf422+q2TOpnuzntj7F+tuCQcn+/kJQbzjNEIuxL8/JWsG57nos 12HoToO23pZxa193rpqfV3crnxmMiG7GoTZP5e/7e2HqGzvPVSB+DBDLJl4M5gnLqvgqBb VbtoT6iyPEs7XwR0aHTJU33MAWcrgg4nirNFAd8ldSL2fCQSKmv1glUisMjWp8NSCRLPcN PElvkxHNmn9FYM/d4Ydhuq/tVRYRV8zPlpiySqDp5w22gwoyPTpXlspsyv3mhG1WowWYvF drc2MR0giAV+NMCdtXozcVm5/MaB7NXVGXeekEmd6dkH40F7MNzPR9oD86uZiw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cXrLS1sMHz7XK for ; Fri, 26 Sep 2025 00:00:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 58Q000xi013377 for ; Fri, 26 Sep 2025 00:00:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 58Q000Os013344 for virtualization@FreeBSD.org; Fri, 26 Sep 2025 00:00:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 289848] Enable by default BHYVE_SNAPSHOT Date: Fri, 26 Sep 2025 00:00:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 16.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vvd@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: mfc-stable15? X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter flagtypes.name Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D289848 Bug ID: 289848 Summary: Enable by default BHYVE_SNAPSHOT Product: Base System Version: 16.0-CURRENT Hardware: Any OS: Any Status: New Severity: Affects Many People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: vvd@FreeBSD.org Flags: mfc-stable15? Ideally, this change would be included in the 15.0 release. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 26 00:58:12 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cXsf80rWZz68w5J; Fri, 26 Sep 2025 00:58:40 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cXsf64KWSz3Qnj; Fri, 26 Sep 2025 00:58:38 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of lists@jnielsen.net designates 2607:f170:34:11::b0 as permitted sender) smtp.mailfrom=lists@jnielsen.net Received: from smtpclient.apple ([IPv6:2607:fa18:e2fe:2d6a:e55f:7fce:3072:eb05]) (authenticated bits=0) by webmail5.jnielsen.net (8.18.1/8.18.1) with ESMTPSA id 58Q0wNZN037575 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 25 Sep 2025 18:58:26 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host [IPv6:2607:fa18:e2fe:2d6a:e55f:7fce:3072:eb05] claimed to be smtpclient.apple From: John Nielsen Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum Message-Id: <5D2A2499-4B43-4FE6-AA5A-AD334BC4E872@jnielsen.net> Date: Thu, 25 Sep 2025 18:58:12 -0600 Cc: np@freebsd.org To: net@freebsd.org, virtualization@freebsd.org X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.87 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.95)[-0.950]; NEURAL_HAM_LONG(-0.62)[-0.622]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; APPLE_MAILER_COMMON(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[jnielsen.net]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,net@freebsd.org]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4cXsf64KWSz3Qnj Hi- Not sure if this is a known issue or even an issue on the FreeBSD side = but decided to ask here first. I=E2=80=99m happy to put in a bug report = if appropriate. I have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected. As workarounds go that isn=E2=80=99t terrible, but I was surprised that = the FreeBSD guests required this adjustment while the Windows one did = not. As a point of reference I have another hypervisor machine that runs = FreeBSD 14-STABLE. It also has a Chelsio T520-CR adapter. FreeBSD guests = on that machine do not require =E2=80=9C-rxcsum=E2=80=9D in order to use = the Chelsio virtual functions. Let me know if anyone can shed some light on this. I=E2=80=99m happy to = provide additional details or run further tests if needed. Thank you! -John From nobody Fri Sep 26 05:39:45 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cXztk6dxQz69FFH for ; Fri, 26 Sep 2025 05:39:58 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10626.protonmail.ch (mail-10626.protonmail.ch [79.135.106.26]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cXztg6yY9z3qK9 for ; Fri, 26 Sep 2025 05:39:55 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=onYYkRWO; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.26 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758865192; x=1759124392; bh=pEwQI5pp2btL8QPNYF3ctTJscpRXfy5i7wDmx+TTajk=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=onYYkRWOKR26AoYIaKLef5/D3oS/6z1z9aqjx8GgL04srBgqEuVsOhoaS6/3vhCqm bi0C2oO1VTklP1NwIV4BBi1UQLSXxY9QDCQTZDmh66yFxea4sRteknE3oMRJM6ayJ7 qgKwgfkW9i++VifOdAIYXLxnOI7o8gpIIn6ZSExRbM77GIMARKrdLypPvCPdTh54ca gFtvpHaZ2kIJE6qaOywnDBXYElEDSdHAy5GlJBJ9kylwJhPAjqfuRuGy7zPrxDW9S9 lM9t1gBhpFiaWZWaz5isWW9lg8bSaUWjCVe9VNcQAEBeQQnEwVdRb2BqfvlQxAQU2x zYad4x01Z5EgA== Date: Fri, 26 Sep 2025 05:39:45 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 526f3dbd2c6d0fb2b7b64ab5c5c76bc8c996e406 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.39 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.26:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[jon]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.26:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cXztg6yY9z3qK9 I'm continuing my investigation of why I don't see the "Shutting down" mess= age when I run "shutdown -r now" but I do if I do "service vm_gaming stop".= I'm wondering if it's actually gracefully shutting down but I just don't s= ee my monitor update because things are happening quickly. There was 1-2 ti= mes I did get the "shutdown -r now" to cause my vm to show me that screen, = but it doesn't show it to me every time. So far I haven't gotten any Window= s disk corruption or anything, but just not confident about the graceful sh= utdown. I implemented a "status" function to see if maybe it's some early terminati= on logic that's causing it not to run (or exit too early), and I've also re= placed my "sleep 10" with a proper while loop that will check the existence= of the child pid. Once the child pid no longer exists, then execution cont= inues in the stop function. The script currently looks like this: #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable vm_path=3D"/atlantis/vms/gaming" vm_name=3D"gaming" start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" status_cmd=3D"${name}_status" pidfile=3D"/var/run/${name}.child.pid" vm_gaming_start() { =09echo "starting gaming vm ..." =09daemon \ =09=09-o /var/log/${name}.log \ =09=09-p "${pidfile}" \ =09=09bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ =09=09-s 0,hostbridge \ =09=09-s 1,nvme,${vm_path}/disk0.img \ =09=09-s 3:0,passthru,3/0/0 \ =09=09-s 3:1,passthru,3/0/1 \ =09=09-s 13:0,passthru,13/0/0 \ =09=09-s 30,xhci,tablet \ =09=09-s 31,lpc \ =09=09-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu = \ =09=09-o console=3Dstdio \ =09=09${vm_name} } vm_gaming_stop() { =09echo "stopping gaming vm ..." =09# Send SIGTERM twice to make sure Windows listens to =09# the ACPI shutdown signal. =09pid=3D"$(cat ${pidfile})" =09kill ${pid} =09kill ${pid} =09# Wait a bit for the guest to shutdown properly before =09# we continue shutting down the host. =09is_pid_running=3D"$(ps ${pid} -o pid=3D)" =09while [ ! -z "${is_pid_running}" ]; do =09=09is_pid_running=3D"$(ps ${pid} -o pid=3D)" =09done =09bhyvectl --vm=3D${vm_name} --destroy =09echo "gaming vm stopped!" } vm_gaming_restart() { =09# NOTE: AMD users will most likely experience the famous =09# AMD Hardware Reset Bug. This means that after you reboot =09# the guest, you most likely won't have video out. If this =09# happens, you'll need to restart the host. Sometimes this =09# command will work for AMD users though. So you can try a =09# restart and see if it works. =09vm_gaming_stop =09vm_gaming_start } vm_gaming_status() { =09if [ -f "${pidfile}" ]; then =09=09echo "$(cat ${pidfile})" =09fi } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_command "$1" From nobody Fri Sep 26 09:41:38 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cY5FZ6DVTz682Jk for ; Fri, 26 Sep 2025 09:41:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cY5FZ5jgbz3Dy5 for ; Fri, 26 Sep 2025 09:41:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758879698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WOFtoZOezqo8Zd9pxRU1lrCJGHeGgCTo8topr9vBE40=; b=okOcfJ8/k+I6sFWq67pbINLEMc6Pg4ky2oG9UoFZg1378PHqJ6rhatP71PB10YdDn2OtVz 7ejBbKX0LKrETwG1TpCsBBqOC927P8qcsZSmX8EVlK78AmApsW0isx31Wj3ElSwC1o+XMM SnkIP3WKCr/H13Py8pxeVDZWATNnl3XAflD7imqsvxSUiQt8hOIripeapKvy/NH7di2I1t jzdw46ecxzkNoKeRcw2C5e1qHRV9gvKGUZPhlnnTUk+K1svWobVYT2DY7+nZWWlJoTCbD4 HZCagicqPNjq2YGL75JqGj1YdEAItaUos/jZbq6PcY1WwnGlQs1HFm+DgKZqAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758879698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WOFtoZOezqo8Zd9pxRU1lrCJGHeGgCTo8topr9vBE40=; b=xcnI3jzeQcepoaq4qpdLbSrgNGKa3KFocJjU3BGa6TGsVVzJF/tjr4xpVc5lUkWlUsqFoI ujIHP9YXloepOGIHDLI/g59GCYFcwvJmcWdlwyR6ff4g7RSaYxXlxPsZV7ppHIhTDBUqLO xMlmc4INrJXVYsj15nmwlJCZSBZZOFrWK9Dtaa3xa91DMtBDrvYuKSnM38X0chKfDy09Mp 48jnzC+0mFc2R78hQ07Bj4AEct3slhsm6r+fI2WXQyvpknS7wJeCI4P+URysbPbip/SD2u dtljTrGfLlzvzU+x/R0BjH21v9TIygLlpjV5FSIMMRXiLEQPiU2X9vAFwgQ2qA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758879698; a=rsa-sha256; cv=none; b=w4d3q81g/Qd55TdcujhtNMals1A1OdhgOEB6jtXHYhWLIjqrypXHst5q9NabScFiasm1UV kBqhnxXaPSHULuM67l/ZpJP61tbh0EX9HxTAE5mEKiq4XgNf6ZMIsSrSLIVDcp1UsdOa4B tQ5K+K1gzoqIN1XvqQWZoiwnyGHD9kVwmgZ5DI1YHbk/TlSo7l/BV5V24vH4k+rX5rWrgr lnMid6/n9Y5JsNn54qEvcNHW0qOaiOGCdi+vGBE5I13HZiuoo5k/iNJuNVWK2iCoYSOEuJ PgA/c5Uu9gOH7K4bZrQPwq1tbMOWYTBHhdxI6B4kD0iKJmfdnrQb1J+dhpDszA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cY5FZ5B7fzjRb for ; Fri, 26 Sep 2025 09:41:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 58Q9fcVo009807 for ; Fri, 26 Sep 2025 09:41:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 58Q9fcSD009806 for virtualization@FreeBSD.org; Fri, 26 Sep 2025 09:41:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 288848] bhyve GPU passthru for NVIDIA not working Date: Fri, 26 Sep 2025 09:41:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 14.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: corvink@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D288848 Corvin K=C3=B6hne changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|New |Closed --- Comment #7 from Corvin K=C3=B6hne --- The fix was committed to CURRENT, stable/15 and stable/14 [1], so all future FreeBSD will include it. Btw. note that QEMU has some additional quirks for NVIDIA GPUs in [2]. So, = if some additional issues occur, we could try to port those to bhyve. For now,= I'm closing this bug, feel free to open a new bug for additional issues. [1] https://reviews.freebsd.org/D51892 [2] https://elixir.bootlin.com/qemu/v10.1.0/source/hw/vfio/pci-quirks.c#L482-L4= 84 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 26 10:38:29 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cY6W96DlBz686hd for ; Fri, 26 Sep 2025 10:38:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cY6W95j97z3KnW for ; Fri, 26 Sep 2025 10:38:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758883109; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qklfx6HQ03PJ5qxZzkixhLjxU+gw4oG9cjjKWQQXdUo=; b=CIkZc6Cn3QO+WMdHlUMJ79Zcg4/cA260PHm0iZdstYsdkqrRb95rkhi9ukjoXkJHk2DLoM Vl6Ks8e6X0RtTHKJv8pXEhFsffeZrLS+516YrRG9PME4S4L6KTvXgBLPxc0pPicu6VmcyU 9L2c9vcig+UGjEL70uXTtA70wPoE3VKvDQ9kF7+oP7xQuwz83VuOSj0P19WmW28jNPzYeH Q0rV3qxZP9ExDPqmgxxuA3CRM8ST6p+DwDbB0mjjqEAk24tiHe5Pqyhhq+VavX0KDNTa3u J9OCsMieZau5olRHAacjiO2Lcyvf73TpQPubq8qtrcaBK8B1WMacLc43Hw/tXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758883109; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qklfx6HQ03PJ5qxZzkixhLjxU+gw4oG9cjjKWQQXdUo=; b=VJvV6Bj3ACcqF8JDCYFzzS58tohMkZ4W5kHdqcaTDPv7YjORF27Wi5q5flebF820SVe9zt qcS2YJdHaUDD0PZ6AhiOCnfX8H0/DyJBKU8J4shZWLo3EH49zKC0Tno8gUWw7o/QnJIEZO tMhRWtSxV/7oiJQ5IGaJM9dJHeFYDA30ewxL4dPcmFDaBRzdqaES9n3uDvt6I4KI6MT0WF 3eBpXSfm0/eOJAYDtWj6OibzwomxqlyEb1T19/PcTbyPTauMixtUTLYrYlim0LpjAot446 VO6rOTQmb0BIKJnfvFas42FWAFXqajt5M2NgxR+Z6JjIvEv1UIwpVkj9TKZTYg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758883109; a=rsa-sha256; cv=none; b=fKmiNStSpf/5ApD1MNff5a0qPbpukY3rbkvJ0bbb7OkNV4xqaIUNCqS/CNIGcmYHuKmFMp 3t8xaD5FP5NIc9Go+fvlB+wz4rHAr1F1M+HwpqyUaHJsdG7rPbtyscPEuKmkWqV+95saJ7 j5mrrbx0r8EAsPKdONIEE1T4HOzkhbnGnZv2urjFSGcniZOOAW73+ZkHNyyUjY4gf2cOIz F88Pu5ruQ0eh29smy47O7TIgOrfGmaU4aggaxWakHDebJdShFNGKtYxBVsPvSJrZagsf5a LiEtHsYysEj6aG9kQ91zfSlkaweQ/pxVnZKBT1XvGk/Pd0I+VghZJoWZweIh0Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cY6W954bxzk45 for ; Fri, 26 Sep 2025 10:38:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 58QAcTps096251 for ; Fri, 26 Sep 2025 10:38:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 58QAcTxP096250 for virtualization@FreeBSD.org; Fri, 26 Sep 2025 10:38:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 288848] bhyve GPU passthru for NVIDIA not working Date: Fri, 26 Sep 2025 10:38:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 14.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: marietto2008@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D288848 mario felicioni changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marietto2008@gmail.com --- Comment #8 from mario felicioni --- When the error mentioned in this bug occurs ? I ask this because I'm not se= eing it when I pass my Geforce RTX 2080 ti to a Linux or Windows VM... --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Sep 26 18:47:15 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cYKMY5YKlz68jSf; Fri, 26 Sep 2025 18:47:37 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cYKMX5VbVz45cs; Fri, 26 Sep 2025 18:47:36 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (161.dallas-19rh15rt.tx.dial-access.att.net [12.74.217.161]) (authenticated bits=0) by webmail5.jnielsen.net (8.18.1/8.18.1) with ESMTPSA id 58QIlKcr051144 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 26 Sep 2025 12:47:23 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host 161.dallas-19rh15rt.tx.dial-access.att.net [12.74.217.161] claimed to be smtpclient.apple Content-Type: multipart/alternative; boundary=Apple-Mail-4385A15C-5568-4AE1-A3D7-1290F071AE10 Content-Transfer-Encoding: 7bit From: John Nielsen List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum Date: Fri, 26 Sep 2025 12:47:15 -0600 Message-Id: References: <1500781860.3093.1758874136857@localhost> Cc: Michael Tuexen , np@freebsd.org, net@freebsd.org, virtualization@freebsd.org In-Reply-To: <1500781860.3093.1758874136857@localhost> To: Ronald Klop X-Mailer: iPhone Mail (22G100) X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6364, ipnet:2607:f170:30::/44, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cYKMX5VbVz45cs --Apple-Mail-4385A15C-5568-4AE1-A3D7-1290F071AE10 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > On Sep 26, 2025, at 2:09=E2=80=AFAM, Ronald Klop wr= ote: >=20 > This sounds very similar to the scenario that is fixed in 16-CURRENT and 1= 5 with delayed checksum in a virtual environment. >=20 > These are some of the commits around this, but there are more commits invo= lved. > https://cgit.freebsd.org/src/commit/?id=3D39d4094173f9a49ff52f5f4408e4dbd5= d6ef0409 > https://cgit.freebsd.org/src/commit/?id=3Dbcb298fa9e23c1192c5707086a67d3b3= 96186abc >=20 > I also remember bug reports about this case, but don't have the link avail= abie. Thank your for the response! I plan to try updating to 15 soon but I=E2=80=99= m not sure those commits are relevant. The first is about epair which I am n= ot using and the second is for a bug with routing under KVM using virtio net= working IIRC. JN= --Apple-Mail-4385A15C-5568-4AE1-A3D7-1290F071AE10 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Sep 26, 2025, at 2:09=E2=80=AFAM, Ronald Klop <= ronald-lists@klop.ws> wrote:

This sounds very similar to the scenario that is fixed in 16-CURRENT and 15 w= ith delayed checksum in a virtual environment.

These are some of the commits around this, but there are more commits involv= ed.
https://cgit.freebsd.org/src/commit/?id=3D39d4094173f9a49= ff52f5f4408e4dbd5d6ef0409
https://cgit.freebsd.org/src/commit/?id=3Dbcb298fa9e23c11= 92c5707086a67d3b396186abc

I also remember bug reports about this case, but don't have the link availab= ie.

Thank your for the response! I plan to tr= y updating to 15 soon but I=E2=80=99m not sure those commits are relevant. T= he first is about epair which I am not using and the second is for a bug wit= h routing under KVM using virtio networking IIRC.

J= N
= --Apple-Mail-4385A15C-5568-4AE1-A3D7-1290F071AE10-- From nobody Fri Sep 26 18:52:41 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cYKTV3pf5z68jqN; Fri, 26 Sep 2025 18:52:46 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cYKTV2cTFz46m3; Fri, 26 Sep 2025 18:52:46 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (161.dallas-19rh15rt.tx.dial-access.att.net [12.74.217.161]) (authenticated bits=0) by webmail5.jnielsen.net (8.18.1/8.18.1) with ESMTPSA id 58QIqgFH051227 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 26 Sep 2025 12:52:45 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host 161.dallas-19rh15rt.tx.dial-access.att.net [12.74.217.161] claimed to be smtpclient.apple Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: John Nielsen List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum Date: Fri, 26 Sep 2025 12:52:41 -0600 Message-Id: References: Cc: net@freebsd.org, virtualization@freebsd.org, np@freebsd.org In-Reply-To: To: Michael Tuexen X-Mailer: iPhone Mail (22G100) X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6364, ipnet:2607:f170:30::/44, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cYKTV2cTFz46m3 > On Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen wrote: > =EF=BB=BF >>=20 >> On 26. Sep 2025, at 02:58, John Nielsen wrote: >>=20 >> Not sure if this is a known issue or even an issue on the FreeBSD side bu= t decided to ask here first. I=E2=80=99m happy to put in a bug report if app= ropriate. >>=20 >> I have a hypervisor machine running Arch Linux with KVM, Qemu and libvirt= d. The machine has a Chelsio T520-CR adapter. I recently began passing throu= gh virtual functions of the NIC to several of the guests I run on the hyperv= isor. One of the guests runs Windows 11, and the change was seamless. Two of= the guests are running FreeBSD (14.3 or so). On each of them the VFs were r= eadily identified and configured (using DHCP in one case), and ping and ARP a= ppeared to work fine. However, TCP and UDP traffic to the guests never recei= ved a response. After some head-scratching and troubleshooting I discovered t= hat running =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed tra= ffic to flow as expected. >=20 > I don't have access to such a network card. Just to be clear: you are runn= ing the =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D command inside the guest ru= nning FreeBSD, right? Yes. > What is the peer, when you mention TCP and UDP do not work? Is it the host= running Linux? Is it another VM? Is it some external host? My laptop on the same subnet primarily, but. Also tested from another physic= al machine running FreeBSD. >> As workarounds go that isn=E2=80=99t terrible, but I was surprised that t= he FreeBSD guests required this adjustment while the Windows one did not. >>=20 >> As a point of reference I have another hypervisor machine that runs FreeB= SD 14-STABLE. It also has a > So the host runs FreeBSD and you use bhyve to run some VM? Yes >> Chelsio T520-CR adapter. FreeBSD guests on that machine do not require =E2= =80=9C-rxcsum=E2=80=9D in order to use the Chelsio virtual functions. > The guests run FreeBSD-STABLE or Freebsd 14.3? > What is the peer, when you mention TCP and UDP do not work? Is it the host= running Linux? Is it another VM? Is it some external host? FreeBSD 14-STABLE circa 14.3 Multiple off-box same-subnet peers in this scenario as well including my lap= top. I=E2=80=99m aware that VF-to-PF communication is a special case. JN= From nobody Sat Sep 27 16:26:54 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cYtC82jZcz68KmM; Sat, 27 Sep 2025 16:27:16 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cYtC76K73z4NYt; Sat, 27 Sep 2025 16:27:15 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple ([IPv6:2607:fa18:e2fe:2d6a:c9d8:f1a0:f9f4:9c10]) (authenticated bits=0) by webmail5.jnielsen.net (8.18.1/8.18.1) with ESMTPSA id 58RGR46A069325 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 27 Sep 2025 10:27:07 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host [IPv6:2607:fa18:e2fe:2d6a:c9d8:f1a0:f9f4:9c10] claimed to be smtpclient.apple From: John Nielsen Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum Date: Sat, 27 Sep 2025 10:26:54 -0600 In-Reply-To: Cc: net@freebsd.org, virtualization@freebsd.org, np@freebsd.org To: Michael Tuexen References: X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cYtC76K73z4NYt --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 26, 2025, at 3:44=E2=80=AFPM, Michael Tuexen = wrote: >=20 >> On 26. Sep 2025, at 20:52, John Nielsen wrote: >>=20 >>> On Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = wrote: >>> =EF=BB=BF >>>>=20 >>>> On 26. Sep 2025, at 02:58, John Nielsen wrote: >>>>=20 >>>> Not sure if this is a known issue or even an issue on the FreeBSD = side but decided to ask here first. I=E2=80=99m happy to put in a bug = report if appropriate. >>>>=20 >>>> I have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected. >>>=20 >>> I don't have access to such a network card. Just to be clear: you = are running the =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D command inside = the guest running FreeBSD, right? >>=20 >> Yes. >>=20 >>> What is the peer, when you mention TCP and UDP do not work? Is it = the host running Linux? Is it another VM? Is it some external host? >>=20 >> My laptop on the same subnet primarily, but. Also tested from another = physical machine running FreeBSD. > OK. That does not seem to be related to what I initially thought. >=20 > Could you run > tcpdump -i outgoing_interface -w laptop.pcap > on your laptop and > tcpdump -i cxlv0 -w vm.pcap > on your vm at the same time and try to do some TCP based = communication. > Maybe two times, one time with ifconfig cxlv0 rxcsum and one time with = ifconfig cxlv0 -rxcsum. >=20 > If you are fine with doing the measurements, you can send the .pcap = files > to tuexen@freebsd.org = . >=20 > At least I would like to understand what is going on. Thank you, I sent you those packet captures under separate cover. =46rom what I can tell so far the VM does receive incoming packets even = when rxcsum is enabled, and the checksums appear correct at least = according to tcpdump. But something prevents the packets from being = processed or sent up the stack. The VM=E2=80=99s sshd never generates a = SYN/ACK in response to a connection attempt. A DNS query from the VM = looks like it gets a response on the wire but the response doesn=E2=80=99t= make it to the process doing the query. -John --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Sep 26, 2025, = at 3:44=E2=80=AFPM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote:

On 26. Sep 2025, at 20:52, John Nielsen = <lists@jnielsen.net> wrote:

On = Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote:
=EF=BB=BF

On 26. Sep 2025, at 02:58, John Nielsen = <lists@jnielsen.net> wrote:

Not sure if this is a known = issue or even an issue on the FreeBSD side but decided to ask here = first. I=E2=80=99m happy to put in a bug report if appropriate.

I = have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected.

I don't have access to such a = network card. Just to be clear: you are running the =E2=80=9Cifconfig = cxlv0 -rxcsum=E2=80=9D command inside the guest running FreeBSD, = right?

Yes.

What is = the peer, when you mention TCP and UDP do not work? Is it the host = running Linux? Is it another VM? Is it some external = host?

My laptop on the same subnet primarily, but. = Also tested from another physical machine running = FreeBSD.
OK. = That does not seem to be related to what I initially thought.

Could you run
tcpdump = -i outgoing_interface -w laptop.pcap
on your = laptop and
tcpdump -i cxlv0 -w = vm.pcap
on your vm at the same = time and try to do some TCP based communication.
Maybe two times, one time with ifconfig = cxlv0 rxcsum and one time with ifconfig cxlv0 -rxcsum.

If you are fine with doing the = measurements, you can send the .pcap files
to tuexen@freebsd.org <mailto:tuexen@freebsd.org>.

At = least I would like to understand what is going on.

Thank you, I sent you those = packet captures under separate cover.

=46rom = what I can tell so far the VM does receive incoming packets even when = rxcsum is enabled, and the checksums appear correct at least according = to tcpdump. But something prevents the packets from being processed or = sent up the stack. The VM=E2=80=99s sshd never generates a SYN/ACK in = response to a connection attempt. A DNS query from the VM looks like it = gets a response on the wire but the response doesn=E2=80=99t make it to = the process doing the = query.

-John

= --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627-- From nobody Sun Sep 28 15:46:30 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cZTFr1clYz68sSZ for ; Sun, 28 Sep 2025 15:46:40 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-24420.protonmail.ch (mail-24420.protonmail.ch [109.224.244.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cZTFn2tKCz3tN5 for ; Sun, 28 Sep 2025 15:46:36 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=fi2mW70G; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 109.224.244.20 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1759074392; x=1759333592; bh=sTAcPxgTmv5oKZMvb6MdOQrqii3YdavefsoIWkpkS2s=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=fi2mW70GetEeBvf7tAgKPzgToBrCrWil9QLI38RVkslLb6TjTGq/A7YfL1yPRCoVa Nl8KqYOOfsUoRcNrVJd4hSTSpp8SH+peDcEfXRNiQsw174pAQTDx7GgWyvRCEMTVbe 1FW/EkHkAfqbgXFqcaeg2dKFg6P24duG5GJ12ENksC2n5djcRG74mVZ3ZDfdsQMIja 0ku6b5BMOwJkjtg5S7117xXuF1EqGKPnjJXI889QhswgNfXfJXWdIeI9QoWbLSvMsf r1KBvjRIMeHBXrhRp1gJpycZRNkdbQ24kbnAI3BjGfkuXdmqgSN9tf8KcXekSlAIQ+ YXeqNHl60tL4A== Date: Sun, 28 Sep 2025 15:46:30 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: =?us-ascii?Q?<6CV-OY6BcErrWRit9jSpi6fWsYBG3E=5FZ3u6eTLPcz6foPAZV1gQpZYaZTR7JA=5F1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=3D@xyinn.org>_<77221679b528434788d667441d1a32a2@userve.net>_<44d9286c-0866-4805-827d-5cb91511800b@tilda.center>__<3HATaLlOogPiMZWbxSnWPCnJC11=5FunXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8=5F5xCLw=3D@xyinn.org>_<7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr=5FiAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=3D@xyinn.org>_<9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org>_<-rzPcOUnhEy7jFcu7cF7V=5FkjAVPV=5F2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd=5FMWIlH4QaMGU-8l2nXKeOY-Eo=3D@xyinn.org>__?= Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 40800ea8fc35fc60338a4f58e2ef16c30dea4936 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[109.224.244.20:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:109.224.244.0/24]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[xyinn.org:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_EQ_ADDR_ALL(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[jon]; ASN(0.00)[asn:62371, ipnet:109.224.244.0/24, country:CH]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[109.224.244.20:from] X-Rspamd-Queue-Id: 4cZTFn2tKCz3tN5 Hey all, I hope everyone is doing well. I wanted to give some final updates= since I feel I'm basically done with my research, set up, and testing. I'm= pleased to say that I'm officially calling my setup as "stable". For the p= ast week, I've been testing the VM by either playing some games on it for a= few hours, or leaving the VM on idle, and so far I haven't really experien= ced any VM crashes. The games seem to be working smoothly when I'm playing = them, and I don't get any sudden VM crashes. I left the VM on idle for the = past 1d20h and no issues. I wanted to finish working on my rc.d start up/sh= utdown script so I had to bring the VM down to do that :P. There were some = instances where the VM wouldn't really display any output and would effecti= vely be frozen when I rebooted the machine multiple consecutive times durin= g my testing, so much so that doing a "service vm_gaming restart" (or stop)= would just leave the script at the "Waiting for PIDs" message. In this cas= e I had to do a "bhyvectl --vm=3Dgaming --force-poweroff" to kill it. The s= ubsequent start of the VM then seemed to work correctly and video was worki= ng again. This doesn't happen all the time but it could happen so I wanted = to mention that for the record. Once the VM is up, it's pretty stable. I do= see the windows start up screen always say "Getting devices ready" and the= n it proceeds to complete the boot with no issues. Not sure what that's abo= ut but there are no negative consequences. In regards to my research regarding the graceful shutdown, given that throu= ghout all of many days of testing, my VM hasn't gotten any disk corruption,= I believe that the KEYWORD: shutdown (or KEYWORD: nojail shutdown) have be= en working correctly. When doing my testing, I do see that if I do a "servi= ce vm_gaming stop" explicitly, I see Windows 10 showing the light blue scre= en with the "Shutting down" message. When I do "shutdown -r now" I sometime= s see the same light blue screen, but usually I don't see the screen, what = I do see though is that if I move my mouse cursor to the middle of the scre= en, once I trigger the shutdown, I would see the light blue circle/loop aro= und the mouse cursor start to spin. This happens every single time I run th= e "shutdown -r now" command. So this means that the shutdown signal is prop= erly being sent, and my theory is that the GPU/OS is just not reflecting th= at screen update due to the speed at which things are happening. I also do = see that the time it takes to reboot after the shutdown -r now is called ta= kes a little bit of time, which I believe is also partly the time it takes = for the system to wait for the bhyve vm to shutdown and for resource to pos= sibly be cleaned (bhyvectl ... destroy). So I believe I'm done with the mai= n research, experimentation, and testing. I've updated my blog post as well= , and below you can find my updated rc.d script. I saw Corvin recently pus= hed some updates to CURRENT to get NVIDIA passthrough working (PIN related = work). I do have my Razer Blade 15" (2020) that has an Intel proc and NVIDI= A GTX 2060. In the future I may try and install FreeBSD on there to test th= e Intel/NVIDIA side of things, but that's not something I'm planning on doi= ng any time soon. Take care all and I hope all of this helps y'all have some fun and relax. S= tay safe, Jonathan! ---- updated script #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN FILESYSTEMS # KEYWORD: shutdown . /etc/rc.subr name=3D"vm_gaming" rcvar=3D"vm_gaming_enable" start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" stop_postcmd=3D"${name}_poststop" restart_cmd=3D"${name}_restart" status_cmd=3D"${name}_status" pidfile=3D"/var/run/${name}.pid" vm_path=3D"/atlantis/vms/gaming" vm_name=3D"gaming" load_rc_config $name : ${vm_gaming_enable:=3D"NO"} vm_gaming_start() { =09daemon \ =09=09-o /var/log/${name}.log \ =09=09-p "${pidfile}" \ =09=09bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ =09=09-s 0,hostbridge \ =09=09-s 1,nvme,${vm_path}/disk0.img \ =09=09-s 3:0,passthru,3/0/0 \ =09=09-s 3:1,passthru,3/0/1 \ =09=09-s 13:0,passthru,13/0/0 \ =09=09-s 30,xhci,tablet \ =09=09-s 31,lpc \ =09=09-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu = \ =09=09-o console=3Dstdio \ =09=09${vm_name} } vm_gaming_stop() { =09if [ ! -f "${pidfile}" ]; then =09=09echo "the gaming vm isn't running!" =09=09return =09fi =09# Send SIGTERM twice to make sure Windows listens to =09# the ACPI shutdown signal. =09pid=3D"$(cat ${pidfile})" =09kill "${pid}" =09kill "${pid}" =09# Wait a bit for the guest to shutdown properly before =09# we continue shutting down the host. =09wait_for_pids "${pid}" } vm_gaming_poststop() { =09bhyvectl --vm=3D"${vm_name}" --destroy } vm_gaming_restart() { =09# NOTE: AMD users will most likely experience the famous =09# AMD Hardware Reset Bug. This means that after you reboot =09# the guest, you most likely won't have video out. If this =09# happens, you'll need to restart the host. Sometimes this =09# command will work for AMD users though. So you can try a =09# restart and see if it works. =09vm_gaming_stop =09vm_gaming_start } vm_gaming_status() { =09if [ -f "${pidfile}" ]; then =09=09echo "$(cat ${pidfile})" =09fi } run_rc_command "$1"