From owner-freebsd-virtualization@freebsd.org Thu Feb 28 05:34:28 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E53C215197A6 for ; Thu, 28 Feb 2019 05:34:27 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D1D137772B for ; Thu, 28 Feb 2019 05:34:26 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x1S5YMk9054622; Wed, 27 Feb 2019 21:34:22 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id x1S5YMQS054621; Wed, 27 Feb 2019 21:34:22 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve PCIe passthrough on AMD Epyc In-Reply-To: <8A44404F-C6E5-47BF-82E8-3CF94D5487E9@dragondata.com> To: Kevin Day Date: Wed, 27 Feb 2019 21:34:22 -0800 (PST) CC: freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: D1D137772B X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [2.34 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.92)[0.919,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.23)[0.227,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: pdx.rh.CN85.dnsmgr.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.30)[0.296,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.01)[ip: (0.06), ipnet: 69.59.192.0/19(0.03), asn: 13868(0.01), country: US(-0.07)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2019 05:34:28 -0000 > > On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes wrote: > > > >> I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now? > > > > No, this should not work now. > > Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now? I do not know what would be special about Epyc so can not say for certain. > Just trying the same thing with a simple PCIe USB card gives me: > > CPU0: local APIC error 0x40 > CPU0: local APIC error 0x40 > ivhd0: Error: completion failed tail:0xda0, head:0x0. > ivhd0: Dump all the commands: > [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0 > [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003 > [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0 > [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0 > [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0 > [...snip... lots of similar stuff] > > Then I get a bunch more APIC errors, and the system freezes for about 60 seconds before recovering. You do not get any host dmesg errors about unaligned BAR ranges or sizes? Can you get the pciconf -l -b info for the device you passed, something like: # pciconf -l -b uhci0 uhci0@pci0:0:26:0: class=0x0c0300 card=0x20f017aa chip=0x29378086 rev=0x03 hdr=0x00 bar [20] = type I/O Port, range 32, base 0x1840, size 32, enabled > -- Kevin -- Rod Grimes rgrimes@freebsd.org