From owner-freebsd-virtualization@freebsd.org Wed Feb 27 06:32:14 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2339B1512AEE for ; Wed, 27 Feb 2019 06:32:14 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0208B8FBD8 for ; Wed, 27 Feb 2019 06:32:12 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x1R6W8hg049420; Tue, 26 Feb 2019 22:32:08 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id x1R6W8ec049419; Tue, 26 Feb 2019 22:32:08 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201902270632.x1R6W8ec049419@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve PCIe passthrough on AMD Epyc In-Reply-To: <2E994604-BD27-464D-9D8C-14FF3C987D54@dragondata.com> To: Kevin Day Date: Tue, 26 Feb 2019 22:32:08 -0800 (PST) CC: freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 0208B8FBD8 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [1.96 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.50)[0.500,0]; NEURAL_HAM_LONG(-0.01)[-0.012,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.58)[0.575,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: pdx.rh.CN85.dnsmgr.net]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.01)[ip: (0.06), ipnet: 69.59.192.0/19(0.03), asn: 13868(0.01), country: US(-0.07)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2019 06:32:14 -0000 > I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now? No, this should not work now. > I have the device blocked off in loader.conf, and added hw.vmm.amdvi.enable="1". After doing so this is how it appears in pciconf: > > ppt0@pci0:97:0:0: class=0x030000 card=0x27173842 chip=0x128b10de rev=0xa1 hdr=0x00 > bar [10] = type Memory, range 32, base 0xe0000000, size 16777216, disabled > bar [14] = type Prefetchable Memory, range 64, base 0xfcf0000000, size 134217728, disabled > bar [1c] = type Prefetchable Memory, range 64, base 0xfcf8000000, size 33554432, disabled > bar [24] = type I/O Port, range 32, base 0x3000, size 128, disabled > cap 01[60] = powerspec 3 supports D0 D3 current D0 > cap 05[68] = MSI supports 1 message, 64 bit > cap 10[78] = PCI-Express 2 legacy endpoint max data 256(256) RO NS > link x8(x8) speed 5.0(5.0) > ecap 0002[100] = VC 1 max VC0 > ecap 0004[128] = Power Budgeting 1 > ecap 000b[600] = Vendor 1 ID 1 > > When trying to attach it to a VM with "-s 9:0,passthru,97/0/0" I get this almost immediately after it starts up: > > Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 504. > > Also on the console I'm seeing a lot of messages like: > > ivhd0: Error: completion failed tail:0x1c0, head:0x0. > ivhd0: Dump all the commands: > [CMD0, off:0xff0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD1, off:0x0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD2, off:0x10] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD3, off:0x20] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD4, off:0x30] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD5, off:0x40] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD6, off:0x50] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD7, off:0x60] opcode= 0x2 0x6101 0x0 0x0 > [CMD8, off:0x70] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD9, off:0x80] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD10, off:0x90] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD11, off:0xa0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD12, off:0xb0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD13, off:0xc0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD14, off:0xd0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD15, off:0xe0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD16, off:0xf0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD17, off:0x100] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD18, off:0x110] opcode= 0x2 0x6100 0x0 0x0 > [CMD19, off:0x120] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD20, off:0x130] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD21, off:0x140] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD22, off:0x150] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD23, off:0x160] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD24, off:0x170] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD25, off:0x180] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD26, off:0x190] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD27, off:0x1a0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > [CMD28, off:0x1b0] opcode= 0x1 0x304f1629 0x30 0xa5a5 > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org