From owner-freebsd-virtualization@freebsd.org Tue Feb 26 21:32:04 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7743D150491E for ; Tue, 26 Feb 2019 21:32:04 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-it1-x136.google.com (mail-it1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D5C17480D for ; Tue, 26 Feb 2019 21:32:02 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: by mail-it1-x136.google.com with SMTP id z131so6154094itf.5 for ; Tue, 26 Feb 2019 13:32:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=from:content-transfer-encoding:mime-version:subject:message-id:date :to; bh=wcF/wL+maRrXtda86VwJpBGtBumYwy48VHTydg+MznU=; b=iIuBbYCr++G6S7U5MKev79oA2sCm9YeJjMhk21lajxrJw0nmJ3iMRSEFU4fNtTF00B EnQeIC4SA/mG59JFZKgwDob5Vyv71uJz3z/IqALY1fYpXZCrvfIz9BVY9Rf24wwU1+3c FBaALyzatIihzjcQi2cbkAXF/UMiJTW9YymGs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=wcF/wL+maRrXtda86VwJpBGtBumYwy48VHTydg+MznU=; b=MePy2NrcKnXyiZiTvtrTzPJdkoRMMkO89Fw1AzSVMdR84FJiHcrK2O/TMQb3Fc+oSs ufZL8I2NW8ng51C5Kn9tuYjl3tw+Q5dZjfwhGYnc4ovqscjKTkUzPXH4mvy756yxDwXe GvjFKToGI2d5eIfnoISbny5iLTxlViKwvDZmIPr1Hygcj5SNyXxGrnKUWEJu23o/NrmX PY9p8d+JIyaFaJx+/w500LeYXUIem5dRcciu0CWYtFFcNldsYmW+skChljhYmRwOnzT+ zpvIKJMy+9r4t0P40Z1VIQiOXzzatGPF4XVjIVDlngbj/d77EiJDUo9Yc/+cyPfEuQPc v2AA== X-Gm-Message-State: APjAAAXHNZfi8na4rsc/wGrAhCv0BXUBO13hf/SkOASEhzY4ljy0xRFT mDSUdeHAZ81FmBLrDFZZPe6pYayZquY= X-Google-Smtp-Source: AHgI3IZzmVGqtG7jk70nsDBM2yyOKVzI9nbJUztUOIy2iyoQlGaw6nnDIlaM2uk+Qt3X3D6ldE1oyw== X-Received: by 2002:a24:4692:: with SMTP id j140mr3958400itb.170.1551216721976; Tue, 26 Feb 2019 13:32:01 -0800 (PST) Received: from i82.cfv.net (i82.cfv.net. [204.9.51.82]) by smtp.gmail.com with ESMTPSA id i5sm169523itb.3.2019.02.26.13.32.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Feb 2019 13:32:01 -0800 (PST) From: Kevin Day Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: bhyve PCIe passthrough on AMD Epyc Message-Id: <2E994604-BD27-464D-9D8C-14FF3C987D54@dragondata.com> Date: Tue, 26 Feb 2019 15:32:00 -0600 To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 0D5C17480D X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=dragondata.com header.s=google header.b=iIuBbYCr X-Spamd-Result: default: False [-4.78 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[dragondata.com:+]; MX_GOOD(-0.01)[ASPMX3.GOOGLEMAIL.com,ALT2.ASPMX.L.GOOGLE.com,ASPMX.L.GOOGLE.com,ALT1.ASPMX.L.GOOGLE.com,ASPMX5.GOOGLEMAIL.com,ASPMX4.GOOGLEMAIL.com,ASPMX2.GOOGLEMAIL.com]; NEURAL_HAM_SHORT(-0.95)[-0.945,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; SUBJECT_ENDS_SPACES(0.50)[]; R_DKIM_ALLOW(-0.20)[dragondata.com:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[dragondata.com]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.53)[ip: (-7.88), ipnet: 2607:f8b0::/32(-2.68), asn: 15169(-2.01), country: US(-0.07)]; RCVD_IN_DNSWL_NONE(0.00)[6.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Feb 2019 21:32:04 -0000 I'm trying to get PCIe device (video card) passthrough working to a = Windows bhyve VM, should this work now? I have the device blocked off in loader.conf, and added = hw.vmm.amdvi.enable=3D"1". After doing so this is how it appears in = pciconf: ppt0@pci0:97:0:0: class=3D0x030000 card=3D0x27173842 = chip=3D0x128b10de rev=3D0xa1 hdr=3D0x00 bar [10] =3D type Memory, range 32, base 0xe0000000, size = 16777216, disabled bar [14] =3D type Prefetchable Memory, range 64, base = 0xfcf0000000, size 134217728, disabled bar [1c] =3D type Prefetchable Memory, range 64, base = 0xfcf8000000, size 33554432, disabled bar [24] =3D type I/O Port, range 32, base 0x3000, size 128, = disabled cap 01[60] =3D powerspec 3 supports D0 D3 current D0 cap 05[68] =3D MSI supports 1 message, 64 bit cap 10[78] =3D PCI-Express 2 legacy endpoint max data 256(256) RO NS link x8(x8) speed 5.0(5.0) ecap 0002[100] =3D VC 1 max VC0 ecap 0004[128] =3D Power Budgeting 1 ecap 000b[600] =3D Vendor 1 ID 1 When trying to attach it to a VM with "-s 9:0,passthru,97/0/0" I get = this almost immediately after it starts up: Assertion failed: (error =3D=3D 0), function modify_bar_registration, = file /usr/src/usr.sbin/bhyve/pci_emul.c, line 504. Also on the console I'm seeing a lot of messages like: ivhd0: Error: completion failed tail:0x1c0, head:0x0. ivhd0: Dump all the commands: [CMD0, off:0xff0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD1, off:0x0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD2, off:0x10] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD3, off:0x20] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD4, off:0x30] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD5, off:0x40] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD6, off:0x50] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD7, off:0x60] opcode=3D 0x2 0x6101 0x0 0x0 [CMD8, off:0x70] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD9, off:0x80] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD10, off:0x90] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD11, off:0xa0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD12, off:0xb0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD13, off:0xc0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD14, off:0xd0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD15, off:0xe0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD16, off:0xf0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD17, off:0x100] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD18, off:0x110] opcode=3D 0x2 0x6100 0x0 0x0 [CMD19, off:0x120] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD20, off:0x130] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD21, off:0x140] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD22, off:0x150] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD23, off:0x160] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD24, off:0x170] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD25, off:0x180] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD26, off:0x190] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD27, off:0x1a0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5 [CMD28, off:0x1b0] opcode=3D 0x1 0x304f1629 0x30 0xa5a5