From owner-freebsd-virtualization@freebsd.org Thu Feb 28 17:58:20 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 885351511007 for ; Thu, 28 Feb 2019 17:58:20 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78C5F7705E for ; Thu, 28 Feb 2019 17:58:19 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: by mail-io1-xd33.google.com with SMTP id p196so17284035iod.9 for ; Thu, 28 Feb 2019 09:58:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=aooDo02jrIeTwma6d/outuNA/CPqkrbbqp8lV0j01bk=; b=WtRHbOaFLCW+t15rVHGK3GtZvfXONPWKeu2b6WwfUlghO1PxdIUNm52uGuCxdFKrFE 5ShjmLIjqC0p0Qf07ciQt0zBAf0KSzJQEBFAsCIYE/1+nWRRhbxPavogzNsZFilG6mMY vhRI6Z9rVuHxMxswjWvL5FvRU7iLhPVBMc4F8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=aooDo02jrIeTwma6d/outuNA/CPqkrbbqp8lV0j01bk=; b=EDdbI8nFBJ88ta37nkG8BNSX6t9FA/wzshAtXg7Ndo6XJEAiKekDE7QVdzaivx6ABc RYy7I3BxqQ+7oZiyFHD7Whqbr6GoE3r+efaeT2lWvkEC2R19rjJKOS2DfLMjP3GOJbVC JhxHdO3VDQS9Sr8XtHwsd4R8mEF8KaZyBngq6SjQOvoqHx/t4cCNhOkaZhO68/3V3beI /V2bLH9QxKasWZal/TCFdp8UesbKcpjYnNnekVI3ZSjmWloKfici9L8I3Y1ICPiHJ40o ca/Q0aDzsJTD6Zse+S5TZhYYyZwBqHC+xgZ1z+sSCWoXbWO8ZfVZqmeCe3DSZnbEJOmC ncmg== X-Gm-Message-State: APjAAAVFH/cXbgY/GXP8k+j7Ze3sG4jZuOGK7+bEZNBvD9ZSZCoA2VYF AtF3z4P74Jwc7lxJz+3ryIzoa54u/No= X-Google-Smtp-Source: APXvYqwK/AXpSSB+d2MhUB9b6Rwnz5FJhvB+da2Jed4sL+eWzoYGfO52mVwgnMaGubLUPoF1vePFOA== X-Received: by 2002:a5d:8c98:: with SMTP id g24mr348921ion.103.1551376698364; Thu, 28 Feb 2019 09:58:18 -0800 (PST) Received: from i82.cfv.net (i82.cfv.net. [204.9.51.82]) by smtp.gmail.com with ESMTPSA id z24sm7618505ioh.57.2019.02.28.09.58.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Feb 2019 09:58:17 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: bhyve PCIe passthrough on AMD Epyc From: Kevin Day In-Reply-To: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> Date: Thu, 28 Feb 2019 11:58:16 -0600 Cc: freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com> References: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> To: "Rodney W. Grimes" X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 78C5F7705E X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=dragondata.com header.s=google header.b=WtRHbOaF X-Spamd-Result: default: False [-5.44 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[dragondata.com:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[dragondata.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[dragondata.com:+]; MX_GOOD(-0.01)[cached: ASPMX3.GOOGLEMAIL.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.94)[-0.940,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-2.69)[ip: (-8.66), ipnet: 2607:f8b0::/32(-2.71), asn: 15169(-2.03), country: US(-0.07)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2019 17:58:20 -0000 > On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes = wrote: >=20 >>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes = wrote: >>>=20 >>>> I'm trying to get PCIe device (video card) passthrough working to a = Windows bhyve VM, should this work now? >>>=20 >>> No, this should not work now. >>=20 >> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right = now? > I do not know what would be special about Epyc so can not say > for certain. I'm just pointing out that it's an AMD Epyc because I know the AMD = IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is = maybe less tested. >> Just trying the same thing with a simple PCIe USB card gives me: >>=20 >> CPU0: local APIC error 0x40 >> CPU0: local APIC error 0x40 >> ivhd0: Error: completion failed tail:0xda0, head:0x0. >> ivhd0: Dump all the commands: >> [CMD0, off:0xff0] opcode=3D 0x0 0x0 0x0 0x0 >> [CMD1, off:0x0] opcode=3D 0x3 0x0 0x0 0x7ffffffffffff003 >> [CMD2, off:0x10] opcode=3D 0x0 0x0 0x0 0x0 >> [CMD3, off:0x20] opcode=3D 0x2 0x0 0x0 0x0 >> [CMD4, off:0x30] opcode=3D 0x2 0x2 0x0 0x0 >> [...snip... lots of similar stuff] >>=20 >> Then I get a bunch more APIC errors, and the system freezes for about = 60 seconds before recovering. >=20 > You do not get any host dmesg errors about unaligned BAR > ranges or sizes? >=20 Correct. I did with the video card, but not with the xhci USB card. I = get a ton of dmesg errors, and the host system repeatedly freezes until = I stop the VM. (Console is non-responsive, no ICMP replies, etc) I didn't actually look at this until now, but Windows actually does see = the USB controller, but it complains that "port reset failed" on any = device I attach to it. > Can you get the pciconf -l -b info for the device you passed, = something like: > # pciconf -l -b uhci0 > uhci0@pci0:0:26:0: class=3D0x0c0300 card=3D0x20f017aa = chip=3D0x29378086 rev=3D0x03 hdr=3D0x00 > bar [20] =3D type I/O Port, range 32, base 0x1840, size 32, = enabled xhci3@pci0:97:0:0: class=3D0x0c0330 card=3D0xffffffff = chip=3D0x00151912 rev=3D0x02 hdr=3D0x00 bar [10] =3D type Memory, range 64, base 0xe1d00000, size 8192, = enabled cap 01[50] =3D powerspec 3 supports D0 D3 current D0 cap 05[70] =3D MSI supports 8 messages, 64 bit enabled with 1 = message cap 11[90] =3D MSI-X supports 8 messages Table in map 0x10[0x1000], PBA in map 0x10[0x1080] cap 10[a0] =3D PCI-Express 2 endpoint max data 128(128) NS link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1) ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected ecap 0018[150] =3D LTR 1 Also just in case it's relevant, "acpidump" on this system hangs after = allocating ~3GB of ram and just spins at 100% CPU. I am seeing that = AMD-Vi is enabled in the boot dmesg though: AMD-Vi: IVRS Info VAsize =3D 64 PAsize =3D 48 GVAsize =3D 2 flags:0