From owner-freebsd-virtualization@freebsd.org Thu Feb 28 18:28:44 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E62F1512586 for ; Thu, 28 Feb 2019 18:28:44 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8390807FB for ; Thu, 28 Feb 2019 18:28:42 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-wr1-x432.google.com with SMTP id o17so23110306wrw.3 for ; Thu, 28 Feb 2019 10:28:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lEwAWBmUIJLyLMaqr3HiVnU8CTj7S/bkrxcGDIp3Sko=; b=IPg/cteTVoEdAtJsJXi1Jh020PzTjiJCtOG1jLRGWsBCks/MhWp7QfVqv0Vta76kRh veAM7Q0pkbvmGdhbNDJWn/2LgE+AelLyYPtpOmgITElpI5P9Dhq4dOTjRy4YB4l6Y5UK UA4BWx5TK5RiTKLhhUoE/OrYN+lRYyzMZR9rak7UywQ0FfiIba/+8uAf1XHJwrqq5Kr5 QDyc0Eo053Xtf4gMky3lWa+cgrQuIXGsUHlVJLssQavyTcuAMI0vkK7RvRlb+x23DUEZ HHnclYtS8qPPMZOklDvL35cySfcEI8NT2cNGmPMRYoaFJMsyvP9FCAxVK1n7uzYk6wvC +P4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lEwAWBmUIJLyLMaqr3HiVnU8CTj7S/bkrxcGDIp3Sko=; b=KLTquaSBr3LUpL7m4cSaDJbnArVThhp3O0YYnp3geThMvJ+pr+yBOQCMkQVS3mJFzq 8JyfL2mHJAOVOPxRr0FjWefPiNkXssImAKYtqATdsLqCZjT7P/GOib8XDdyEcyiM4Jar lhU5M7RdiIEW1qtH0ANXokHom0idmuWEbdI1uq6kK604gTr0IGt2x45P3qcQCgahXUdF VwucAgBuoZerNQBemxzhYD27mdZI6iR1yCb66vPoB3/xxFH1cCR9V7QaGMD4sYRGN8Sz WL/W75K2UZ3WEloS5a2nd6Y/LOJojkUo1D9k/+WWMZ+Ac+9Ye0Uue9Epex8fECf1XPut njXw== X-Gm-Message-State: APjAAAUs0Kp14nYaeKRTQrPnTBnCMRx7QSDcLKMR+EOLSr38Kxl/7tRF Cb3M+CYwIYKqm0zK0zcvB1JvdHvYtI6GVSeMEPwUvjBldvE= X-Google-Smtp-Source: APXvYqzpmfjKEHKo5myZgQwWbaF2J8ReGPgXFEFeo5Rx4KVmDrviCmMUOZkstLX52V3pHE5/e5keYlMk586t4IYsyxY= X-Received: by 2002:adf:f5d0:: with SMTP id k16mr489529wrp.325.1551378521636; Thu, 28 Feb 2019 10:28:41 -0800 (PST) MIME-Version: 1.0 References: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com> In-Reply-To: <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com> From: Anish Date: Thu, 28 Feb 2019 10:28:30 -0800 Message-ID: Subject: Re: bhyve PCIe passthrough on AMD Epyc To: Kevin Day Cc: "Rodney W. Grimes" , "freebsd-virtualization@freebsd.org" X-Rspamd-Queue-Id: E8390807FB X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=IPg/cteT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of akgupt3@gmail.com designates 2a00:1450:4864:20::432 as permitted sender) smtp.mailfrom=akgupt3@gmail.com X-Spamd-Result: default: False [-6.82 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.83)[ip: (-9.77), ipnet: 2a00:1450::/32(-2.28), asn: 15169(-2.03), country: US(-0.07)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2019 18:28:44 -0000 Hi Kevin, >I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested. I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc systems. >CPU0: local APIC error 0x40 This is concerning, as per AMD docs, bit6 is for illegal vector which indicate CPU is getting invalid interrupts. Can you share hw.vmm sysctl output? -Anish On Thu, Feb 28, 2019 at 9:59 AM Kevin Day wrote: > > > > On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes < > freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: > > > >>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes < > freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: > >>> > >>>> I'm trying to get PCIe device (video card) passthrough working to a > Windows bhyve VM, should this work now? > >>> > >>> No, this should not work now. > >> > >> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now? > > I do not know what would be special about Epyc so can not say > > for certain. > > I'm just pointing out that it's an AMD Epyc because I know the AMD > IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is > maybe less tested. > > >> Just trying the same thing with a simple PCIe USB card gives me: > >> > >> CPU0: local APIC error 0x40 > >> CPU0: local APIC error 0x40 > >> ivhd0: Error: completion failed tail:0xda0, head:0x0. > >> ivhd0: Dump all the commands: > >> [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0 > >> [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003 > >> [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0 > >> [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0 > >> [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0 > >> [...snip... lots of similar stuff] > >> > >> Then I get a bunch more APIC errors, and the system freezes for about > 60 seconds before recovering. > > > > You do not get any host dmesg errors about unaligned BAR > > ranges or sizes? > > > > Correct. I did with the video card, but not with the xhci USB card. I get > a ton of dmesg errors, and the host system repeatedly freezes until I stop > the VM. (Console is non-responsive, no ICMP replies, etc) > > I didn't actually look at this until now, but Windows actually does see > the USB controller, but it complains that "port reset failed" on any device > I attach to it. > > > Can you get the pciconf -l -b info for the device you passed, something > like: > > # pciconf -l -b uhci0 > > uhci0@pci0:0:26:0: class=0x0c0300 card=0x20f017aa chip=0x29378086 > rev=0x03 hdr=0x00 > > bar [20] = type I/O Port, range 32, base 0x1840, size 32, enabled > > xhci3@pci0:97:0:0: class=0x0c0330 card=0xffffffff chip=0x00151912 > rev=0x02 hdr=0x00 > bar [10] = type Memory, range 64, base 0xe1d00000, size 8192, enabled > cap 01[50] = powerspec 3 supports D0 D3 current D0 > cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message > cap 11[90] = MSI-X supports 8 messages > Table in map 0x10[0x1000], PBA in map 0x10[0x1080] > cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS > link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1) > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0018[150] = LTR 1 > > Also just in case it's relevant, "acpidump" on this system hangs after > allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi > is enabled in the boot dmesg though: > > AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0 > > > > > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" >