From owner-freebsd-virtualization@freebsd.org Tue Nov 14 04:22:08 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A732ACFF095 for ; Tue, 14 Nov 2017 04:22:08 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-qk0-x236.google.com (mail-qk0-x236.google.com [IPv6:2607:f8b0:400d:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 662696D9C7; Tue, 14 Nov 2017 04:22:08 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-qk0-x236.google.com with SMTP id a194so20413948qkc.9; Mon, 13 Nov 2017 20:22:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=M9qqoydr9T0p+HwXsVxsTltBoqxbakTrxxCn6G0lD3M=; b=DDk7dYoZclwKX7cuNw6nRzDJq+lDfF6sgUky7BeNHWrN1XYyJoOAVMtflhcBW7ywjs ipTPKGV9gYHeqqkk0hqrB+05z//G2SxNb98kcms+5mAdgUrgYyfJ/SVoeL0EZlO1DOrB FOTUssJlenZCyHG00Hs8gDyXGGQ3JP/qiEGc4pmLsUoeOxv79Vl9RPJJCWZTwFoohctR JXYJ25xdtZJWSZ6VxukJGJu+kDqZskwGlHfsOhhCqwVY/V9pq091WljV8PPip8X3Gz65 3clQEtvkcXtysimKoaZvbKZoCbqpgu1T7tRdpAqm3FJXGmQBLdlx/ZN64Dujh4VbEDSP iDPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=M9qqoydr9T0p+HwXsVxsTltBoqxbakTrxxCn6G0lD3M=; b=coJinr8iw0RoCr0xkCbXw9g0CCHve/GRp+f/GvnSFsCIsB3uBmdMDEO0l5si9UTC0k m24FFZ4ax/eLDKmir2NzurWflOaPuh1AG4NqsaU5cb049gk7pjLP7kLasazYrrMM9VBM jLlfog2C2E5gpsJaNopan0i/9/QN6bzXLZYptH465FEwmlWbIh4C406lDXGTLu6eQxfA UpD3l+RvEZz+KvCCgTjfvw/un7bwnbG/5KWzm5uHU+RVtwXOptqVQ8URnCyAtJ8Jjfda Gyf9om1fqpD/qh7CZlZxyaC20fZGTOF0E/Uuk+el7Mn2rAsKsqBH7ieigQnl7mRVI2EI oMfQ== X-Gm-Message-State: AJaThX473CO2E6ANTNAPMnMY2ZL/mIxouY38Hc65If7z+/Zmr50fVL3q oXhHmAEdilRCdvr4PVMjRoINyFdQqHA4GO6WRq6GcQ== X-Google-Smtp-Source: AGs4zMbsrm2EGop89i7WSyNLNCW8UYKYQwNZFNEw2xDeTTPVZc9SSoE+3lYhM3RskD80OODoHqF0OQWCH8YvpmlxbIA= X-Received: by 10.233.232.13 with SMTP id a13mr17138146qkg.186.1510633327270; Mon, 13 Nov 2017 20:22:07 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.172.247 with HTTP; Mon, 13 Nov 2017 20:22:06 -0800 (PST) In-Reply-To: References: From: Anish Date: Mon, 13 Nov 2017 20:22:06 -0800 Message-ID: Subject: Re: problem with pass-through on amd To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2017 04:22:08 -0000 Hi Andriy, Newer AMD IOMMU has EFR capability which is missing here, something like: ivhd0: PCI cap 0x190b640f@0x40 feature:19 Is this an old chipset? Also ivhd has fault interrupt enabled which is very helpful in debugging: [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh irq256: ivhd0:fault 0 0 irq257: ivhd1:fault 0 0 Regards, Anish On Mon, Nov 13, 2017 at 12:55 PM, Andriy Gapon wrote: > On 13/11/2017 22:41, Anish wrote: > > Hi Andriy, > > > >>What's suspicious is that there are no interrupts either for the > > ppt device on the host or the ahci device in the guest. > > > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is > disabled > > and can be confirmed by sysctl. > > > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > > hw.vmm.amdvi.domain_id: 3 > > hw.vmm.amdvi.disable_io_fault: 0 > > hw.vmm.amdvi.ptp_level: 4 > > hw.vmm.amdvi.host_ptp: 1 > > hw.vmm.amdvi.enable: 1 > > hw.vmm.amdvi.count: 2 > > > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should > work. > > > I followed your how-to, so I didn't make this mistake :-) > > $ sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 1 > > Could it be that AMD-Vi support is broken in the hardware or BIOS? > Are there any checks that I could do? > > Here is what I see in dmesg, but it is not a full dump of IVRS, of course: > $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' > Table 'IVRS' at 0xbdafaf80 > ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD RD890S 00202031 AMD > 00000000) > AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 > ivhd0: on acpi0 > ivhd0: Unknown dev entry:0xff > ivhd0: Flag:3e > ivhd0: max supported paging level:7 restricting to: 4 > ivhd0: device supported range [0x0 - 0xb01] > ivhd0: device [0xa0 - 0xa0]config:d7 > ivhd0: PCI cap 0x10b540f@0x40 feature:1 > pci0: at device 0.2 (no driver attached) > > > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon > > wrote: > > > > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > > I decided to experiment with PCI pass-through and used an AHCI > controller for > > the experiment. It seems like that partially works. The guest > detects the > > controller and attaches a driver to it, the messages (reported > capabilities, > > etc) look identical to those on the host. But the guest can not > detect any > > disks behind the controller. From what I can tell, the guest has no > problems > > accessing PCI configuration registers and a memory mapped BAR. Not > sure about > > the I/O ports. What's suspicious is that there are no interrupts > either for the > > ppt device on the host or the ahci device in the guest. Maybe that > is what > > causes the failure to see the disks. > > > > When the guest driver attaches to the controller I see these > messages on the > > host: > > ppt0: attempting to allocate 1 MSI vectors (1 supported) > > ppt0: using IRQ 265 for MSI > > and in the guest: > > ahci0: attempting to allocate 1 MSI vectors (1 supported) > > ahci0: using IRQ 279 for MSI > > > > But vmstat -i does not report any interrupts. > > > > Just in case, the controller has some complex topology where the > AHCI PCI device > > is behind two PCI-PCI bridges from the main bus. But I think that > that should > > not matter and the guest needs to see only the AHCI device and the > host handles > > the bridges. > > > > Here are some more messages from the guest: > > ahci0: port > > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef > mem > > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > > ahci0: attempting to allocate 1 MSI vectors (1 supported) > > ahci0: using IRQ 279 for MSI > > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with > FBS > > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA > 2ports > > ahcich0: at channel 0 on ahci0 > > ahcich0: Caps: CPD ESP FBSCP > > ahcich1: at channel 1 on ahci0 > > ahcich1: Caps: CPD ESP FBSCP > > ahcich0: AHCI reset... > > ahcich0: SATA connect time=100us status=00000123 > > ahcich0: AHCI reset: device found > > ahcich1: AHCI reset... > > ahcich1: SATA connect time=100us status=00000123 > > ahcich1: AHCI reset: device found > > ahcich1: AHCI reset: device ready after 100ms > > ahcich1: Poll timeout on slot 1 port 15 > > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 > serr 00000000 > > cmd 00318017 > > ahcich0: AHCI reset: device ready after 200ms > > ahcich0: Poll timeout on slot 1 port 15 > > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 > serr 00000000 > > cmd 00318017 > > And some more... > > > > Any ideas or suggestions? > > Or maybe some additional information from me? > > > > Thank you! > > > > -- > > Andriy Gapon > > _______________________________________________ > > freebsd-virtualization@freebsd.org > > mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > > > To unsubscribe, send any mail to > > "freebsd-virtualization-unsubscribe@freebsd.org > > " > > > > > > > -- > Andriy Gapon >