From owner-freebsd-virtualization@freebsd.org Mon Nov 13 21:02:13 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDA06DD5B34 for ; Mon, 13 Nov 2017 21:02:13 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f50.google.com (mail-lf0-f50.google.com [209.85.215.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DAE07E1EF for ; Mon, 13 Nov 2017 21:02:13 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f50.google.com with SMTP id a132so19912144lfa.7 for ; Mon, 13 Nov 2017 13:02:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=z15i16mZU0b8r+cRo0OVMEv5Nfj8fi718LG/3fGV3Qs=; b=eiiVRZ3CB9W1X+zMZBjiF5BninchLK+0bQ2Bg2X8u9rxIk8gw3UPajN9lUvFidy9T3 tdGv0U1LaIAsmQ87XadfzoUPhY3nozvQa7FptJPDIznuuIj6+WByEmgmhKERA0akVx/C 5KRZjEooAhcnYJycoKYqvxOXR2JNYMcqdeE1k2NfSMaTIEaMAWUiMrUWdYfEH4DvTPPw e3eCYnwWznS4H09n7HWHo48xEACEjjCWtIXRW2ETcbqbE1fvm3viIMZT9zyM16JDEjzc i6HDJ0SS+AnfEb7tcW6QEtc90DODFtL+u3b4Xon3e3jNp4bBD5eX68E2VHhOesBaaVpS LdgA== X-Gm-Message-State: AJaThX4EWuCF6vjuw+ca/TmDEYtV0s6OYq+GA7DuMzrcOkq3RDyhSq/o GVcmx4a8i28kXl2asHx386C/Q44D X-Google-Smtp-Source: AGs4zMYwC+ScCo2JvuJTfIG529P0awSc9IBeU8Vt/UitwkKtUt6y8gzTuVJlAiJxAkQvyKc727FzbA== X-Received: by 10.46.85.208 with SMTP id g77mr1447322lje.114.1510606548486; Mon, 13 Nov 2017 12:55:48 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id o21sm1135408lff.24.2017.11.13.12.55.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Nov 2017 12:55:47 -0800 (PST) Subject: Re: problem with pass-through on amd To: Anish Cc: "freebsd-virtualization@freebsd.org" References: From: Andriy Gapon Message-ID: Date: Mon, 13 Nov 2017 22:55:46 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Nov 2017 21:02:14 -0000 On 13/11/2017 22:41, Anish wrote: > Hi Andriy, > >>What's suspicious is that there are no interrupts either for the > ppt device on the host or the ahci device in the guest. > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled > and can be confirmed by sysctl. > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 2  > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work. I followed your how-to, so I didn't make this mistake :-) $ sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 1 Could it be that AMD-Vi support is broken in the hardware or BIOS? Are there any checks that I could do? Here is what I see in dmesg, but it is not a full dump of IVRS, of course: $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' Table 'IVRS' at 0xbdafaf80 ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD RD890S 00202031 AMD 00000000) AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 ivhd0: on acpi0 ivhd0: Unknown dev entry:0xff ivhd0: Flag:3e ivhd0: max supported paging level:7 restricting to: 4 ivhd0: device supported range [0x0 - 0xb01] ivhd0: device [0xa0 - 0xa0]config:d7 ivhd0: PCI cap 0x10b540f@0x40 feature:1 pci0: at device 0.2 (no driver attached) > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon > wrote: > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller for > the experiment.  It seems like that partially works.  The guest detects the > controller and attaches a driver to it, the messages (reported capabilities, > etc) look identical to those on the host.  But the guest can not detect any > disks behind the controller.  From what I can tell, the guest has no problems > accessing PCI configuration registers and a memory mapped BAR.  Not sure about > the I/O ports.  What's suspicious is that there are no interrupts either for the > ppt device on the host or the ahci device in the guest.  Maybe that is what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on the > host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI device > is behind two PCI-PCI bridges from the main bus.  But I think that that should > not matter and the guest needs to see only the AHCI device and the host handles > the bridges. > > Here are some more messages from the guest: > ahci0: port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=00000123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=00000123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000 > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000 > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org > mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org > " > > -- Andriy Gapon