From owner-freebsd-virtualization@freebsd.org Mon Nov 13 20:41:12 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17B4FDD518F for ; Mon, 13 Nov 2017 20:41:12 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-qk0-x22e.google.com (mail-qk0-x22e.google.com [IPv6:2607:f8b0:400d:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA5A47D34F; Mon, 13 Nov 2017 20:41:11 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-qk0-x22e.google.com with SMTP id d125so6940723qkc.12; Mon, 13 Nov 2017 12:41:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BUjjeOcYNmy85HowepM8FS3vhcr+14acNr4Eykd353U=; b=GiuCB0aepSCc7g8jgEpVHExNQUhfhU/QvDDPFd0YmT+rZ/4uQxzLMoR8nF0gJ3lm4n D9oY4gOCiUGXzzZxywPGdxTZTkuxqB1RrT9vDPXddrpWNpvK9ZgwvXodEP/DoCLnFRjZ QJT0lsv5jRhffGGO5Dq9uQOO4mUebo7fsAXyU8Wnk8NJDUUxYY/bTTvKRfEtOBy6Ap0Y 7UbZS2uGliRNNp+PYU7L8CO0AlhxMWIUl3+SaNmoPKJzFeLrgIHDXdRaX9hMlxVeVRbE lBibn/hdTxFexfCmMet8hmntzmZr5uL/B0nvDWGGIgpz4zZfgG2299aK56mWepIDcanA Vo5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BUjjeOcYNmy85HowepM8FS3vhcr+14acNr4Eykd353U=; b=DDlZWTmHbWFOVTxbQJYIjgQ1wieyNtVE+ADMmJnTmmZLt8vlDe64JeQRrIinC4hD54 F0Rsdc4gteoW1EmsTEf8hoEyFRr2JwIY6VJzy0Ct+AJudv10252i32boc4wvZOGQjy7Y BBR5CSeLWOEOrXPXc2DLs/0RILHR6wgXLpcQyxSQEh+xiG/O5AWZRWv/Uqr/vVftstO4 mzrXsomTKO8fRCwaYGnUGdSz5139p0R2gRMXIQiV/cs81PUm1jsPsDZAglYgsE/6SF1L KVFIpF7tuSdYtBkf2ti8W4uH2AXhrdxfTcNoypqmeWvDANw2UHpKBL3AmjL3Nwp5sy6M 3w7Q== X-Gm-Message-State: AJaThX5yO+x70QF8xdKb/RN4llk+JADumPM8tIpEdz4VdGuAKnfktVEe IqLZ0JA5v0SBvivJ1V4JM5laCtGtoQOld8lBYr68QQ== X-Google-Smtp-Source: AGs4zMZJEjGHBLIO4gl9jw0cLxyA9i88LQqHWlywnQU2Si3lwznJAI9knlnWbUAZgPCScXQv8ZzM1UZjcs/CM54g5Mk= X-Received: by 10.55.183.134 with SMTP id h128mr15281501qkf.96.1510605670473; Mon, 13 Nov 2017 12:41:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.172.247 with HTTP; Mon, 13 Nov 2017 12:41:10 -0800 (PST) In-Reply-To: References: From: Anish Date: Mon, 13 Nov 2017 12:41:10 -0800 Message-ID: Subject: Re: problem with pass-through on amd To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Nov 2017 20:41:12 -0000 Hi Andriy, >What's suspicious is that there are no interrupts either for the ppt device on the host or the ahci device in the guest. I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled and can be confirmed by sysctl. [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 2 If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work. Regards, Anish On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon wrote: > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller > for > the experiment. It seems like that partially works. The guest detects the > controller and attaches a driver to it, the messages (reported > capabilities, > etc) look identical to those on the host. But the guest can not detect any > disks behind the controller. From what I can tell, the guest has no > problems > accessing PCI configuration registers and a memory mapped BAR. Not sure > about > the I/O ports. What's suspicious is that there are no interrupts either > for the > ppt device on the host or the ahci device in the guest. Maybe that is what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on > the host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI > device > is behind two PCI-PCI bridges from the main bus. But I think that that > should > not matter and the guest needs to see only the AHCI device and the host > handles > the bridges. > > Here are some more messages from the guest: > ahci0: port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=00000123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=00000123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr > 00000000 > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr > 00000000 > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" >