From owner-freebsd-virtualization@freebsd.org Mon Nov 13 19:11:01 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 450B6DC1BF0 for ; Mon, 13 Nov 2017 19:11:01 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f45.google.com (mail-lf0-f45.google.com [209.85.215.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DEDFB792B3 for ; Mon, 13 Nov 2017 19:11:00 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f45.google.com with SMTP id 73so5244251lfu.10 for ; Mon, 13 Nov 2017 11:11:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=XcXy2XO1ek6srEf2ZX8xr0WRo4BPVDQ7KIAcKMTtpXI=; b=Lp6b1p4vUvzWjOwqBvsQ3F65I++OvWA5cbxa37EbwpB6i4XuCmVvV7Eg6Kn7NnkT+j Hcis8oFT+dL7ldxpp8HUgCzrMt8g8mCryRneIng4LLsPgNVpAq8e4aOOeplzHwId0CMD IprDHB3NUcMylaHBjOfx7412x1CFgrBQhKZkd8ar9AHwVYO3XVOmhYR/ZSvbSwYMtf/u uAogWsNBrWfjAso8I5Zqk/Ecau0/4MCn+9ulfIOAQhRiT0G/R8negesnQmaUU+LbWSpQ 0xQePUBUNoV58gaprB9UoIVwGc7+EwS5psoc7o5RQ9F+KGxzyYJsIBdbfImhteKhz56H dzWA== X-Gm-Message-State: AJaThX7tpSLaDQxMcFJXI4pGO2r1ypBCZj3k+ioLuzBBq0Zx5Fp4RoTA fI+LgmRzFusPzDEzTI6rw7DXnDFw X-Google-Smtp-Source: AGs4zMaZRiiHzuYBY4S3sF7kVYSLYMR2IHVGFiHUOU0bBRUrKgq+32w069TiKX4fwSP36iEy6zVTNA== X-Received: by 10.46.23.20 with SMTP id l20mr2973846lje.25.1510599771360; Mon, 13 Nov 2017 11:02:51 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id z81sm2973121lff.16.2017.11.13.11.02.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Nov 2017 11:02:50 -0800 (PST) To: freebsd-virtualization@freebsd.org From: Andriy Gapon Subject: problem with pass-through on amd Message-ID: Date: Mon, 13 Nov 2017 21:02:48 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Nov 2017 19:11:01 -0000 I have a FreeBSD guest on a FreeBSD host, both amd64 head. I decided to experiment with PCI pass-through and used an AHCI controller for the experiment. It seems like that partially works. The guest detects the controller and attaches a driver to it, the messages (reported capabilities, etc) look identical to those on the host. But the guest can not detect any disks behind the controller. From what I can tell, the guest has no problems accessing PCI configuration registers and a memory mapped BAR. Not sure about the I/O ports. What's suspicious is that there are no interrupts either for the ppt device on the host or the ahci device in the guest. Maybe that is what causes the failure to see the disks. When the guest driver attaches to the controller I see these messages on the host: ppt0: attempting to allocate 1 MSI vectors (1 supported) ppt0: using IRQ 265 for MSI and in the guest: ahci0: attempting to allocate 1 MSI vectors (1 supported) ahci0: using IRQ 279 for MSI But vmstat -i does not report any interrupts. Just in case, the controller has some complex topology where the AHCI PCI device is behind two PCI-PCI bridges from the main bus. But I think that that should not matter and the guest needs to see only the AHCI device and the host handles the bridges. Here are some more messages from the guest: ahci0: port 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 ahci0: attempting to allocate 1 MSI vectors (1 supported) ahci0: using IRQ 279 for MSI ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports ahcich0: at channel 0 on ahci0 ahcich0: Caps: CPD ESP FBSCP ahcich1: at channel 1 on ahci0 ahcich1: Caps: CPD ESP FBSCP ahcich0: AHCI reset... ahcich0: SATA connect time=100us status=00000123 ahcich0: AHCI reset: device found ahcich1: AHCI reset... ahcich1: SATA connect time=100us status=00000123 ahcich1: AHCI reset: device found ahcich1: AHCI reset: device ready after 100ms ahcich1: Poll timeout on slot 1 port 15 ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000 cmd 00318017 ahcich0: AHCI reset: device ready after 200ms ahcich0: Poll timeout on slot 1 port 15 ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000 cmd 00318017 And some more... Any ideas or suggestions? Or maybe some additional information from me? Thank you! -- Andriy Gapon