From owner-freebsd-virtualization@FreeBSD.ORG Tue Sep 27 20:05:40 2011 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A5111065670; Tue, 27 Sep 2011 20:05:40 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 094DB8FC0A; Tue, 27 Sep 2011 20:05:39 +0000 (UTC) Received: by yxk36 with SMTP id 36so7202507yxk.13 for ; Tue, 27 Sep 2011 13:05:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=lEROCE6Okdx2PqhuFNddL3rxxkvSwgusXXW/S+Xwuos=; b=PdaaLMXrhAPK6n6TF2QtoM8JgVchT7/86y4dWimnXiNunlNToSoGMx2c79iiWnUWiI bJaPVXnmBceuhScjeV7Wd+qG0cZmuKnNSFUjf6b3c2gxgl7E1R78ES/RrI2Xb9zkQswU 6nGicDP/NmP08qF7VYDaOuIDCw44obF+vawJY= MIME-Version: 1.0 Received: by 10.236.123.37 with SMTP id u25mr51917240yhh.50.1317152149534; Tue, 27 Sep 2011 12:35:49 -0700 (PDT) Received: by 10.236.108.145 with HTTP; Tue, 27 Sep 2011 12:35:49 -0700 (PDT) In-Reply-To: <4E8168BB.9050802@freebsd.org> References: <4E8168BB.9050802@freebsd.org> Date: Tue, 27 Sep 2011 12:35:49 -0700 Message-ID: From: Neel Natu To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-virtualization@freebsd.org Subject: Re: question on Bhyve X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 20:05:40 -0000 Hi Julian, On Mon, Sep 26, 2011 at 11:10 PM, Julian Elischer wrote: > does anyone know what is needed for a hypervisor to support PCI pass > through? > I can speak about the bhyve implementation of pci passthru but I suspect that other hypervisors do it similarly. BHyVe requires that the platform support nested page tables and has an iommu to be able to support pci passthru. There are two parts to it. The simpler part is "hiding" the pci device that is to be used to passthru. In bhyve we do this by setting the tunable "pptdevs" to a list of pci devices that we may want to assign as passthru devices. This tunable is used by the "blackhole" pci device driver to claim each device in "pptdevs" during probe but then reject it in attach. The net effect of this is leave the pci device un-attached from the host's perspective. For e.g. in /boot/loader.conf: pptdevs="1/0/0 2/0/0 3/0/0" blackhole_load="YES" The second part is a bit more work for the hypervisor. We assign the pci device to a guest with the following command line option to /usr/sbin/bhyve: -s 1,passthru,3/0/0. This says to make the pci device identified by "3/0/0" appear as a passthru pci device on slot 1 of the virtual pci bus. When we assign the device to the guest we need to provide it with passthru access to the device's config space, mmio space, io space. In the other direction we need to provide the pci device with access to the guest's address space. The pci config space is a bit tricky because we need to intercept accesses to the PCI BARs and also to the MSI capability. But beyond those registers we simply intercept config space accesses and fulfill them by issuing PCIIOCREAD and PCIIOCWRITE ioctls on /dev/pci Accesses to the MMIO space are straightforward and require no hypervisor intervention after setting up the nested page tables. Accesses to the I/O space allocated to the device are trapped by the hypervisor and handled by the opening /dev/io and issuing IODEV_PIO ioctl on it. And finally dma transfers generated by the pci device into the guest's address space are transparent to the hypervisor once it has set up the IOMMU. Hope that helps. best Neel > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" >