From owner-freebsd-virtualization@FreeBSD.ORG Tue May 27 01:51:23 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3EC6D811 for ; Tue, 27 May 2014 01:51:23 +0000 (UTC) Received: from mail-oa0-x234.google.com (mail-oa0-x234.google.com [IPv6:2607:f8b0:4003:c02::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D1BD2FCC for ; Tue, 27 May 2014 01:51:23 +0000 (UTC) Received: by mail-oa0-f52.google.com with SMTP id eb12so8754746oac.25 for ; Mon, 26 May 2014 18:51:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=I/uHRj0oK7oJUTf3/aYWJaRIDQnjkc2fogexU2MINqc=; b=mI2Vlw+6yrAHKSZd6TIod/4r3/NIayEB77bbFBmw3F3JvIWLsqk6cQS7E6ZhhE5pP3 0FSoRVh+9U0M0FRXzj3Iu7wltrMU1fYEFKcqdWN/F1w0MoHcdEj/Pl5y4qT/q7dItafB hJaahUVVPUz3fZdYa4VIuY7B4U1wf7J3MHMp5vtrduj4K7Oo8g5JWOkfw++eYkdEEtP3 JS/9vkOovx1WWx8/Bw1KtmoO8BJRxHk9xDwdlSQAs9OQL6siec9gEQ5drOyEF9QgEhua wt7zw0Qn7vr9GwJlZfGemZqmy4eYIhtZmqz1p8/pxd2WlGVUkTDjUcmSooZ0k7qfv+/4 a4/A== MIME-Version: 1.0 X-Received: by 10.182.126.137 with SMTP id my9mr29178314obb.61.1401155482423; Mon, 26 May 2014 18:51:22 -0700 (PDT) Received: by 10.76.23.130 with HTTP; Mon, 26 May 2014 18:51:22 -0700 (PDT) Date: Mon, 26 May 2014 21:51:22 -0400 Message-ID: Subject: SR-IOV Patch Series 2/7: bhyve integration From: Ryan Stone To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2014 01:51:23 -0000 The bhyve work to interoperate is quite simple. PCI Passthrough through the ppt driver works just fine with SR-IOV VFs. The three changes are: - allow ppt devices to detach. This would happen if the administrator destroyed VFs with iovctl -D - allow the SR-IOV infrastructure to force the ppt driver to attach to certain devices by using the special BUS_PROBE_NOWILDCARD return value from device_probe(). This will have no effect on regular devices, but the SR-IOV infrastructure can create VF devices in a way where BUS_PROBE_NOWILDCARD will match the device. This allows passthrough of a given VF to be configured through iovctl (see ioctl.conf(5) for an example of how to do this). - Currently if vmm.ko is loaded and no ppt devices are present, the IOMMU is not enabled. If VFs are subsequently created as passthrough devices it will not be possible to pass them through to VMs. This patch adds a new tunable that will force the IOMMU to be enabled. Note: Currently ppt suffers from a lack of locking. This is now a problem because ppt devices can be destroyed, so there would be races here. I tried to address this by adding a mutex but that failed in ways that I could not figure out how to resolve (basically, some parts of ppt are called into from within a critical section, so a spinlock is needed, but other parts of ppt call into the VM subsystem to do M_WAITOK allocations, so an sx lock is needed. It wasn't at all obvious to me how to resolve that conflict, so I'm hoping that a more experienced bhyve person has an idea). http://people.freebsd.org/~rstone/patches/iov/0004-Allow-passthrough-devices-to-be-hinted.patch [PATCH 04/21] Allow passthrough devices to be hinted. Allow the ppt driver to attach to devices that were hinted to be passthrough devices by the PCI code creating them with a driver name of "ppt". Add a tunable that allows the IOMMU to be forced to be used. With SR-IOV passthrough devices the VFs may be created after vmm.ko is loaded. The current code will not initialize the IOMMU in that case, meaning that the passthrough devices can't actually be used. --- sys/amd64/vmm/io/ppt.c | 72 +++++++++++++++++++++++++++----------------------- sys/amd64/vmm/vmm.c | 7 ++++-