From owner-freebsd-hackers@FreeBSD.ORG Tue May 27 01:38:32 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AA1CA129 for ; Tue, 27 May 2014 01:38:32 +0000 (UTC) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com [IPv6:2607:f8b0:4003:c01::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 770CD2E71 for ; Tue, 27 May 2014 01:38:32 +0000 (UTC) Received: by mail-ob0-f170.google.com with SMTP id uy5so8716049obc.1 for ; Mon, 26 May 2014 18:38:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=FTyYj5xeY4u+4A5oifDETxV66ezDnLizpC1+bUQnY9g=; b=lZv+LY9yHKPwA5o5C1ghxddltE14P/jIzjptfA9mj76ZhqTgmbuBaxGgASINDkgoJa SnlJJuF5At6T5iIe1sjFfmsYFP6xp1pFFwUxjcGaZnycRxQYyE2/QYeCoxXOcAUqnA2G Juv1qedGeGQfG5/AWDm+qyOjVhr0lx6lImh+CfS2PPgZpFo/pYBu9+HVgbSVIVf9JJau ZH/xmsPXoJxgrg9AxCRYLUtOYGL+MMV7hCSNC4k4pV+PQ9q89onAsY292wZRk76nQFGz PMYk1kHfkO8ZdwocSow0QaCOqZ+YHgZKfarVQVzLQIXpOVg59WH8Skv8KDKnRCysWxJg cLoA== MIME-Version: 1.0 X-Received: by 10.60.62.211 with SMTP id a19mr13130918oes.71.1401154711635; Mon, 26 May 2014 18:38:31 -0700 (PDT) Received: by 10.76.23.130 with HTTP; Mon, 26 May 2014 18:38:31 -0700 (PDT) Date: Mon, 26 May 2014 21:38:31 -0400 Message-ID: Subject: PCI SR-IOV patches for review From: Ryan Stone To: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2014 01:38:32 -0000 My work on the PCI SR-IOV driver framework is complete and ready for review. This includes: - extending the PCI infrastructure to be able to create SR-IOV Virtual Functions, allocating PCI resources to them and probing/attaching the VF driver - bhyve integration to make it easy to pass VFs to bhyve VMs as PCI passthrough devices - a flexible infrastructure for defining driver-specific configuration parameters - a unified configuration tool, iovctl, for configuring SR-IOV on PF devices At this point I do not yet have a fully working PF driver. I'm still working on the i40e and ixgbe drivers at this time, and I don't have an estimated completion date yet. The main reason why I would like this reviewed and committed ahead of the drivers is that some driver maintainers would like access to the SR-IOV infrastructure so that they can do the work in their own drivers. Ideally, we can have multiple SR-IOV drivers in place in time for 10.1-RELEASE. This work was sponsored by Sandvine Inc. Thanks to John Baldwin, who provided advice on how to integrate this into the pci driver, and to Jack Vogel, who gave pointers on the Intel drivers and hardware. I described the driver interface on a post to -arch earlier in the month: http://lists.freebsd.org/pipermail/freebsd-arch/2014-May/015332.html The interface is largely unchanged (but yes, I did eliminate the header pollution caused by nvlist_t :) ). The main difference is that the get_iov_config_schema method has been eliminated. Now, the PF and VF schemas are passed directly to pci_iov_attach() during the device_attach method. I did this because the proposed API could be abused by a PF driver that dynamically changed the schema that it provided at runtime, which was not the intention of the API. As the schema is administrator-facing I felt that changing it dynamically would be potentially very confusing to an administrator, and allowing it really doesn't bring any benefits. A driver might want to modify the schema based on the hardware type, but the driver can just as easily detect the hardware at attach as they can at runtime, so I don't think that this will be a problem. The driver interface and iovctl have manpages, so please refer to those before asking about the details of the interface from the driver maintaner's or administrator's perspective. The internal representation of the configuration schema and the SR-IOV configuration itself is described in depth in comments in . I've also put sample versions of both in the comments, which should prove useful when reviewing code that deals with those data structures. The patch series can be see here: http://people.freebsd.org/~rstone/patches/iov/ This is a fairly large change, so I've done my best to break it up into a lot of smaller, more digestible patches. Keep in mind that an individual patch may seem pointless in isolation (for example, it introduces a new API that is unused). Rest assured that another patch will be consuming the new APIs that I have added. The patches can be broken up into the following series of patches. I plan on start separate email threads on the relevant list for review of individual patches. If you have any comments or questions about an individual patch, please respond to the thread for that patch series. I will note the mailing list that I intended to send each patch series to below. This work depends on a port of libnv to the kernel. This work is nearly complete; I'm working with pjd@ to get some technical details ironed out and then I'll send those patches out for review. Any comments that anybody can offer would be greatly appreciated. Series 1/7: PCI Infrastructure Refactoring (freebsd-hackers) http://people.freebsd.org/~rstone/patches/iov/0001-Refactor-PCI-device-creation.patch http://people.freebsd.org/~rstone/patches/iov/0002-Refactor-PCI-resource-allocation.patch http://people.freebsd.org/~rstone/patches/iov/0003-Add-some-pcib-methods-to-get-ARI-related-information.patch Series 2/7: bhyve integration (freebsd-virtualization) http://people.freebsd.org/~rstone/patches/iov/0004-Allow-passthrough-devices-to-be-hinted.patch Series 3/7: manual pages (freebsd-doc) http://people.freebsd.org/~rstone/patches/iov/0005-Document-pci_iov_attach-detach-in-pci.9.patch http://people.freebsd.org/~rstone/patches/iov/0006-Add-manpages-for-SR-IOV-enable-disable-driver-interf.patch http://people.freebsd.org/~rstone/patches/iov/0014-Add-a-manpage-for-iovctl-8.patch http://people.freebsd.org/~rstone/patches/iov/0015-Add-manpage-documenting-iovctl-config-file-format.patch http://people.freebsd.org/~rstone/patches/iov/0020-Document-the-interface-for-defining-a-configuration-.patch Series 4/7: Core SR-IOV infrastructure in pci driver (freebsd-hackers) http://people.freebsd.org/~rstone/patches/iov/0007-Implement-interface-to-create-SR-IOV-Virtual-Functio.patch http://people.freebsd.org/~rstone/patches/iov/0008-Emulate-the-Device-ID-and-Vendor-ID-registers-for-VF.patch http://people.freebsd.org/~rstone/patches/iov/0009-Allocate-PCI-I-O-memory-spaces-for-VFs.patch http://people.freebsd.org/~rstone/patches/iov/0010-Add-interface-to-destroy-SR-IOV-VFs.patch Series 5/7: SR-IOV Configuration Interface (freebsd-hackers) http://people.freebsd.org/~rstone/patches/iov/0011-Add-infrastructure-for-exporting-config-schema-from-.patch http://people.freebsd.org/~rstone/patches/iov/0012-Add-function-to-validate-the-consistency-of-SR-IOV-c.patch http://people.freebsd.org/~rstone/patches/iov/0013-Pass-SR-IOV-configuration-to-kernel-using-an-nvlist.patch http://people.freebsd.org/~rstone/patches/iov/0021-Validate-the-schema-that-the-PF-driver-passed-to-us.patch Series 6/7: iovctl Configuration Tool (freebsd-hackers) http://people.freebsd.org/~rstone/patches/iov/0016-Add-iovctl-functions-for-validating-config.patch http://people.freebsd.org/~rstone/patches/iov/0017-Add-functions-for-parsing-the-iovctl-config-file.patch http://people.freebsd.org/~rstone/patches/iov/0018-Add-main-for-iovctl-and-hook-iovctl-into-build.patch Series 7/7: rc.d script (freebsd-rc) http://people.freebsd.org/~rstone/patches/iov/0019-Add-an-rc.d-script-to-invoke-iovctl-8-during-boot.patch