Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 26 May 2014 21:38:31 -0400
From:      Ryan Stone <rysto32@gmail.com>
To:        "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
Subject:   PCI SR-IOV patches for review
Message-ID:  <CAFMmRNwHM3i6WPbwftnn_O4TyMJsLHu%2B=X_CvzRfus6MHUaYcg@mail.gmail.com>

next in thread | raw e-mail | index | archive | help
My work on the PCI SR-IOV driver framework is complete and ready for review.

This includes:
- extending the PCI infrastructure to be able to create SR-IOV Virtual
Functions, allocating PCI resources to them and probing/attaching the
VF driver
- bhyve integration to make it easy to pass VFs to bhyve VMs as PCI
passthrough devices
- a flexible infrastructure for defining driver-specific configuration
parameters
- a unified configuration tool, iovctl, for configuring SR-IOV on PF devices

At this point I do not yet have a fully working PF driver.  I'm still
working on the i40e and ixgbe drivers at this time, and I don't have
an estimated completion date yet.  The main reason why I would like
this reviewed and committed ahead of the drivers is that some driver
maintainers would like access to the SR-IOV infrastructure so that
they can do the work in their own drivers.  Ideally, we can have
multiple SR-IOV drivers in place in time for 10.1-RELEASE.

This work was sponsored by Sandvine Inc.  Thanks to John Baldwin, who
provided advice on how to integrate this into the pci driver, and to
Jack Vogel, who gave pointers on the Intel drivers and hardware.


I described the driver interface on a post to -arch earlier in the month:
http://lists.freebsd.org/pipermail/freebsd-arch/2014-May/015332.html

The interface is largely unchanged (but yes, I did eliminate the
header pollution caused by nvlist_t :) ).  The main difference is that
the get_iov_config_schema method has been eliminated.  Now, the PF and
VF schemas are passed directly to pci_iov_attach() during the
device_attach method.  I did this because the proposed API could be
abused by a PF driver that dynamically changed the schema that it
provided at runtime, which was not the intention of the API.  As the
schema is administrator-facing I felt that changing it dynamically
would be potentially very confusing to an administrator, and allowing
it really doesn't bring any benefits.  A driver might want to modify
the schema based on the hardware type, but the driver can just as
easily detect the hardware at attach as they can at runtime, so I
don't think that this will be a problem.

The driver interface and iovctl have manpages, so please refer to
those before asking about the details of the interface from the driver
maintaner's or administrator's perspective.

The internal representation of the configuration schema and the SR-IOV
configuration itself is described in depth in comments in <sys/iov.h>.
 I've also put sample versions of both in the comments, which should
prove useful when reviewing code that deals with those data
structures.

The patch series can be see here:
http://people.freebsd.org/~rstone/patches/iov/

This is a fairly large change, so I've done my best to break it up
into a lot of smaller, more digestible patches.  Keep in mind that an
individual patch may seem pointless in isolation (for example, it
introduces a new API that is unused).  Rest assured that another patch
will be consuming the new APIs that I have added.

The patches can be broken up into the following series of patches.  I
plan on start separate email threads on the relevant list for review
of individual patches.  If you have any comments or questions about an
individual patch, please respond to the thread for that patch series.
I will note the mailing list that I intended to send each patch series
to below.

This work depends on a port of libnv to the kernel.  This work is
nearly complete; I'm working with pjd@ to get some technical details
ironed out and then I'll send those patches out for review.

Any comments that anybody can offer would be greatly appreciated.

Series 1/7: PCI Infrastructure Refactoring (freebsd-hackers)
http://people.freebsd.org/~rstone/patches/iov/0001-Refactor-PCI-device-creation.patch
http://people.freebsd.org/~rstone/patches/iov/0002-Refactor-PCI-resource-allocation.patch
http://people.freebsd.org/~rstone/patches/iov/0003-Add-some-pcib-methods-to-get-ARI-related-information.patch

Series 2/7: bhyve integration (freebsd-virtualization)
http://people.freebsd.org/~rstone/patches/iov/0004-Allow-passthrough-devices-to-be-hinted.patch

Series 3/7: manual pages (freebsd-doc)
http://people.freebsd.org/~rstone/patches/iov/0005-Document-pci_iov_attach-detach-in-pci.9.patch
http://people.freebsd.org/~rstone/patches/iov/0006-Add-manpages-for-SR-IOV-enable-disable-driver-interf.patch
http://people.freebsd.org/~rstone/patches/iov/0014-Add-a-manpage-for-iovctl-8.patch
http://people.freebsd.org/~rstone/patches/iov/0015-Add-manpage-documenting-iovctl-config-file-format.patch
http://people.freebsd.org/~rstone/patches/iov/0020-Document-the-interface-for-defining-a-configuration-.patch

Series 4/7: Core SR-IOV infrastructure in pci driver (freebsd-hackers)
http://people.freebsd.org/~rstone/patches/iov/0007-Implement-interface-to-create-SR-IOV-Virtual-Functio.patch
http://people.freebsd.org/~rstone/patches/iov/0008-Emulate-the-Device-ID-and-Vendor-ID-registers-for-VF.patch
http://people.freebsd.org/~rstone/patches/iov/0009-Allocate-PCI-I-O-memory-spaces-for-VFs.patch
http://people.freebsd.org/~rstone/patches/iov/0010-Add-interface-to-destroy-SR-IOV-VFs.patch

Series 5/7: SR-IOV Configuration Interface (freebsd-hackers)
http://people.freebsd.org/~rstone/patches/iov/0011-Add-infrastructure-for-exporting-config-schema-from-.patch
http://people.freebsd.org/~rstone/patches/iov/0012-Add-function-to-validate-the-consistency-of-SR-IOV-c.patch
http://people.freebsd.org/~rstone/patches/iov/0013-Pass-SR-IOV-configuration-to-kernel-using-an-nvlist.patch
http://people.freebsd.org/~rstone/patches/iov/0021-Validate-the-schema-that-the-PF-driver-passed-to-us.patch

Series 6/7: iovctl Configuration Tool (freebsd-hackers)
http://people.freebsd.org/~rstone/patches/iov/0016-Add-iovctl-functions-for-validating-config.patch
http://people.freebsd.org/~rstone/patches/iov/0017-Add-functions-for-parsing-the-iovctl-config-file.patch
http://people.freebsd.org/~rstone/patches/iov/0018-Add-main-for-iovctl-and-hook-iovctl-into-build.patch

Series 7/7: rc.d script (freebsd-rc)
http://people.freebsd.org/~rstone/patches/iov/0019-Add-an-rc.d-script-to-invoke-iovctl-8-during-boot.patch



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFMmRNwHM3i6WPbwftnn_O4TyMJsLHu%2B=X_CvzRfus6MHUaYcg>