From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 15 02:06:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BF89B66 for ; Sat, 15 Mar 2014 02:06:13 +0000 (UTC) Received: from mail-ob0-x231.google.com (mail-ob0-x231.google.com [IPv6:2607:f8b0:4003:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0A537B8C for ; Sat, 15 Mar 2014 02:06:12 +0000 (UTC) Received: by mail-ob0-f177.google.com with SMTP id wo20so3248803obc.8 for ; Fri, 14 Mar 2014 19:06:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=YaIYaiQd08vdQ+EWS22qpJ2ejspFJk5vCrUkzjA7k0w=; b=PWoS2LPIgg79chvQAf46yjeDAIvhs1Ybkm92SUgvdTmUd1pbs8bx5Dhc0Q8YBLj7DM KtMVuz2ZWZbhs3/Vs7pXRPDbwrcF/EURb2EdpD+RkGLWCbads5xRIddpfzI1xxxAiy5q EM64Ia+JlhIoM5feSGUa1HXYidb8TtUScKhXoI4FMELKUBmzLspHYLG1eLk2uOJOnibp Fr5GE3h1zC8rwHntXgsaGjoLfx83KlzlrkCVNJtwRNHHQlnO4lgwpjgsxqDnkohLcvKF gCITq4fPuxi0/iBM3+R82aVHQz87KkM+PY3tegYiorg04pAFQoz+6o8n+G0MSIYwOPv0 +O9w== MIME-Version: 1.0 X-Received: by 10.182.65.199 with SMTP id z7mr9634628obs.7.1394849172404; Fri, 14 Mar 2014 19:06:12 -0700 (PDT) Received: by 10.76.132.8 with HTTP; Fri, 14 Mar 2014 19:06:12 -0700 (PDT) Date: Fri, 14 Mar 2014 22:06:12 -0400 Message-ID: Subject: [PATCH] Support PCIe Alternative RID Interpretation (ARI) From: Ryan Stone To: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Mar 2014 02:06:13 -0000 http://people.freebsd.org/~rstone/patches/pci_ari.diff This patch add support for PCIe Alternative RID Interpretation (ARI) to our PCI implementation. ARI is an optional feature in PCIe; when it is enabled on a endpoint device that device can have up to 256 PCI functions (increased from 8). This is implemented by re-interpreting the 5-bit slot number as being part of the function number. The slot number for all such functions will implicitly be 0. There are two main changes here. The first changes PCI enumeration to explicitly probe slot 0, function 0 separate from all other devices. This is necessary because we must check whether the device supports ARI and enable it before enumerating the rest of the devices. The other main change affects pci configuration space accesses. When pcib_read_config/pcib_write_config is called on a downstream port that has ARI enabled, we convert the 8-bit function number back into a 5-bit slot number and 3-bit function number before passing the request up the device tree. This was the simplest way to insulate the rest of the PCI subsystem from knowledge of ARI. I have also added a new tunable, hw.pci.enable_ari, that controls whether ARI will be enabled on devices that support it. I plan on defaulting it to 1 on HEAD but will probably default to 0 when I MFC. I've tested on a few different motherboards at $WORK, but the only ARI-capable device that I have access to is Intel 82599 NICs (ixgbe driver), so if anybody is able to test with other hardware that would be really helpful. Testing that the driver is able to attach and testing basic functionality (e.g. ping on a network interface) would be sufficient. In order for ARI to be enabled both the downstream port (the physical PCIe slot) and the endpoint device (the PCIe card) have to both support ARI. My patch adds support to pciconf -lc to have it print whether a downstream port supports ARI. You can check for ARI support by looking for: cap 10[90] = PCI-Express 2 root port slot max data 128(256) link x0(x4) speed 0.0(5.0) ASPM disabled(L0s/L1) ARI disabled If it says "ARI enabled" or "ARI disabled" then the port supports ARI. If ARI is disabled please confirm that no ARI-capable device is in the slot (or else it's a bug and please report it). If ARI is enabled then please confirm that there is an ARI-capable device in the slot (otherwise please report it to me) pciconf -lc already will print out if a endpoint device supports ARI. Look for the ARI extended capability: ecap 000e[150] = ARI 1