From owner-freebsd-arch@FreeBSD.ORG Tue Apr 19 18:50:26 2011 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89406106566C for ; Tue, 19 Apr 2011 18:50:26 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3F2408FC16 for ; Tue, 19 Apr 2011 18:50:26 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id C5A6246B46 for ; Tue, 19 Apr 2011 14:50:25 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3A81F8A02B for ; Tue, 19 Apr 2011 14:50:25 -0400 (EDT) From: John Baldwin To: arch@freebsd.org Date: Tue, 19 Apr 2011 14:50:23 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201104191450.23758.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 19 Apr 2011 14:50:25 -0400 (EDT) Cc: Subject: Changing how PCI-PCI bridges do resource allocation X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2011 18:50:26 -0000 I have various patches to allow PCI-PCI bridges to more properly manage their I/O resource windows. So far it is only ported to x86, but I believe it should not be too hard to enable on other platforms. I've broken the patch down into three patches: 1) Some small cleanup to the rman code. Specifically, change rman_manage_region() to actually honor the rm_start and rm_end constraints on the rman and reject attempts to manage a region that is out of range. This exposed several bugs and the patch includes various places that set rm_end incorrectly (typically to ~0 or ~0u instead of ~0ul). Also, to preserve existing behavior, I've changed rman_init() to set rm_start and rm_end to allow managing the full range (0 to ~0ul) if they are not set by the caller when rman_init() is called. http://people.freebsd.org/~jhb/patches/rman_init.patch 2) Extend the rman and bus_if APIs to add some helper routines for managing resources. Specifically, rman_adjust_resource() allows an existing exclusive resource to grow or shrink one or both ends of a resource so that it has a new range. The PCI-PCI bridge driver uses this to grow an I/O window. At the new-bus level there is a new BUS_ADJUST_RESOURCE() method added to the bus interface that a driver that actually allocates resources (such as nexus(4) on x86) can implement via rman_adjust_resource(). This patch also adds methods to fetch the start and end of the first and last free regions within an rman. I use these last two methods to figure out the smallest amount needed to grow an I/O window when growing is required. Note that this patch only includes bus_adjust_resource support for PCI bridges, busses, and nexus devices on x86. It should not be hard to implement this for other platforms however. http://people.freebsd.org/~jhb/patches/bus_adjust_resource.patch 3) Finally, the last patch actually rototills the resource allocation code in the PCI-PCI bridge driver itself. To aid transition, there is a new kernel option (NEW_PCIB) that must be enabled to use the new code. This can allow us to transition some architectures over until all the other architectures have cuaght up for example. The way this is implemented is that each I/O window allocates is full range of addresses from its parent device. The driver then initializes an rman for each I/O window that is "backed" by the resource allocated from the parent. Resource allocation requests for child devices are allocated from the rman in the associated I/O window, growing the I/O window if necessary. This is very similar to how the ACPI bus driver allocates system resources and then suballocates them to child devices. As with the ACPI driver, this does require that bus_activate_resource() in the parent driver of Host to PCI bridges actually works (on many of the embedded platforms it does not, bus_alloc_resource() checks for RF_ACTIVE and implements the logic directly). I e-mailed patches out to fix this several years ago but they were not tested at the time. Those things will have to be addressed before NEW_PCIB can be used. The other requirement for other platforms is to implement appropriate bus_adjust_resource() methods. I think this should not be too difficult however. In general if a bus driver does not create rmans, it can just pass the request up via bus_generic_adjust_resource(). However, if it creates rmans and a resource belongs to one of its rmans, it should use rman_adjust_resource(). http://people.freebsd.org/~jhb/patches/pcib_window.patch I've already had at least one testing report that this fixes the issues with some machines' BIOS clearing the I/O windows on some PCI-PCI bridges when ACPI is enabled as this code re-discovers the original windows and programs them correctly. More testing would be good however. Beyond this I am also hacking on some changes to allow Host to PCI bridge drivers that know what ranges they decode to restrict allocation requests to those ranges (some ACPI systems and MP Tables provide this info). I am also hacking on changes to add support for PCI bus renumbering as well. In general all of my recent PCI hacking can be found in p4 at //depot/projects/pci/.... -- John Baldwin