From owner-freebsd-current@FreeBSD.ORG Tue May 3 12:21:09 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D15F410657A1 for ; Tue, 3 May 2011 12:21:09 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8C7BF8FC16 for ; Tue, 3 May 2011 12:21:09 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 4639646B43; Tue, 3 May 2011 08:21:09 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D8D6E8A02A; Tue, 3 May 2011 08:21:08 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Date: Tue, 3 May 2011 07:58:12 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: <4DBF68A4.6050405@protected-networks.net> In-Reply-To: <4DBF68A4.6050405@protected-networks.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201105030758.12792.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 03 May 2011 08:21:08 -0400 (EDT) Cc: Michael Butler Subject: Re: cardbus memory allocation problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 12:21:10 -0000 On Monday, May 02, 2011 10:29:56 pm Michael Butler wrote: > I've stared at this for a (long) while but haven't come to any > reasonable conclusion as to why it does what it does or how to fix it :-( > > Specifically, the BIOS in this machine doesn't set up a memory window > for the cardbus controller nor does it properly configure the PCI bridge > to route to the correct buses. BSD tries but allocates memory from the > wrong space. > > My question is - how to get PCI-cardbus bridge to allocate memory inside > the window of the parent PCI-PCI bridge? .. the bus tree looks like .. > > imb@toshi:/home/imb> sudo lspci -t > -[0000:00]-+-00.0 > +-02.0 > +-02.1 > +-1b.0 > +-1c.0-[02]-- > +-1c.1-[03-04]-- > +-1c.2-[05-06]----00.0 > +-1d.0 > +-1d.1 > +-1d.2 > +-1d.3 > +-1d.7 > +-1e.0-[07]--+-06.0 > | +-06.1 > | +-06.2 > | +-06.3 > | \-08.0 > +-1f.0 > +-1f.2 > \-1f.3 > > I've annotated the verbose dmesg below to highlight the issues .. > > pcib4: at device 30.0 on pci0 > pcib4: domain 0 > pcib4: secondary bus 7 > pcib4: subordinate bus 7 > *** subordinate bus needs to be '9' so as to include both '8' & '9' > *** for the PCI-cardbus bridge I have WIP patches to fix this but they aren't ready yet. > pcib4: I/O decode 0x4000-0x4fff > pcib4: memory decode 0xf0900000-0xf09fffff > *** this memory widow is what I expected all children to allocate from > > pcib4: no prefetched decode > pcib4: Subtractively decoded bridge. It's a subtractive bridge, so the resources do not have to be allocated from the window. That said, I'm committing the last of my patches to HEAD today to rework how PCI-PCI bridges handle I/O windows to support growing windows, etc. and the new PCI-PCI bridge driver will attempt to grow the memory window to allocate a new range before falling back to depending on the subtractive decode. > cbb0: at device 6.0 on pci7 > pcib4: cbb0 requested memory range 0x0-0xffffffff: good > *** what appears to be a "wildcard" alloc request > > cbb0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xbf670000 > *** but which isn't constrained to be within the parent bridge's space Yes, it is a subtractive bridge, so that should be fine. The problem may be that the bf670000 address may not be decoded by the parent Host-PCI bridge. There are some tunable hacks you can try to force this address higher, but I am also working on other patches (before the bus numbering ones) to query ACPI for the list of valid decoded ranges of PCI addresses for Host-PCI bridges and to restrict PCI allocations to coming from those ranges. You can try increasing hw.acpi.host_mem_start or hw.cbb.memory_start loader tunables. (There should be sysctl's with the current values I think.) -- John Baldwin