Date: Wed, 18 Jun 2003 14:33:16 +0200 From: Bernd Walter <ticso@cicely12.cicely.de> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: freebsd-alpha@freebsd.org Subject: Re: 5.1beta2 on Alpha ES40 w/ 32GB pys ram Message-ID: <20030618123315.GH3626@cicely12.cicely.de> In-Reply-To: <16111.54084.284151.643826@grasshopper.cs.duke.edu> References: <16094.16676.273466.121560@grasshopper.cs.duke.edu> <20030613190624.W36168-100000@mail.chesapeake.net> <16110.27799.68056.48679@grasshopper.cs.duke.edu> <20030618005958.GD3626@cicely12.cicely.de> <16111.54084.284151.643826@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jun 17, 2003 at 10:49:40PM -0400, Andrew Gallatin wrote: > > Bernd Walter writes: > > On Mon, Jun 16, 2003 at 09:19:19PM -0400, Andrew Gallatin wrote: > > > Most alphas have 4 DMA windows. We typically use 1 window for ISA > > > > Do you have an overview of which ones don't have 4 windows? > > Is the direct map sufficient for them or can we increase the ISA map > > size in such a case? > > Its been so long that I do not remember. I know for certain that the > API UP1x00 boxes don't but they use an AMD chipset. All the important > ones do, that I know of. (mcpcia, tsunami, marvel) OK. > > > dma, meaning we can do zero-copy (no bounce buffer) DMA to any ISA > > > device. This is only 16MB in size. > > > > > > We use another window as a 'direct map' window, meaning we map > > > physical addresses 0..1GB (or 0..2GB) into PCI address space, > > > generally at offset 1GB in PCI address space. > > > > What are the reasons for this difference? > > Is it just because of SRM interaction or some chipset limitation? > > Its because of where the SRM puts the memory space. Ie, on some > machiens, SRM puts the memory space on PCI adaptors at 2GB. so > if we put a window at 2GB, then there would be 2 devices with address > 2GB on the same bus. If we did what I've heard linux does, and took > care of reallocating the memory regions on the PCI bus, this problem > would go away. We may need support to initialize the bus ourself anyway. E.g. on AS4100: [136]cicely12# pciconf -l sym0@pci0:1:0: class=0x010000 card=0x00000000 chip=0x00011000 rev=0x02 hdr=0x00 pcib2@pci0:2:0: class=0x060400 card=0x00000000 chip=0x00011011 rev=0x02 hdr=0x01 sym1@pci0:3:0: class=0x010000 card=0x00000000 chip=0x000f1000 rev=0x03 hdr=0x00 pcib3@pci0:4:0: class=0x060400 card=0x00000000 chip=0x00011011 rev=0x02 hdr=0x01 pcib4@pci0:5:0: class=0x060400 card=0x00000000 chip=0x00011011 rev=0x02 hdr=0x01 isp0@pci2:0:0: class=0x010000 card=0x00000000 chip=0x10201077 rev=0x02 hdr=0x00 none0@pci3:4:0: class=0x010000 card=0x00000000 chip=0x73789004 rev=0x03 hdr=0x00 none1@pci3:5:0: class=0x058000 card=0x00000000 chip=0x10789004 rev=0x00 hdr=0x00 none2@pci3:8:0: class=0x010000 card=0x00000000 chip=0x73789004 rev=0x03 hdr=0x00 none3@pci3:12:0: class=0x010000 card=0x00000000 chip=0x73789004 rev=0x03 hdr=0x00 de0@pci4:4:0: class=0x020000 card=0x24001109 chip=0x00091011 rev=0x20 hdr=0x00 de1@pci4:5:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x20 hdr=0x00 de2@pci4:6:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x20 hdr=0x00 none4@pci4:7:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x20 hdr=0x00 eisab0@pci0:1:0: class=0x000000 card=0x00000000 chip=0x04828086 rev=0x15 hdr=0x00 none5@pci0:2:0: class=0x010000 card=0x00000000 chip=0x74789004 rev=0x03 hdr=0x00 ohci0@pci0:3:0: class=0x0c0310 card=0x00351033 chip=0x00351033 rev=0x43 hdr=0x00 ohci1@pci0:3:1: class=0x0c0310 card=0x00351033 chip=0x00351033 rev=0x43 hdr=0x00 ehci0@pci0:3:2: class=0x0c0320 card=0x33830ee4 chip=0x00e01033 rev=0x04 hdr=0x00 de3@pci0:4:0: class=0x020000 card=0x12001109 chip=0x00091011 rev=0x20 hdr=0x00 none6@pci0:5:0: class=0x010000 card=0x00000000 chip=0x81789004 rev=0x00 hdr=0x00 You can see that the second hose has pci0 as well. Much worse a bridged card could get a bus number that is also used on another hose and it fails obscurly within FreeBSD. Currently I'm forced to keep bridged cards in the same hose. This machine also doesn't initialize io space on bridges and some drivers have problems with memory space. And if we do it ourself than it might be reasonable to think about hot-plug PCI some day. I havn't followed that idea very much because I worried about what SRM would do if we tamper with his configuration. Your Linux example sounds promissing. > > > We should use another window for 32-bit PCI DMA, and we should enable > > > the "monster window" bit, so that devices which are DAC (64-bit) > > > address capable don't need to deal with the TLB on the DMA > > > controller. We don't currently do either. > > > > In the first step I would say we should start checking cards for DAC > > support and if there are enough of them then do the monster window > > thing first. > > How does DAC fit into busdma? How do we indicate a device is DAC > capable to the busdma system? A good question. We need a full path inquiry as well to know if we have limited bridges. > > I asume it's not that critical for >2G machine owners to replace non DAC > > cards, especially as I believe many non DAC cards have currently non > > busdma drivers as well. > > Sounds reasonable. > > > The S/G map can be done later if still required. > > Maybe with a tuneable size as a 4G map takes 4M map - checked for 21172. > > I'm not sure what you're saying. Are you saying that a sparse map can > still be effective and we should share with the ISA map if we need to? > If so, I agree... More or less. Are sparse maps possible? I asume there is no use for a 4G map on a system with only 1G memory plugged in. Beside fragmentation loss there is no need to have a bigger window. The 21172 supports 8G memory and max 4G sized windows so we are already limited. -- B.Walter BWCT http://www.bwct.de ticso@bwct.de info@bwct.de
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030618123315.GH3626>