From owner-freebsd-smp Wed Oct 16 02:15:27 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id CAA27074 for smp-outgoing; Wed, 16 Oct 1996 02:15:27 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id CAA27054 for ; Wed, 16 Oct 1996 02:15:21 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.8.0/8.8.0) with ESMTP id RAA23430; Wed, 16 Oct 1996 17:15:06 +0800 (WST) Message-Id: <199610160915.RAA23430@spinner.DIALix.COM> X-Mailer: exmh version 1.6.9 8/22/96 To: cbrown@aracnet.com cc: smp@freebsd.org Subject: Re: dual-cpu PPRO motherboards... In-reply-to: Your message of "Wed, 16 Oct 1996 01:13:22 MST." <32649922.2A14@aracnet.com> Date: Wed, 16 Oct 1996 17:15:06 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Chris Browning wrote: > > >> Ah hem, the limit on the APIC bus, according to page 7-11 of the Pentium > > >> (TM) Pro > > >> Family Developers Manual, Volume 3 is 255. I believe this is standard > > >> ... > > > > > >We were talking about the software side of it. :-) The limit used to be > > >16 hardware interrupts, we can now deal with 24, although we could > > >probably squeeze a few more in. The software "interrupts" take a few > > >slots too. > > > > Even more specifically, we are talking about the INPUT capabilities of the > > IO APIC, a discrete component, NOT the "local APICs" found on P5/P6 CPUs. > > Actually, I am talking about the APIC bus spec. It supports 255 > interrupt > sources. The P6 (I'm not sure about the P5) atleast supports this. > When you say IP APIC, which one are you talking about? I know of > atleast > 3 that are commonly found on systems, probably more. Each of them > has a different number of interrupts and capabilies. Not to mention > systems with more than one IO APIC. Actually, there's two versions of the APIC bus, the 2-wire version found with the P5/P6 and the 82093AA APIC, as well as the older 4(?) wire version found on the i82489DX Local+IO Apic. The P5/P6 version supports 15 devices on the APIC bus (unless cluster managers are used), while the older bus protocol supported 255 devices. The 8-bit address field has been reduced to 4 bits. Every input on any apic device has an 8-bit "vector number", so each hardware input signal can generate any of the 256 possible "interrupts" (of which the first 32 are reserved for processor use and about 18 or so are already defined). So far as I can see, each IO APIC device could have 239 hardware interrupts if anybody was crazy enough to build one. Since there can be 14 APIC devices on the current P5/P6 bus, of which one has to be a cpu, you (I think) have a theoretical limit of 3107 interrupt sources in a system. Each of them can be programmed with any of the 255 IDT vector numbers, assuming the OS is prepared to have multiple sources generating the same IDT vector (FreeBSD doesn't at present). 32 of those IDT vectors are unavailable, and one is used for something else (linux system call processing). > > >Only one IO apic is supported at present, but I think we know about one > > >motherboard that has two IO apic's already (and two pci busses, etc), so > > >we've got to deal with having 48 or so hardware IRQ's on currently > > >existing hardware. The present irq masking architecture depends on being > > >able to locate the originating IO apic quickly to mask further interrupts > > >during the handler, so having both IO apics generating overlapping IDT > > >vectors is "too hard" at present. I'm sure Steve will yell if I've > > >misunderstood his code. > > > > Here I define 'too hard" more as "unknown". Not enough info is available to > > me right now to do support for multiple IO APICs. Once I see how a "2 IO A PIC" > > machine "looks" it might not turn out to be a big problem. In general, > > Peter has explained what I've been doing better than I could! > > What do you need to know? The machine I am tying this message on right > now has > 2 IO APICs in it. Neither are PIIX/PIIX3. The PIIX/PIIX3 only provide address decoding for something like an 82093AA APIC, it's got nothing else to do with the IO APIC other than housing the 8259 that we mask out. Supporting multiple IO apics should be pretty simple once we've got around the kernel's existing interrupt masking mechanism which was limited to 16 discreet hardware sources, Steve's bumped this up to 24. There's another "wall" at 28, because 4 bits of the 32-bit mask are used for software interrupts. To go beyond 28 interrupt sources and still maintain the 1:1 irq->IDT mapping we need an alternate masking strategy. On some EISA systems, some irq's (such as the timer) are hardwired internally to the 8259's and are not available as APIC inputs. This is a problem that will complicate things no end. On those systems both the 8259's and the APICs must be active, with the 8259 feeding it's output through the ExINT on the cpu's local apic. But that puts us well over the 32 interrupt source limit, so we are not even thinking about this yet until the basics are working right (which is pretty much there apart from some latency issues apparently, is that right Steve?). > Chris Cheers, -Peter