Date: Thu, 12 Sep 1996 19:00:34 +0800 From: Peter Wemm <peter@spinner.dialix.com> To: Steve Passe <smp@csn.net> Cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks Message-ID: <199609121100.TAA03569@spinner.DIALix.COM> In-Reply-To: Your message of "Thu, 12 Sep 1996 02:12:56 CST." <199609120812.CAA15528@clem.systemsix.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Steve Passe wrote:
> Hi,
>
> > Hmm, something seems to have gone wrong, my second CPU isn't found with
> > this patch present, so I'll see if I can see what went wrong..
>
> I stripped out some debug while making the report, perhaps I broke something.
> I am currently rebuilding the world, I will apply the reported patch and
> verify whether it works when I've finished supping/building SMP.
> For my information, what motherboard are you using?
2xP90 ASUS Neptune PCI/E-P54NP4..
BTW, Some other things we do not do.. We don't set the ERROR LVT to handle
a non-delivered or failed message.
I noticed you took out the second STARTUP IPI.. The docs I've been
reading say "the startup IPI can only be used once after a reset or INIT",
and that it only has any effect once, and is ignored on the old 486 APIC,
and the example code suggests two STARTUP IPI's. From that, I read that
the second one is for insurance in case the first one was missed, and that
the second will normally be ignored. I notice that there is no way in the
world that we are waiting 10miliseconds after the initial INIT IPI.. My
system is probably missing the single startup IPI. Time do do some more
accurate times I think.
> ---
> > Hint: ALL the APIC ID registers are read/write.
> >
> > Can you see it yet? :-)
>
> we tried that on the XXPRESS and (Russel, please confirm this) an instant res
et
> of the hardware. My mail notes show:
> ----------------------------- mail from Russel ------------------------------
> > one last experiment. the book says that you can write the local apic id.
> > so the locore.s I just sent does that. you need to turn OFF "#define XXPRE
SS"
> > in i386/include/smpasm.h (and anywhere else you have set it). specifically
it
> > causes the 2nd CPU to change its apic ID from 2 to 1 during its early
> > bootup code, so we still want CPUNUMBER=0x02000000 for the 1st CPU INIT IPI
> > code, but after that everything should be in the context of 2nd CPU apic ID
=1.
>
> I removed my 'options XXPRESS' line and did a 'config SMP' and
> re-compiled with the new locore.s. Now the machine reboots as
> soon as I do the sysctl.
>
> ------------------------------- end mail notes ------------------------------
--
> We could have had another problem at this point so further experimentation
> wouldn't hurt.
Some other interesting quotes: "The operating system is responsible for
assigning non-conflicting ID's to all the IO APIC's". Is it possible that
the APIC's will reset the system if there's a conflict? You probably have
an IO APIC in the system, I wonder what the odds are that the BIOS has
already set it to ID 1, and when you moved #2 down to ID 1 there was a
M.A.D. reset.
> >Apart from the existance of the XXPRESS board, the nice easy solution is
> >to have the boot cpu assign it's ID to zero, and send a broadcast STARTUP
> >IPI and have the application cpu's fight for a lock, and assign themselves
> >sequentially increasing ID's, and assign the IO apic's ID to the end of
> >the list.
>
> my databook says:
>
> 110( Startup) This delivery mode is used as a special message
> between two processors ...
>
> ie, are you sure you can do broadcast 'Startup IPIs'?
The pppbios.pdf specifically says: "The BSP sends a StartUp APIC message
broadcast......" One of the various other tables in the P5 docs say that
startup IPI broadcasts are always edge triggered when used in "all but
self" mode, so who knows.. :-)
> as an alternative I would suggest starting to use the MP table:
>
> #define MAX_CPU 16
> int SMPIDMap[ MAX_CPU ];
>
> int nextCpu( void** fps ) { /* finds next cpu record in MP table @ fps */ }
>
> infoFromMPTable()
> {
> int mp_ncpus = 0;
> void* fps = mpfps;
> while ( nextCpu( &fps ) )
> {
> id = CpuId( fps )
> SMPIDMap[ id ] = mp_ncpus++;
> }
> }
>
> cpunumber()
> {
> return SMPIDMap[ GETCPUID() ];
> }
>
> I've already written code that dumps the entire table, I could produce routin
es
> like nextCpu() in quick order.
Hmm. I'd rather avoid an extra indirection map if possible. Considering
that we are #define'ing things like curproc and some other heavily used
variables to lookup the cpuid for array indexes, we would better spend the
effort to just deal with a sparse ID set properly.
Another thought.. We do not use the timer on the apic. It has a 32 bit
read/write register for the "initial count". We could cheat and use that
as a 32 bit pointer to a cpu-specific data page with each cpu's scratch
area etc,
> another issue to consider is that when you write the APIC ID register you als
o
> write its ARB (arbitration) register. Since I haven't yet deciphered
> the arbitration stuff I can't say whether this is an issue...
Hmm..
> ---
> >One thing I'm not clear about from the IO apic docs yet is whether there
> >are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on
> >the APIC bus.
>
> my belief is that it is a total of 15.
>
> the true 82489 has an 8-bit register, but the IO APIC in 586/686
> has a 4-bit field as does the 82379AB and 82093AA.
I've read more info now, I think you're right. the 82489 also has a 4-bit
bus and a different wire protocol. I think we can safely ignore it, since
it's discontinued(?) and we're very unlikely to see any. We'd be far
better off putting in a check for an 82489 apic in the boot code and
simply panic if so. Then we can concentrate on the P5 and P6
implementation, and then once it's working on the common hardware we can
consider making it run on the 486 theoretical platforms...
> On my mb the IO APIC is numbered 2, while the XXPRESS numbers it 14.
Ah, ok, well I guess that kills my theory about the IO apic being at #1
then. :-)
> >BTW, those .pdf docs you pointed out have done the trick and describe the
> .IO apic (version 0x11 at least :-) completely, although it is scattered
> >around different doc files a bit..
>
> did you get the book info I mailed earlier today about finding the true
> datasheet for the 82489?
Yes, I think so, but it's well out of my reach being in the USA. I'm
happy with the 82093AA docs that you found. At least, there's enough
detail there to try out a couple of things that I wanted to try.
> >As I understand it, the 82371SB PIIX3 chip has an IO apic address decoder,
> >but no apic. The IOAPIC chip (82093AA) is an optional extra part of the
> >430HX suite and would only be present on the multi-processor boards. I'm
>
> good eye, I didn't see this detail on my first scan of that doc.
The PIIX3 just provides address decode, it was the only reference in the
docs you pointed out. :-)
Cheers,
-Peter
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199609121100.TAA03569>