From owner-freebsd-smp Thu Sep 12 13:05:01 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id NAA27405 for smp-outgoing; Thu, 12 Sep 1996 13:05:01 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id NAA27370 for ; Thu, 12 Sep 1996 13:04:53 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id OAA19225; Thu, 12 Sep 1996 14:04:27 -0600 Message-Id: <199609122004.OAA19225@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Peter Wemm cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org, Terry Lambert Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 12:17:16 MDT." <199609121817.MAA18639@clem.systemsix.com> Mime-Version: 1.0 Content-Type: text/plain Date: Thu, 12 Sep 1996 14:04:27 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >Hmm. I'd rather avoid an extra indirection map if possible. Considering >that we are #define'ing things like curproc and some other heavily used >variables to lookup the cpuid for array indexes, we would better spend the >effort to just deal with a sparse ID set properly. further thoughts on dense vs sparse ID mapping: the simple indirection to get an int from a table of 16 ints is pretty cheap. I suggest cleanup of any areas of code that repeatedly call macros like curproc. I did a quick grep in /kern and found a few offenders like: curproc->p_flag |= P_PHYSIO; (I wonder if gcc will optimize this to only 1 APIC read?) Most of the code does a 1 time assignment to a struct ptr via curproc and uses that. if we have sparse tables, ie 16 entries (most of which will be empty), code like the following becomes very inefficient: #ifdef SMP for (j = i = 0; i < SPARSE_TABLE_SIZE; i++) { if (p == SMPcurproc[i]) j++; } ... --- of course all the above is assumming my current belief that we can't get away with reprogramming the APIC IDs to a dense sequence. I know that the APICs initially get their IDs from the state of 4 pins during hardware (motherboard) RESET. Whether there are problems when the ID you reprogam to are different I haven't a clue... --- 2nd STARTUP pentium proc. vol 1, 20.1.1.4, par 5: "It is the responsibility of the system software to resend the STARTUP IPI if there is an error... [enable APIC, use LVT3 ERROR vector to check it ] Otherwise the system software would have to poll the delivery status bit ... to determine if IPI is pending ... and resend STARTUP IPI if the IPI remains pending after an appropiate amount of time." since we poll this bit I think that (logically anyways) we should be safe only sending the one STARTUP IPI. From the XXPRESS port we know that if there isn't an APIC sitting there @ that ID the loop hangs forever. If we go to a broadcast scheme then polling may or may not work. I could see it working like a "wired-or", ie stays pending till all CPUs have received it, OR screwing up totally. Does anyone out there reading this list have a board with more than 2 CPUs? We might want to run some tests before we paint ourselves into either corner. --- # of IDs: pentium proc. vol 3, 19.3.1.5, 'Physical Dest Mode': "... A single destination (ID = 0 thru 14) or a broadcast to all (ID = 15) can be specified in the physical mode. ... Note that in this mode, the Pentium Processor ... APIC supports up to 15 agents." So this is probably why the XXPRESS IDs its IO APIC #14, ie at the top end. The board has a 2nd IO APIC for the 2nd PIC bus (we never looked at the MP table while it was enabled) which probably would get ID #13. Ie., I am guessing they start IDing IO APICs at the top and work down. Why they skip ID #1 for CPUs I have not a clue!!! Terry earlier wondered how the Sequent supported 32 CPUs. There is a more complex 'heirarxchial model' that supports up to 60 APICs. This model requires additional custom hardware support. I don't think I wanna go there. --- I'm going to cleanup mptable.c next, and will post it in an hour or 2. It is the user level program that we used to dump the MP tables I published earlier today. -- Steve Passe | powered by smp@csn.net | FreeBSD