From owner-freebsd-smp Sun Sep 8 15:47:45 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA00650 for smp-outgoing; Sun, 8 Sep 1996 15:47:45 -0700 (PDT) Received: from groa.uct.ac.za (groa.uct.ac.za [137.158.128.7]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id PAA00643 for ; Sun, 8 Sep 1996 15:47:39 -0700 (PDT) Received: by groa.uct.ac.za via sendmail with stdio id for freebsd-smp@freebsd.org; Mon, 9 Sep 1996 00:47:29 +0200 (SAT) (Smail-3.2 1996-Jul-4 #1 built 1996-Jul-21) Message-Id: From: rv@groa.uct.ac.za (Russell Vincent) Subject: Intel XXpress - the conclusion To: freebsd-smp@freebsd.org Date: Mon, 9 Sep 1996 00:47:29 +0200 (SAT) X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Just a final status report on the work that Steve and I have done on the Intel XXpress and FreeBSD's SMP. We managed to get SMP working very well on the machine, although only with some debug code included, which makes for some suspicious timing bugs. I am sure Steve will elaborate when he submits all his mods - very worthwhile changes too. I am amazed at how well it worked. General interactive and network response was much more snappy with both processors running. I only have the machine for a few more hours, so can't do extensive tests. I am quite pleased with what I have seen, though. Many thanks to everyone who made suggestions and a special thanks to Steve who has done a huge amount of work on getting it going. Many late nights for both of us finally paid off. The whole experience has been fun and a very worthwhile learning curve for us both. Keep up the good work and I hope to help out some more when I actually purchase an MP board (not the XXpress - it is a little out of my price range. :-) ). -Russell From owner-freebsd-smp Mon Sep 9 00:07:58 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id AAA04515 for smp-outgoing; Mon, 9 Sep 1996 00:07:58 -0700 (PDT) Received: from who.cdrom.com (who.cdrom.com [204.216.27.3]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id AAA04510 for ; Mon, 9 Sep 1996 00:07:56 -0700 (PDT) Received: from critter.tfs.com ([140.145.230.252]) by who.cdrom.com (8.7.5/8.6.11) with ESMTP id AAA03903 ; Mon, 9 Sep 1996 00:07:54 -0700 (PDT) Received: from critter.tfs.com (localhost.tfs.com [127.0.0.1]) by critter.tfs.com (8.7.5/8.7.3) with ESMTP id IAA09387; Mon, 9 Sep 1996 08:53:57 +0200 (MET DST) To: rv@groa.uct.ac.za (Russell Vincent) cc: freebsd-smp@freebsd.org Subject: Re: Intel XXpress - the conclusion In-reply-to: Your message of "Sat, 09 Sep 1996 00:47:29 +0200." Date: Mon, 09 Sep 1996 08:53:57 +0200 Message-ID: <9385.842252037@critter.tfs.com> From: Poul-Henning Kamp Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >We managed to get SMP working very well on the machine, although only >with some debug code included, which makes for some suspicious >timing bugs. I am sure Steve will elaborate when he submits all his >mods - very worthwhile changes too. Thanks for the work guys! We're looking forward to seeing your patches (please send-pr them so we don't loose them). It's always such a nice feeling when somebody says "We fixed/coded/did this" rather than "Please fix/code/do this", much appreciated! :-) -- Poul-Henning Kamp | phk@FreeBSD.ORG FreeBSD Core-team. http://www.freebsd.org/~phk | phk@login.dknet.dk Private mailbox. whois: [PHK] | phk@ref.tfs.com TRW Financial Systems, Inc. Future will arrive by its own means, progress not so. From owner-freebsd-smp Mon Sep 9 02:26:45 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id CAA10171 for smp-outgoing; Mon, 9 Sep 1996 02:26:45 -0700 (PDT) Received: from groa.uct.ac.za (groa.uct.ac.za [137.158.128.7]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id CAA10157 for ; Mon, 9 Sep 1996 02:26:07 -0700 (PDT) Received: by groa.uct.ac.za via sendmail with stdio id for freebsd-smp@freebsd.org; Mon, 9 Sep 1996 11:25:31 +0200 (SAT) (Smail-3.2 1996-Jul-4 #1 built 1996-Jul-21) Message-Id: From: rv@groa.uct.ac.za (Russell Vincent) Subject: Intel XXpress - some SMP benchmarks To: freebsd-smp@freebsd.org Date: Mon, 9 Sep 1996 11:25:31 +0200 (SAT) X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk 'lmbench 1.0' results for: Intel XXpress (dual P5-133), PCI/EISA Each processor has 1MB L2 write-back cache 92MB DRAM (kernel configured for 64MB) (ahc0:0:0): "CDC 94191-15 0136" type 0 fixed SCSI 1 de0 rev 17 int a irq 9 on pci0:12 de0: DC21041 [10Mb/s] pass 1.1 Note: I have only performed these benchmarks for interest comparison between the various tests below. Comparing them to another machine and/or another OS will make little/no sense. i.e: Don't bother. Various tests: 1) UP-1P : FreeBSD 2.2-current as of Fri, 6 Sep 1996 2) SMP-1P : FreeBSD 2.2-smp as of Fri, 6 Sep 1996 (after -current merge and with extra code added by Steve Passe ) o Single processor running 3) SMP-2P-1: FreeBSD 2.2-smp as of Fri, 6 Sep 1996 (after -current merge and with extra code added by Steve Passe ) o Dual processor running 4) SMP-2P-2: FreeBSD 2.2-smp as of Fri, 6 Sep 1996 (after -current merge and with extra code added by Steve Passe ), but less debug code than (3) - this appears to indicate timing bugs. o Dual processor running Comments: o The SMP kernel on the XXpress would not run without the extra code and without the debug statements. Removing the debug statements causes the machine to appear to 'freeze' when the second processor is started - I suspect it is still running, but not getting around to servicing interrupts properly. o By 'less debug code' in (4), I mean that a statement like: pushal; pushl _mp_lock; call _mp1; addl $4, %esp; popal gets changed to: pushal; popal in get_mplock() and rel_mplock(). Hence the possible timing bugs. o The hard drive is slow - that was all I had available, but that isn't what is being tested. it did the job well enough. o Option (3), although not that good in the benchmarks, certainly appears faster in interactive use. That could just be my imagination, though. :-) L M B E N C H 1 . 0 S U M M A R Y ------------------------------------ Processor, Processes - times in microseconds -------------------------------------------- Host OS Mhz Null Null Simple /bin/sh Mmap 2-proc 8-proc Syscall Process Process Process lat ctxsw ctxsw --------- ------------- ---- ------- ------- ------- ------- ---- ------ ------ SMP-1P FreeBSD 2.2-C 133 8 1.6K 8.8K 15K 78 29 31 SMP-2P-1 FreeBSD 2.2-C 134 36 2.6K 13.0K 20K 135 14 45 SMP-2P-2 FreeBSD 2.2-C 132 516 20.5K 184.3K 298K 1163 17 41 UP-1P FreeBSD 2.2-C 133 5 1.5K 7.9K 13K 71 15 17 *Local* Communication latencies in microseconds ----------------------------------------------- Host OS Pipe UDP RPC/ TCP RPC/ UDP TCP --------- ------------- ------- ------- ------- ------- ------- SMP-1P FreeBSD 2.2-C 81 197 318 232 397 SMP-2P-1 FreeBSD 2.2-C 172 309 503 366 617 SMP-2P-2 FreeBSD 2.2-C 212 285 520 1743 758 UP-1P FreeBSD 2.2-C 49 160 280 195 356 *Local* Communication bandwidths in megabytes/second ---------------------------------------------------- Host OS Pipe TCP File Mmap Bcopy Bcopy Mem Mem reread reread (libc) (hand) read write --------- ------------- ---- ---- ------ ------ ------ ------ ---- ----- SMP-1P FreeBSD 2.2-C 56 17.9 34.8 48.2 26 24 58 39 SMP-2P-1 FreeBSD 2.2-C 28 12.7 34.4 31.2 16 15 39 24 SMP-2P-2 FreeBSD 2.2-C 17 13.9 32.3 30.4 15 15 38 24 UP-1P FreeBSD 2.2-C 58 18.2 35.6 49.3 26 24 58 39 Memory latencies in nanoseconds (WARNING - may not be correct, check graphs) -------------------------------------------- Host OS Mhz L1 $ L2 $ Main mem TLB Guesses --------- ------------- --- ---- ---- -------- --- ------- SMP-1P FreeBSD 2.2-C 133 7 79 323 401 SMP-2P-1 FreeBSD 2.2-C 133 7 56 528 521 SMP-2P-2 FreeBSD 2.2-C 132 7 62 523 568 UP-1P FreeBSD 2.2-C 133 7 53 322 392 L M B E N C H 1 . 0 S U M M A R Y ------------------------------------ Comparison to best of the breed ------------------------------- (Best numbers are starred, i.e., *123) Processor, Processes - factor slower than the best -------------------------------------------------- Host OS Mhz Null Null Simple /bin/sh Mmap 2-proc 8-proc Syscall Process Process Process lat ctxsw ctxsw --------- ------------- ---- ------- ------- ------- ------- ---- ------ ------ SMP-1P FreeBSD 2.2-C 133 1.6 1.1 1.1 1.1 1.1 2.1 1.8 SMP-2P-1 FreeBSD 2.2-C 134 7.2 1.8 1.6 1.5 1.9 *14 2.6 SMP-2P-2 FreeBSD 2.2-C 132 103 14 23 22 16 1.2 2.4 UP-1P FreeBSD 2.2-C 133 *5 *1.4K *7.7K *13.1K *71 1.1 *17 *Local* Communication latencies - factor slower than the best ------------------------------------------------------------- Host OS Pipe UDP RPC/ TCP RPC/ UDP TCP --------- ------------- ------- ------- ------- ------- ------- SMP-1P FreeBSD 2.2-C 1.7 1.2 1.1 1.2 1.1 SMP-2P-1 FreeBSD 2.2-C 3.5 1.9 1.8 1.9 1.7 SMP-2P-2 FreeBSD 2.2-C 4.3 1.8 1.9 8.9 2.1 UP-1P FreeBSD 2.2-C *49 *160 *280 *195 *356 *Local* Communication bandwidths - percentage of the best --------------------------------------------------------- Host OS Pipe TCP File Mmap Bcopy Bcopy Mem Mem reread reread (libc) (hand) read write --------- ------------- ---- ---- ------ ------ ------ ------ ---- ----- SMP-1P FreeBSD 2.2-C 97% 98% 97% 97% 99% *24 99% 99% SMP-2P-1 FreeBSD 2.2-C 47% 69% 96% 63% 59% 61% 67% 62% SMP-2P-2 FreeBSD 2.2-C 29% 76% 90% 61% 58% 61% 65% 62% UP-1P FreeBSD 2.2-C *57 *18 *35 *49 *26 99% *57 *38 Memory latencies in nanoseconds - factor slower than the best (WARNING - may not be correct, check graphs) ------------------------------------------------------------- Host OS Mhz L1 $ L2 $ Main mem TLB Guesses --------- ------------- --- ---- ---- -------- --- ------- SMP-1P FreeBSD 2.2-C 133 *7 1.5 1.0 1.0 SMP-2P-1 FreeBSD 2.2-C 133 *7 1.1 1.6 1.3 SMP-2P-2 FreeBSD 2.2-C 132 *7 1.2 1.6 1.4 UP-1P FreeBSD 2.2-C 133 *7 *53 *322 *392 From owner-freebsd-smp Mon Sep 9 03:54:50 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id DAA14851 for smp-outgoing; Mon, 9 Sep 1996 03:54:50 -0700 (PDT) Received: from spinner.DIALix.COM (spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id DAA14846 for ; Mon, 9 Sep 1996 03:54:44 -0700 (PDT) Received: from spinner.DIALix.COM (localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id SAA08111; Mon, 9 Sep 1996 18:43:58 +0800 (WST) Message-Id: <199609091043.SAA08111@spinner.DIALix.COM> X-Mailer: exmh version 1.6.7 5/3/96 To: rv@groa.uct.ac.za (Russell Vincent) cc: freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Sat, 09 Sep 1996 11:25:31 +0200." Date: Mon, 09 Sep 1996 18:43:58 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Russell Vincent wrote: > 'lmbench 1.0' results for: Ahem.. Enough said. :-) But regardless of the accuracy issue, it certainly gives an indication of the various bottlenecks. > o Option (3), although not that good in the benchmarks, certainly > appears faster in interactive use. That could just be my imagination, > though. :-) Several things to consider: - the second cpu is never pre-empted while running. This is bad (obviously :-) since a process that does a while(1); till run on the cpu forever unless it gets killed or paged. And on that note, we don't make any allowance for the page tables being changed while one cpu is in user mode. (we flush during the context switch, but that doesn't help if a page is stolen). I've been trying to decipher some of the more obscure parts of the apic docs, and it appears that we can sort-of simulate a round-robin approach on certain interrupts without too much reliability, but it's better than nothing I think. (I have in mind setting all the cpu "priorities" the same, and let the apic's use their internal tie-breaking weighting. I've not read enough on it yet, but I think it's possible...) - the smp_idleloop is currently killing the performance when one process is running, because the idleloop is constantly bouncing back and forwards between the two idle procs. ie: _whichidqs is always true, so it's constantly locking, and unlocking causing extreme congestion on that lock. There has got to be a better way to do the locking (I have ideas). When one process leaves kernel mode, it's got a fight on it's hands to get back in. It's got to try and get the MESI cache line in a favourable state so that it can try a lock. I'm suprised this hasn't turned up before now that I think about it. I would expect the system would not do too well under heavy paging load... :-( - several major subsystems run a fair bit of code without spl protection (I'm thinking of VFS and VM). If we could ever figure out how to clean the trap/exception/interrupt handling up enough to cleanly enter and exit a "locked" state, we could probably do wonders like having some parts of the kernel reentrant on both cpus. Unfortunately, the trap code is extremely optimised for the single-processor case (and I do mean extreme.. :-), and is quite difficult to follow. We had to introduce reference counting on the kernel mutex lock some time ago simply because parts of the kernel are reentered via the trap code from within the kernel. A rethink needs to happen here to figure out how we can cut downt he locking overheads without penalising the uniprocessor case much. That may mean having a seperate lock for the trap layer and the kernel, where only one cpu can be within the trap layer (with a simple, non-stacking lock), and the "kernel proper" lock is reference counted. The "kernel proper" lock could probably then have the vfs and perhaps vm split off into seperate locks or locking strategies. (and if somebody starts spouting jargon from his graph-theory book, that I for one don't understand a word of, I'll scream. :-) - "less debug code".. Have you looked very closely at the implications of your chipset bios settings? Is it possible that some of the speedups are deferring cpu cache writebacks too long and one cpu is getting data from RAM that has just been entered into the other chipset's "write buffer"? (ie: cache thinks it's been written back, but it's not in RAM yet, so the MESI protocol is defeated? I have no idea if this is possible or not.. just a wild guess. if "lock cmpxchg" is truely atomic, then the problem you see should not be happening... I presume you have tried the motherboard on "maximum pessimitic settings"? Anyway, I've got a deadline in a few hours, I've already spent way too long on this.. :-] Cheers, -Peter From owner-freebsd-smp Mon Sep 9 13:25:35 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id NAA17022 for smp-outgoing; Mon, 9 Sep 1996 13:25:35 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id NAA17017 for ; Mon, 9 Sep 1996 13:25:24 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id NAA01555; Mon, 9 Sep 1996 13:23:55 -0700 From: Terry Lambert Message-Id: <199609092023.NAA01555@phaeton.artisoft.com> Subject: Re: Intel XXpress - some SMP benchmarks To: peter@spinner.dialix.com (Peter Wemm) Date: Mon, 9 Sep 1996 13:23:55 -0700 (MST) Cc: rv@groa.uct.ac.za, freebsd-smp@freebsd.org In-Reply-To: <199609091043.SAA08111@spinner.DIALix.COM> from "Peter Wemm" at Sep 9, 96 06:43:58 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > The "kernel proper" lock > could probably then have the vfs and perhaps vm split off into seperate > locks or locking strategies. (and if somebody starts spouting jargon from > his graph-theory book, that I for one don't understand a word of, I'll > scream. :-) How about Clifford algebras or set theory instead? 8-) 8-). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Mon Sep 9 14:08:11 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA19249 for smp-outgoing; Mon, 9 Sep 1996 14:08:11 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id OAA19241 for ; Mon, 9 Sep 1996 14:08:06 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id PAA27929; Mon, 9 Sep 1996 15:07:42 -0600 Message-Id: <199609092107.PAA27929@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Peter Wemm cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Mon, 09 Sep 1996 18:43:58 +0800." <199609091043.SAA08111@spinner.DIALix.COM> Mime-Version: 1.0 Content-Type: text/plain Date: Mon, 09 Sep 1996 15:07:42 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > Russell Vincent wrote: > > 'lmbench 1.0' results for: > > Ahem.. Enough said. :-) But regardless of the accuracy issue, it > certainly gives an indication of the various bottlenecks. exactly, I wanted a benchmark from which to measure improvement. > > o Option (3), although not that good in the benchmarks, certainly > > appears faster in interactive use. That could just be my imagination, > > though. :-) > > Several things to consider: > - the second cpu is never pre-empted while running. This is bad > (obviously :-) since a process that does a while(1); till run on the cpu > forever unless it gets killed or paged. And on that note, we don't make at the very least something needs to be done along the lines of using the 2nd CPU's internal timer to context-switch it on the time-quantum. or perhaps using the apic InterProcessorInterrupt facility to allow the 1st CPU to tell the others when to call cpu_switch. Program the context switch timer for quantum/NCPU, then send each timer INT to the next CPU. this method would tend to keep the context switching by each CPU separated in time, thus avoiding mp_lock contention. (excuse me if I'm suggesting something stupid here, I have minimal knowledge of the kernel's internals at this point). > any allowance for the page tables being changed while one cpu is in user > mode. (we flush during the context switch, but that doesn't help if a > page is stolen). I will elaborate on the differences between options 3 & 4 in a later mailing, at this point it is my guess that either cache or page tables/page flushing is the issue. > I've been trying to decipher some of the more obscure > parts of the apic docs, and it appears that we can sort-of simulate a > round-robin approach on certain interrupts without too much reliability, > but it's better than nothing I think. (I have in mind setting all the cpu > "priorities" the same, and let the apic's use their internal tie-breaking > weighting. I've not read enough on it yet, but I think it's possible...) won't this also require code to enable/manage the I/O APIC? > - the smp_idleloop is currently killing the performance when one process > is running, because the idleloop is constantly bouncing back and forwards > between the two idle procs. ie: _whichidqs is always true, so it's > constantly locking, and unlocking causing extreme congestion on that lock. my debug code for tracking the mp_lock shows long periods where the 1st CPU is running the count thru 1,2,3,3,2,2,3,3,2,1,2,3... type progressions. I think these are INTerrupt periods. I can't explain the following progressions: ... cpu #1 requests mplock, lock is free cpu #1 gets mplock, count: 1 cpu #2 requests mplock, count: 1 cpu #1 enters free, count: 1 cpu #1 leaves free, lock is free cpu #1 requests mplock, count: 1 cpu #2 enters free, count: 1 cpu #2 leaves free, lock is free cpu #1 gets mplock, count: 1 cpu #2 requests mplock, count: 1 cpu #1 enters free, count: 1 cpu #1 leaves free, lock is free cpu #1 requests mplock, lock is free cpu #2 gets mplock, count: 1 cpu #2 enters free, count: 1 cpu #2 leaves free, lock is free cpu #1 gets mplock, count: 1 ... Note that the 2nd CPU goes directly from requesting the lock to entering rel_mplock(), without "gets get_mplock". the second time #2 requests the lock the progression looks valid. Note that the code that records these mp_lock changes is itself subject to race conditions and thus the data could become corrupt, but I see it happen too often to believe that this is what I am seeing here... I will get into more detail in the mailing about this debug code (to follow later...) > There has got to be a better way to do the locking (I have ideas). When if you would like me to start coding some test cases let me know. > - "less debug code".. Have you looked very closely at the implications of > your chipset bios settings? Is it possible that some of the speedups are > deferring cpu cache writebacks too long and one cpu is getting data from > RAM that has just been entered into the other chipset's "write buffer"? > (ie: cache thinks it's been written back, but it's not in RAM yet, so the > MESI protocol is defeated? I have no idea if this is possible or not.. > just a wild guess. if "lock cmpxchg" is truely atomic, then the problem > you see should not be happening... I presume you have tried the > motherboard on "maximum pessimitic settings"? we tried alot of different settings, nothing noticable. the Intel BIOS doesn't give one alot of choices, it doesn't even let you enable/disable cache (at least that we could find!). Note that the XXPRESS differs from many boards in that it has seperate cache sections for each CPU. Again, I'll get into the "less code" issue in the mailing specific to this test. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Tue Sep 10 01:12:52 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id BAA27012 for smp-outgoing; Tue, 10 Sep 1996 01:12:52 -0700 (PDT) Received: from groa.uct.ac.za (groa.uct.ac.za [137.158.128.7]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id BAA27001 for ; Tue, 10 Sep 1996 01:12:47 -0700 (PDT) Received: by groa.uct.ac.za via sendmail with stdio id for freebsd-smp@freebsd.org; Tue, 10 Sep 1996 10:11:52 +0200 (SAT) (Smail-3.2 1996-Jul-4 #1 built 1996-Jul-21) Message-Id: From: rv@groa.uct.ac.za (Russell Vincent) Subject: Re: Intel XXpress - some SMP benchmarks To: smp@csn.net (Steve Passe) Date: Tue, 10 Sep 1996 10:11:52 +0200 (SAT) Cc: peter@spinner.dialix.com, freebsd-smp@freebsd.org In-Reply-To: <199609092107.PAA27929@clem.systemsix.com> from "Steve Passe" at Sep 9, 96 03:07:42 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Steve wrote: > we tried alot of different settings, nothing noticable. the Intel BIOS doesn't > give one alot of choices, it doesn't even let you enable/disable cache (at > least that we could find!). Note that the XXPRESS differs from many boards > in that it has seperate cache sections for each CPU. Again, I'll get into > the "less code" issue in the mailing specific to this test. Sorry, I thought I had mentioned it to you. I did eventually find an option to disable the cache completely, but it didn't help with the problems we were having. I was amazed at what a difference it makes to system performance, though - felt like I was working on a slow 386! -Russell From owner-freebsd-smp Tue Sep 10 02:27:47 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id CAA03882 for smp-outgoing; Tue, 10 Sep 1996 02:27:47 -0700 (PDT) Received: from groa.uct.ac.za (groa.uct.ac.za [137.158.128.7]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id CAA03874 for ; Tue, 10 Sep 1996 02:27:39 -0700 (PDT) Received: by groa.uct.ac.za via sendmail with stdio id for freebsd-smp@freebsd.org; Tue, 10 Sep 1996 11:26:53 +0200 (SAT) (Smail-3.2 1996-Jul-4 #1 built 1996-Jul-21) Message-Id: From: rv@groa.uct.ac.za (Russell Vincent) Subject: Re: Intel XXpress - some SMP benchmarks To: smp@csn.net Date: Tue, 10 Sep 1996 11:26:52 +0200 (SAT) Cc: peter@spinner.dialix.com, freebsd-smp@freebsd.org In-Reply-To: from "Russell Vincent" at Sep 10, 96 10:11:52 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk I wrote: [ With regard to the write-back L2 cache affecting the SMP code ] > Sorry, I thought I had mentioned it to you. I did eventually find > an option to disable the cache completely, but it didn't help with > the problems we were having. Aaaaaarrggh! I still happen to have the machine, so just double checked this with the latest code and changing the write-back to write-through cache has fixed the problem. The machine now runs fine. The debug options are set to: # define TRACE_CPU_SWITCH_NOT # define FAKE_MP_NOT /* do nothing but return */ # define REAL_MP_NOT /* print data */ # define DO_MP_CALL_NOT /* push regs, call mpX(), pop regs */ # define PUSH_REGS_NOT /* push regs, pop regs */ I am not sure what happened the last time I tried - perhaps I had something else wrong in the code. If the machine doesn't get removed under my feet, I will post a new set of benchmarks, including this new breakthrough. -Russell From owner-freebsd-smp Tue Sep 10 07:13:21 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id HAA17931 for smp-outgoing; Tue, 10 Sep 1996 07:13:21 -0700 (PDT) Received: from groa.uct.ac.za (groa.uct.ac.za [137.158.128.7]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id HAA17910 for ; Tue, 10 Sep 1996 07:12:54 -0700 (PDT) Received: by groa.uct.ac.za via sendmail with stdio id for freebsd-smp@freebsd.org; Tue, 10 Sep 1996 16:12:05 +0200 (SAT) (Smail-3.2 1996-Jul-4 #1 built 1996-Jul-21) Message-Id: From: rv@groa.uct.ac.za (Russell Vincent) Subject: Intel XXpress - more SMP benchmarks To: freebsd-smp@freebsd.org Date: Tue, 10 Sep 1996 16:12:04 +0200 (SAT) Cc: smp@csn.net X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Here are some more benchmark results for the Intel XXpress. These results are when the machine was switched from write-back to write-through for the L2 cache. This had to be done because each processor has it's own 1MB L2 cache and it seems we were encountering cases where the data written by one processor wasn't in main memory quickly enough for the other processor to access it (or that was how I read it :-) ). It is also interesting comparing the difference between write-back and write-through (see previous message). It gives you an indication of how much the benchmarks can be affected by machine config. [ See machine spec, notes and descriptions in my previous message ] My favourite is the time for the 2-proc ctxsw (I was able to duplicate that in a second run). :-) Note: These benchmarks were only made for comparison between the various configs shown. Don't bother comparing them to anything else, because it won't make sense and you can't duplicate the config/environment/code base I have. So there. Anything else you would like to see? Seems I have the machine until tomorrow. L M B E N C H 1 . 0 S U M M A R Y ------------------------------------ Processor, Processes - times in microseconds -------------------------------------------- Host OS Mhz Null Null Simple /bin/sh Mmap 2-proc 8-proc Syscall Process Process Process lat ctxsw ctxsw --------- ------------- ---- ------- ------- ------- ------- ---- ------ ------ SMP-1P-WT FreeBSD 2.2-C 130 164 19.0K 107.2K 187K 547 80 97 SMP-2P-WT FreeBSD 2.2-C 134 123 9.0K 72.1K 115K 369 -5 33 UP-1P-WT FreeBSD 2.2-C 132 67 8.4K 37.8K 67K 254 32 32 *Local* Communication latencies in microseconds ----------------------------------------------- Host OS Pipe UDP RPC/ TCP RPC/ UDP TCP --------- ------------- ------- ------- ------- ------- ------- SMP-1P-WT FreeBSD 2.2-C 688 1679 3118 1791 3908 SMP-2P-WT FreeBSD 2.2-C 432 892 1761 949 2207 UP-1P-WT FreeBSD 2.2-C 277 733 1362 787 1714 *Local* Communication bandwidths in megabytes/second ---------------------------------------------------- Host OS Pipe TCP File Mmap Bcopy Bcopy Mem Mem reread reread (libc) (hand) read write --------- ------------- ---- ---- ------ ------ ------ ------ ---- ----- SMP-1P-WT FreeBSD 2.2-C 4 1.5 3.1 12.4 4 4 23 4 SMP-2P-WT FreeBSD 2.2-C 8 3.6 7.1 14.2 3 3 24 4 UP-1P-WT FreeBSD 2.2-C 9 3.4 7.2 27.7 8 8 58 9 Memory latencies in nanoseconds (WARNING - may not be correct, check graphs) -------------------------------------------- Host OS Mhz L1 $ L2 $ Main mem TLB Guesses --------- ------------- --- ---- ---- -------- --- ------- SMP-1P-WT FreeBSD 2.2-C 129 7 66 918 820 SMP-2P-WT FreeBSD 2.2-C 133 7 56 922 903 UP-1P-WT FreeBSD 2.2-C 132 7 80 325 393 L M B E N C H 1 . 0 S U M M A R Y ------------------------------------ Comparison to best of the breed ------------------------------- (Best numbers are starred, i.e., *123) Processor, Processes - factor slower than the best -------------------------------------------------- Host OS Mhz Null Null Simple /bin/sh Mmap 2-proc 8-proc Syscall Process Process Process lat ctxsw ctxsw --------- ------------- ---- ------- ------- ------- ------- ---- ------ ------ SMP-1P-WT FreeBSD 2.2-C 130 2.4 2.3 2.8 2.8 2.2 -16.0 3.0 SMP-2P-WT FreeBSD 2.2-C 134 1.8 1.1 1.9 1.7 1.5 *-5 1.0 UP-1P-WT FreeBSD 2.2-C 132 *67 *8.2K *36.9K *65.2K *254 -6.4 *32 *Local* Communication latencies - factor slower than the best ------------------------------------------------------------- Host OS Pipe UDP RPC/ TCP RPC/ UDP TCP --------- ------------- ------- ------- ------- ------- ------- SMP-1P-WT FreeBSD 2.2-C 2.5 2.3 2.3 2.3 2.3 SMP-2P-WT FreeBSD 2.2-C 1.6 1.2 1.3 1.2 1.3 UP-1P-WT FreeBSD 2.2-C *277 *733 *1362 *787 *1714 *Local* Communication bandwidths - percentage of the best --------------------------------------------------------- Host OS Pipe TCP File Mmap Bcopy Bcopy Mem Mem reread reread (libc) (hand) read write --------- ------------- ---- ---- ------ ------ ------ ------ ---- ----- SMP-1P-WT FreeBSD 2.2-C 43% 41% 43% 44% 44% 45% 40% 43% SMP-2P-WT FreeBSD 2.2-C 97% *3 99% 51% 42% 43% 41% 41% UP-1P-WT FreeBSD 2.2-C *8 93% *7 *27 *8 *7 *57 *9 Memory latencies in nanoseconds - factor slower than the best (WARNING - may not be correct, check graphs) ------------------------------------------------------------- Host OS Mhz L1 $ L2 $ Main mem TLB Guesses --------- ------------- --- ---- ---- -------- --- ------- SMP-1P-WT FreeBSD 2.2-C 129 *7 1.2 2.8 2.1 SMP-2P-WT FreeBSD 2.2-C 133 *7 *56 2.8 2.3 UP-1P-WT FreeBSD 2.2-C 132 *7 1.4 *325 *393 From owner-freebsd-smp Tue Sep 10 12:32:10 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id MAA08175 for smp-outgoing; Tue, 10 Sep 1996 12:32:10 -0700 (PDT) Received: from spinner.DIALix.COM (spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id MAA08169 for ; Tue, 10 Sep 1996 12:32:04 -0700 (PDT) Received: from spinner.DIALix.COM (localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id DAA00413; Wed, 11 Sep 1996 03:31:55 +0800 (WST) Message-Id: <199609101931.DAA00413@spinner.DIALix.COM> X-Mailer: exmh version 1.6.7 5/3/96 To: Steve Passe cc: smp@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Tue, 10 Sep 1996 12:44:22 CST." <199609101844.MAA04340@clem.systemsix.com> Date: Wed, 11 Sep 1996 03:31:55 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Steve Passe wrote: > Hi, > > > page is stolen). I've been trying to decipher some of the more obscure > > parts of the apic docs, and it appears that we can sort-of simulate a > > So far I've been working with just the APIC info in the MP spec and the > pentium manual. I just spent a half-hour playing phone tag with Intel > trying to get some 82489 specific data books. They can't seem to > find the proper documents. Could you tell me the document names & numbers > of the books you have/know of? > > thanx in advance... Ha! If you find out, **please** let me know! There seems to be no documentation for the IO APIC available anywhere, apart from the fact that it's got address space reserved for it. As far as intel is concerned, it appears as though the 82489DX practically never existed, and the vague reference "See your chipset documentation" for the IO APIC specs isn't much help either. Does that mean we need to get hold of a Neptune or the 430HX-formerly-known-as-Triton-II chipset manuals?? Or is this one of those areas that the BIOS is meant to program and the OS is to keep it's grubby hands off? Somebody once mentioned to me some unrelated manual that had a secion on the IO apic, but I've lost the mail and cannot remember who it was. It would be really, really nice if Intel could grab the apic section of one of their "chipset manuals" and stick it on their web server in .pdf format next to the pentium and pentium pro manuals. (non-encrypted pdf please, it's a royal pain having to go to a windows machine and print the damn thing from acrobat to a fake postscript printer to capture the .ps) Anybody got any friends inside Intel? :-) BTW, the MPSPEC 1.1 doc says: 82489DX Advanced Programmable Interrupt Controller (data book), Intel order number 290446. Let me know if you had better luck than I did... > -- > Steve Passe | powered by > smp@csn.net | FreeBSD Cheers, -Peter From owner-freebsd-smp Tue Sep 10 15:12:04 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA18579 for smp-outgoing; Tue, 10 Sep 1996 15:12:04 -0700 (PDT) Received: from GndRsh.aac.dev.com (GndRsh.aac.dev.com [198.145.92.241]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id PAA18564; Tue, 10 Sep 1996 15:11:56 -0700 (PDT) Received: (from rgrimes@localhost) by GndRsh.aac.dev.com (8.6.12/8.6.12) id PAA20594; Tue, 10 Sep 1996 15:11:23 -0700 From: "Rodney W. Grimes" Message-Id: <199609102211.PAA20594@GndRsh.aac.dev.com> Subject: Re: 82489 data books To: peter@spinner.dialix.com (Peter Wemm) Date: Tue, 10 Sep 1996 15:11:23 -0700 (PDT) Cc: smp@csn.net, smp@freebsd.org, davidg@freebsd.org In-Reply-To: <199609101931.DAA00413@spinner.DIALix.COM> from Peter Wemm at "Sep 11, 96 03:31:55 am" X-Mailer: ELM [version 2.4ME+ PL11 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Steve Passe wrote: > > Hi, > > > > > page is stolen). I've been trying to decipher some of the more obscure > > > parts of the apic docs, and it appears that we can sort-of simulate a > > > > So far I've been working with just the APIC info in the MP spec and the > > pentium manual. I just spent a half-hour playing phone tag with Intel > > trying to get some 82489 specific data books. They can't seem to > > find the proper documents. Could you tell me the document names & numbers > > of the books you have/know of? > > > > thanx in advance... > > Ha! If you find out, **please** let me know! There seems to be no > documentation for the IO APIC available anywhere, apart from the fact that > it's got address space reserved for it. As far as intel is concerned, it > appears as though the 82489DX practically never existed, and the vague > reference "See your chipset documentation" for the IO APIC specs isn't > much help either. Does that mean we need to get hold of a Neptune or the > 430HX-formerly-known-as-Triton-II chipset manuals?? The Neptune databook _might_ have it, Triton-II defanitly does not. > Or is this one of those areas that the BIOS is meant to program and the OS > is to keep it's grubby hands off? No, the OS needs to reprogram the I/O APIC if it wishes to use fully symtrical interrupt delivery. > Somebody once mentioned to me some unrelated manual that had a secion on > the IO apic, but I've lost the mail and cannot remember who it was. It is in one of the volumes in the Pentium databook set, I just sent David Greenman home with an old set that I know has the 82489 datasheet in it. > It would be really, really nice if Intel could grab the apic section of > one of their "chipset manuals" and stick it on their web server in .pdf > format next to the pentium and pentium pro manuals. (non-encrypted pdf > please, it's a royal pain having to go to a windows machine and print the > damn thing from acrobat to a fake postscript printer to capture the .ps) > Anybody got any friends inside Intel? :-) Got friends inside, but they just have hard copies of the full databook sets. > BTW, the MPSPEC 1.1 doc says: 82489DX Advanced Programmable Interrupt > Controller (data book), Intel order number 290446. Let me know if you had > better luck than I did... You mean when you called 1-800-548-4725 they could not find that order number? Humm... item is discontiuned, that means that it has been folded into one of the 10 or so volumes of the full data book set and is only avaliable that way. If you got a current set of the Pentium databooks (3 or 4 volume set) you would find it in there. -- Rod Grimes rgrimes@gndrsh.aac.dev.com Accurate Automation Company Reliable computers for FreeBSD From owner-freebsd-smp Tue Sep 10 15:45:07 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA20840 for smp-outgoing; Tue, 10 Sep 1996 15:45:07 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id PAA20824; Tue, 10 Sep 1996 15:45:03 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id QAA05579; Tue, 10 Sep 1996 16:44:10 -0600 Message-Id: <199609102244.QAA05579@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: "Rodney W. Grimes" cc: peter@spinner.dialix.com (Peter Wemm), smp@csn.net, smp@freebsd.org, davidg@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Tue, 10 Sep 1996 15:11:23 PDT." <199609102211.PAA20594@GndRsh.aac.dev.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 10 Sep 1996 16:44:09 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > You mean when you called 1-800-548-4725 they could not find that order > number? Humm... item is discontiuned, that means that it has been folded > into one of the 10 or so volumes of the full data book set and is > only avaliable that way. If you got a current set of the Pentium databooks > (3 or 4 volume set) you would find it in there. can't remember where I started, ended up at 800 628-8686, anyways Intel called back, she found the last reference to the 82489 on an old 1993 CD-ROM. It was originally designed to use with the 80486 and is now discontinued. The IO APIC we need to deal with is the one in the TritonII, ie 430HX chip set, part# '82371SB (PIIX3)'. I found *.pdf files for these as well as discrete versions of the APIC. xpdf (in ports) can display these and produce .ps files that are 99% there (a few pages go into/outof the wrong font size). I haven't yet had time to actually read them to tell you which has the best coverage of the APIC. =============================================================================== Data-Sheets relavant to the FreeBSD SMP kernel ------------------ The MP spec, v1.4: http://www.intel.com/IAL/processr/mpovr.htm http://www.intel.com/design/pro/prodspec/multipro.htm (24201604.pdf) ------------------------- Intel PCIsets-Datasheets: http://www.intel.com/design/pcisets/datashts/index.htm -- The 82093AA I/O Advanced Programmable Interrupt Controller (IOAPIC): http://www.intel.com/design/pcisets/datashts/82093aa.htm (29056601.pdf) -- The 82378ZB System I/O (SIO) and 82379AB System I/O APIC (SIO.A): http://www.intel.com/design/pcisets/datashts/82378zb.htm (29057101.pdf) -- The 82439HX System Controller (TXC) (1st half of 430HX): http://www.intel.com/design/pcisets/datashts/inte2.htm (29055101.pdf) -- The 82371SB (PIIX3) (2nd half of 430HX): http://www.intel.com/design/pcisets/datashts/82371fb.htm (29055001.pdf) -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Tue Sep 10 20:03:58 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id UAA03582 for smp-outgoing; Tue, 10 Sep 1996 20:03:58 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id UAA03576 for ; Tue, 10 Sep 1996 20:03:52 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id VAA06791 for ; Tue, 10 Sep 1996 21:03:49 -0600 Message-Id: <199609110303.VAA06791@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: smp@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Tue, 10 Sep 1996 16:44:09 MDT." <199609102244.QAA05579@clem.systemsix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Tue, 10 Sep 1996 21:03:48 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, those previous data sheets I cited were a disappointment, not much meat t= here. here's another document with useful info, chapter 4 deals with MP BIOS se= tup: Pentium=AE Pro Processor BIOS Writer's Guide V2.0 = http://www.intel.com/IAL/processr/p6/pppbios.pdf -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Tue Sep 10 22:49:48 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id WAA13587 for smp-outgoing; Tue, 10 Sep 1996 22:49:48 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id WAA13582 for ; Tue, 10 Sep 1996 22:49:45 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id XAA07579 for ; Tue, 10 Sep 1996 23:49:42 -0600 Message-Id: <199609110549.XAA07579@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: smp@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Tue, 10 Sep 1996 21:03:48 MDT." <199609110303.VAA06791@clem.systemsix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 10 Sep 1996 23:49:41 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, -- an interesting tutorial from IBM that covers cache issues as well as APIC and 'OpenAPIC': X86 Multiprocessing Basics: http://www.chips.ibm.com/products/x86/appnote/40208.ps -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Wed Sep 11 09:02:08 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id JAA11235 for smp-outgoing; Wed, 11 Sep 1996 09:02:08 -0700 (PDT) Received: from news1.gtn.com (news1.gtn.com [192.109.159.3]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id JAA11204 for ; Wed, 11 Sep 1996 09:01:22 -0700 (PDT) Received: (from uucp@localhost) by news1.gtn.com (8.7.2/8.7.2) id RAA03894 for smp@freebsd.org; Wed, 11 Sep 1996 17:45:31 +0200 (MET DST) Received: from localhost (localhost [127.0.0.1]) by klemm.gtn.com (8.7.5/8.7.3) with SMTP id RAA00615 for ; Wed, 11 Sep 1996 17:42:22 +0200 (MET DST) Date: Wed, 11 Sep 1996 17:42:21 +0200 (MET DST) From: Andreas Klemm To: smp@freebsd.org Subject: Will FreeBSD with smp patches support Tyan Tomcat II board with 2 CPUs ? Message-ID: X-try-apsfilter: ftp://sunsite.unc.edu/pub/Linux/system/Printing/aps-491.tgz X-Fax: +49 2137 2018 X-Phone: +49 2137 2020 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Additionally to the subject, could someone please set me on this list ?! Didn't receive a greetings mail, only got information, that my request was sent to the owner of this list ... Thanks Andreas /// andreas@klemm.gtn.com /\/\___ Wiechers & Partner Datentechnik GmbH Andreas Klemm ___/\/\/ Support Unix -- andreas.klemm@wup.de pgp p-key http://www-swiss.ai.mit.edu/~bal/pks-toplev.html >>> powered by <<< ftp://sunsite.unc.edu/pub/Linux/system/Printing/aps-491.tgz >>> FreeBSD <<< From owner-freebsd-smp Wed Sep 11 09:43:01 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id JAA14083 for smp-outgoing; Wed, 11 Sep 1996 09:43:01 -0700 (PDT) Received: (from jmb@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id JAA14072; Wed, 11 Sep 1996 09:42:58 -0700 (PDT) From: "Jonathan M. Bresler" Message-Id: <199609111642.JAA14072@freefall.freebsd.org> Subject: Re: Will FreeBSD with smp patches support Tyan Tomcat II board with 2 CPUs ? To: andreas@klemm.gtn.com (Andreas Klemm) Date: Wed, 11 Sep 1996 09:42:58 -0700 (PDT) Cc: smp@freebsd.org In-Reply-To: from "Andreas Klemm" at Sep 11, 96 05:42:21 pm X-Mailer: ELM [version 2.4 PL24] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Andreas Klemm wrote: > > Additionally to the subject, could someone please set me on this > list ?! Didn't receive a greetings mail, only got information, that > my request was sent to the owner of this list ... when you try to subscribe to a list using an address that is *not* identical to the address you are sending the mail from. the request gets forwarded to me for approval. i do all the approvals once a day (generally). there must be an approval process (when the addresses dont mach) to prevent errors, mischief, cross-list-subscriptions and a host of other problems. jmb -- Jonathan M. Bresler FreeBSD Postmaster jmb@FreeBSD.ORG FreeBSD--4.4BSD Unix for PC clones, source included. http://www.freebsd.org/ PGP 2.6.2 Fingerprint: 31 57 41 56 06 C1 40 13 C5 1C E3 E5 DC 62 0E FB From owner-freebsd-smp Wed Sep 11 11:40:48 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA20872 for smp-outgoing; Wed, 11 Sep 1996 11:40:48 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA20862 for ; Wed, 11 Sep 1996 11:40:42 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id MAA11277; Wed, 11 Sep 1996 12:40:22 -0600 Message-Id: <199609111840.MAA11277@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Peter Wemm cc: smp@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Wed, 11 Sep 1996 03:31:55 +0800." <199609101931.DAA00413@spinner.DIALix.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 11 Sep 1996 12:40:22 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >> find the proper documents. Could you tell me the document names & numbers >> of the books you have/know of? > ... > Ha! If you find out, **please** let me know! I found it on the shelf @ Softpro Books: Intel486(TM) MicroProcessors and related Products, order #241731-002 pp 4-220 thru 4-302: 82489DX Adnavced Programmable Interrupt Controller > ... As far as intel is concerned, it > appears as though the 82489DX practically never existed, and the vague This book is dated 1995 on the spine, and the 82489DX pages are marked "Preliminary". The bottom of the 82489DX pages say "October 1993, order Number: 290446-002", a # we already know is useless. --- I also found a 45 page chapter on the APIC in: Pentium Processor System Architecture, 2nd edition PC System Architecture Series, MindShare Inc. ISBN 0-201-40992-5 -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Wed Sep 11 13:54:30 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id NAA29532 for smp-outgoing; Wed, 11 Sep 1996 13:54:30 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id NAA29527 for ; Wed, 11 Sep 1996 13:54:26 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id OAA11976; Wed, 11 Sep 1996 14:54:05 -0600 Message-Id: <199609112054.OAA11976@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: rv@groa.uct.ac.za (Russell Vincent) cc: freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Sat, 10 Sep 1996 11:26:52 +0200." Mime-Version: 1.0 Content-Type: text/plain Date: Wed, 11 Sep 1996 14:54:05 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, I have submitted formal bug reports via send-pr for the following 4 issues: ------------------------------------------------------------------------------- Subject: SMP kernel fix 960909.1 --- It has the internal identification `kern/1591'. >Category: kern >Responsible: freebsd-bugs >Synopsis: i386/i386/mpcore.s stores _mpfps at incorrect address ------------------------------------------------------------------------------- Subject: SMP kernel fix 960909.2 --- It has the internal identification `kern/1592'. >Category: kern >Responsible: freebsd-bugs >Synopsis: kernel incorrectly reads CPU # from APIC ID register ------------------------------------------------------------------------------- Subject: SMP kernel fix 960909.3 --- It has the internal identification `kern/1593'. >Category: kern >Responsible: freebsd-bugs >Synopsis: i386/i386/locore.s contains useless line ------------------------------------------------------------------------------- Subject: SMP kernel fix 960909.4 --- It has the internal identification `kern/1594'. >Category: kern >Responsible: freebsd-bugs >Synopsis: apic_startup() needs work ------------------------------------------------------------------------------- =============================================================================== In addition to the kernel patches enumerated in send-pr reports `kern/1592' and `kern/1594' the following patches need to be done for the XXPRESS to run the SMP kernel. They are just band-aids and as such I didn't feel they should be submitted to send-pr. The correct solution will require some sort of 'physical' to 'virtual' mapping table for IDs. The basis of the problem is that the kernel expects the BSP to have ID #0 and the AP to have ID #1. The Intel XXPRESS numbers it's CPUs 0 (BSP), 2, 3 and 4. These band-aids merely shift 2,3,4 to 1,2,3. You also need to set the BIOS to use cache 'write-thru' as oppossed to cache 'write-back' to make the XXPRESS happy. ------------------------------------------------------------------------------- --- sys/i386/include/smp.h, line 25, change: from: return (apic_base[APIC_ID] >> 24); to: /* XPPRESS numbers CPUs 0, 2, 3, 4 */ unsigned int num = ((apic_base[APIC_ID] >> 24) & 0xf); return num ? num-1 : 0; --- sys/i386/include/smpasm.h: add: #define CPUNBR 0x02000000 add: /* XPPRESS numbers CPUs 0, 2, 3, 4 */ #define MODIFY_XXPRESS_ID(reg) \ andl $0x0f000000, reg; \ je 9f; \ shrl $24,reg; \ decl reg; \ shll $24,reg; \ 9: nop change "GETCPUID()" macro to: #define GETCPUID(reg) \ movl _apic_base, reg; \ movl APIC_ID(reg), reg; \ shrl $24, reg; \ andl $15, reg; \ je 9f; \ decl reg; \ 9: nop --- sys/i386/i386/locore.s: line 268, change: andl $0xff000000, %eax to: MODIFY_XXPRESS_ID(%eax) --- sys/i386/i386/swtch.s: line 452, change: andl $0xff000000, %eax to: MODIFY_XXPRESS_ID(%eax) --- sys/i386/i386/mplock.s: line 48, change: andl $0xff000000, %ecx to: MODIFY_XXPRESS_ID(%ecx) line 59, change: andl $0xff000000, %ecx to: MODIFY_XXPRESS_ID(%ecx) line 87, change: andl $0xff000000, %ecx to: MODIFY_XXPRESS_ID(%ecx) line 99, change: andl $0xff000000, %ecx to: MODIFY_XXPRESS_ID(%ecx) =============================================================================== The last issue in the XXPRESS port was the "less code" works better bug. Russel has since discovered that changing the cache option to "write-thru" fixes this so I will not bother to descibe those experiments. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Wed Sep 11 14:02:06 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA00654 for smp-outgoing; Wed, 11 Sep 1996 14:02:06 -0700 (PDT) Received: from critter.tfs.com ([140.145.230.177]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id OAA00633; Wed, 11 Sep 1996 14:02:01 -0700 (PDT) Received: from critter.tfs.com (localhost.tfs.com [127.0.0.1]) by critter.tfs.com (8.7.5/8.7.3) with ESMTP id XAA04596; Wed, 11 Sep 1996 23:01:22 +0200 (MET DST) To: Steve Passe cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Wed, 11 Sep 1996 14:54:05 MDT." <199609112054.OAA11976@clem.systemsix.com> Date: Wed, 11 Sep 1996 23:01:22 +0200 Message-ID: <4594.842475682@critter.tfs.com> From: Poul-Henning Kamp Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk In message <199609112054.OAA11976@clem.systemsix.com>, Steve Passe writes: >Hi, > >I have submitted formal bug reports via send-pr for the following 4 issues: and as "punishment" we're thinking of offering you commit access to the central smp tree so we don't have to do anything about it :-) How's that ? -- Poul-Henning Kamp | phk@FreeBSD.ORG FreeBSD Core-team. http://www.freebsd.org/~phk | phk@login.dknet.dk Private mailbox. whois: [PHK] | phk@ref.tfs.com TRW Financial Systems, Inc. Future will arrive by its own means, progress not so. From owner-freebsd-smp Wed Sep 11 14:44:26 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA03872 for smp-outgoing; Wed, 11 Sep 1996 14:44:26 -0700 (PDT) Received: (from peter@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA03857 for freebsd-smp; Wed, 11 Sep 1996 14:44:19 -0700 (PDT) Date: Wed, 11 Sep 1996 14:44:19 -0700 (PDT) From: Peter Wemm Message-Id: <199609112144.OAA03857@freefall.freebsd.org> To: freebsd-smp Subject: cvs commit: sys/i386/isa clock.c random_machdep.c sys/i386/i386 db_trace.c identcpu.c microtime.s sys/i386/include clock.h Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk peter 96/09/11 14:44:18 Modified: i386/isa clock.c random_machdep.c i386/i386 db_trace.c identcpu.c microtime.s i386/include clock.h Log: More completely disable the use of the pentium cycle counter under SMP, this needs a lot more work before it can be used since each cycle counter is different on each cpu. It will need to be calibrated on each cpu, the base offset kept, and then somehow they will need to be kept in sync. For the time being, it's far easier to just simply ignore it and pretend that we're a 486 for the clock code and use the 8254 counter. Revision Changes Path 1.2 +425 -109 sys/i386/isa/clock.c 1.2 +10 -12 sys/i386/isa/random_machdep.c 1.2 +56 -22 sys/i386/i386/db_trace.c 1.2 +23 -5 sys/i386/i386/identcpu.c 1.8 +1 -1 sys/i386/i386/microtime.s 1.2 +28 -19 sys/i386/include/clock.h From owner-freebsd-smp Wed Sep 11 14:56:28 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA05588 for smp-outgoing; Wed, 11 Sep 1996 14:56:28 -0700 (PDT) Received: (from peter@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA05561 for freebsd-smp; Wed, 11 Sep 1996 14:56:24 -0700 (PDT) Date: Wed, 11 Sep 1996 14:56:24 -0700 (PDT) From: Peter Wemm Message-Id: <199609112156.OAA05561@freefall.freebsd.org> To: freebsd-smp Subject: cvs commit: sys/sys resourcevar.h sys/miscfs/procfs procfs_status.c sys/kern init_main.c kern_acct.c kern_exit.c kern_resource.c kern_shutdown.c kern_synch.c tty.c Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk peter 96/09/11 14:56:22 Modified: sys resourcevar.h kern init_main.c kern_acct.c kern_exit.c kern_resource.c kern_shutdown.c kern_synch.c tty.c miscfs/procfs procfs_status.c Log: Most of this came about as a result of trying to figure out WTF's going on with the negative times in calcru(). I added an extra arg for calcru to use in it's diagnostic message. It appears that it's somehow related to the way we fork our processes and connect them to the run queues relative to the start time we set. The most often case of failure was for very short lived processes that exited very quickly, eg: find /usr -exec echo {} \; Eventually I gave up and commented the warning out under a #ifndef SMP block. Also, do some work on the smp_idleloop(). We were breaking a LOT of rules in there, including modifying the run queues with no spl protection, calling code without raising to the required spl level, etc. It's a wonder things worked at all...... :-] Attempt to allow safer shutdowns, but I'm not sure I've got it right. If boot() is called on cpu != #0, it sets a trap to catch non-zero cpus and repeatedly sleeps, hoping that the other cpu will schedule the process. I do not recall if I got this working, I must check it again. (cpu#1 etc cannot shut the system down, as they do not have interrupts enabled, so they cannot complete the disk IO, and hence do not clean unmount the filesystems.) Revision Changes Path 1.2 +2 -8 sys/sys/resourcevar.h 1.21 +20 -3 sys/kern/init_main.c 1.2 +1 -1 sys/kern/kern_acct.c 1.7 +1 -1 sys/kern/kern_exit.c 1.2 +11 -10 sys/kern/kern_resource.c 1.3 +18 -0 sys/kern/kern_shutdown.c 1.10 +5 -4 sys/kern/kern_synch.c 1.2 +8 -6 sys/kern/tty.c 1.2 +1 -1 sys/miscfs/procfs/procfs_status.c From owner-freebsd-smp Wed Sep 11 15:06:33 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA06483 for smp-outgoing; Wed, 11 Sep 1996 15:06:33 -0700 (PDT) Received: (from peter@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA06467 for freebsd-smp; Wed, 11 Sep 1996 15:06:29 -0700 (PDT) Date: Wed, 11 Sep 1996 15:06:29 -0700 (PDT) From: Peter Wemm Message-Id: <199609112206.PAA06467@freefall.freebsd.org> To: freebsd-smp Subject: cvs commit: sys/i386/i386 locore.s Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk peter 96/09/11 15:06:28 Modified: i386/i386 locore.s Log: remove bogus line Submitted by: Steve Passe , PR#1593 Revision Changes Path 1.24 +0 -1 sys/i386/i386/locore.s From owner-freebsd-smp Wed Sep 11 15:09:33 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA06722 for smp-outgoing; Wed, 11 Sep 1996 15:09:33 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id PAA06693 for ; Wed, 11 Sep 1996 15:09:16 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id GAA06869; Thu, 12 Sep 1996 06:08:33 +0800 (WST) Message-Id: <199609112208.GAA06869@spinner.DIALix.COM> X-Mailer: exmh version 1.6.7 5/3/96 To: Peter Wemm cc: smp@freebsd.org, CVS-committers@freefall.freebsd.org, cvs-all@freefall.freebsd.org, cvs-other@freefall.freebsd.org Subject: Re: cvs commit: /home/smp/sys/i386/i386 mpcore.s In-reply-to: Your message of "Wed, 11 Sep 1996 15:01:07 MST." <199609112201.PAA05973@freefall.freebsd.org> Date: Thu, 12 Sep 1996 06:08:32 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Peter Wemm wrote: > peter 96/09/11 15:01:06 > > Modified: home/smp/sys/i386/i386 mpcore.s > Log: > store mpfps in the correct place > > Submitted by: Steve Passe , PR#1591 > > Revision Changes Path > 1.15 +1 -1 /home/smp/sys/i386/i386/mpcore.s Don't panic, this was not in the real CVS tree, it was a remote-cvs command line hiccup. this was actually committed to the SMP tree... Cheers, -Peter From owner-freebsd-smp Wed Sep 11 15:38:52 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA08791 for smp-outgoing; Wed, 11 Sep 1996 15:38:52 -0700 (PDT) Received: (from peter@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA08783 for freebsd-smp; Wed, 11 Sep 1996 15:38:50 -0700 (PDT) Date: Wed, 11 Sep 1996 15:38:50 -0700 (PDT) From: Peter Wemm Message-Id: <199609112238.PAA08783@freefall.freebsd.org> To: freebsd-smp Subject: cvs commit: sys/i386/i386 locore.s mplock.s swtch.s sys/i386/include apic.h smp.h smpasm.h Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk peter 96/09/11 15:38:49 Modified: i386/i386 locore.s mplock.s swtch.s i386/include apic.h smp.h smpasm.h Log: Only use the 4 bits of the apic ID, not the extras. Submitted by: Steve Passe , PR#1592 Revision Changes Path 1.25 +1 -1 sys/i386/i386/locore.s 1.12 +5 -5 sys/i386/i386/mplock.s 1.22 +1 -1 sys/i386/i386/swtch.s 1.2 +4 -1 sys/i386/include/apic.h 1.6 +2 -2 sys/i386/include/smp.h 1.4 +1 -0 sys/i386/include/smpasm.h From owner-freebsd-smp Wed Sep 11 19:08:00 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA22276 for smp-outgoing; Wed, 11 Sep 1996 19:08:00 -0700 (PDT) Received: (from peter@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA22269 for freebsd-smp; Wed, 11 Sep 1996 19:07:57 -0700 (PDT) Date: Wed, 11 Sep 1996 19:07:57 -0700 (PDT) From: Peter Wemm Message-Id: <199609120207.TAA22269@freefall.freebsd.org> To: freebsd-smp Subject: cvs commit: sys/i386/include smp.h Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk peter 96/09/11 19:07:56 Modified: i386/include smp.h Log: oops, typo. Revision Changes Path 1.7 +1 -1 sys/i386/include/smp.h From owner-freebsd-smp Wed Sep 11 23:48:24 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id XAA15535 for smp-outgoing; Wed, 11 Sep 1996 23:48:24 -0700 (PDT) Received: from wave.cyberbeach.net (wave.cyberbeach.net [205.150.79.11]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id XAA15530 for ; Wed, 11 Sep 1996 23:48:22 -0700 (PDT) Received: from www (sail.cyberbeach.net [205.150.79.24]) by wave.cyberbeach.net (8.7.5/8.7.3) with SMTP id CAA00532 for ; Thu, 12 Sep 1996 02:48:21 -0400 (EDT) Message-Id: <1.5.4.32.19960912184920.008bcb20@post.cyberbeach.net> X-Sender: kurt@post.cyberbeach.net X-Mailer: Windows Eudora Light Version 1.5.4 (32) Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Thu, 12 Sep 1996 14:49:20 -0400 To: freebsd-smp@freebsd.org From: Kurt Schafer Subject: subscribe Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk subscribe freebsd-smp From owner-freebsd-smp Wed Sep 11 23:48:14 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id XAA15528 for smp-outgoing; Wed, 11 Sep 1996 23:48:14 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id XAA15519 for ; Wed, 11 Sep 1996 23:48:03 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id OAA02113; Thu, 12 Sep 1996 14:40:42 +0800 (WST) Message-Id: <199609120640.OAA02113@spinner.DIALix.COM> X-Mailer: exmh version 1.6.7 5/3/96 To: Steve Passe cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Wed, 11 Sep 1996 14:54:05 CST." <199609112054.OAA11976@clem.systemsix.com> Date: Thu, 12 Sep 1996 14:40:41 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Steve Passe wrote: > Hi, > > I have submitted formal bug reports via send-pr for the following 4 issues: > [Applied, thanks] > Subject: SMP kernel fix 960909.4 > --- > It has the internal identification `kern/1594'. > >Category: kern > >Responsible: freebsd-bugs > >Synopsis: apic_startup() needs work Hmm, something seems to have gone wrong, my second CPU isn't found with this patch present, so I'll see if I can see what went wrong.. [..] > feel they should be submitted to send-pr. The correct solution will > require some sort of 'physical' to 'virtual' mapping table for IDs. > > The basis of the problem is that the kernel expects the BSP to have ID #0 > and the AP to have ID #1. The Intel XXPRESS numbers it's CPUs 0 (BSP), > 2, 3 and 4. These band-aids merely shift 2,3,4 to 1,2,3. I have a better solution in mind. :-) Hint: ALL the APIC ID registers are read/write. Can you see it yet? :-) Apart from the existance of the XXPRESS board, the nice easy solution is to have the boot cpu assign it's ID to zero, and send a broadcast STARTUP IPI and have the application cpu's fight for a lock, and assign themselves sequentially increasing ID's, and assign the IO apic's ID to the end of the list. One thing I'm not clear about from the IO apic docs yet is whether there are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on the APIC bus. BTW, those .pdf docs you pointed out have done the trick and describe the IO apic (version 0x11 at least :-) completely, although it is scattered around different doc files a bit.. As I understand it, the 82371SB PIIX3 chip has an IO apic address decoder, but no apic. The IOAPIC chip (82093AA) is an optional extra part of the 430HX suite and would only be present on the multi-processor boards. I'm not sure where the 82378ZB SIO and 82379AB SIO.A fit into this picture, but it looks to me like they are a standalone version of the "complete motherboard chipset". All the information is there... The IO apic has only two memory mapped registers, one address, one data. The internal registers, selected and accessed through the visible pair, look (strangely enough) very much like the registers in the local apic..... They could be very easily missed or mistaken for local apic docs. The IO redirection table entries (what we're looking for) are the same layout as the Local APIC's ICR, and very similar to the 3 local vector table entries for LINT0, LINT1 and ERROR. Probably no accident at all.. > You also need to set the BIOS to use cache 'write-thru' as oppossed to > cache 'write-back' to make the XXPRESS happy. Incidently, in one of the books that I did find when I was looking some time ago, it mentions that with the MESI caches in action, write-through was "for speed" (to avoid cache conflict resolution cycles or something) versus write-back and may be required for correct multiprocessor operation.. I'm not sure I trust the book on that, as it's only got a very slim section on the apic and multiprocessing (no detail at all) and is quite cryptic and (I think) self contradicting in places. Naturally I cannot find the reference, so I'm denying this comment exists. :-) Cheers, -Peter From owner-freebsd-smp Thu Sep 12 01:15:26 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id BAA21272 for smp-outgoing; Thu, 12 Sep 1996 01:15:26 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id BAA21261 for ; Thu, 12 Sep 1996 01:15:21 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id CAA15528; Thu, 12 Sep 1996 02:12:56 -0600 Message-Id: <199609120812.CAA15528@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Peter Wemm cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 14:40:41 +0800." <199609120640.OAA02113@spinner.DIALix.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Sep 1996 02:12:56 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > Hmm, something seems to have gone wrong, my second CPU isn't found with > this patch present, so I'll see if I can see what went wrong.. I stripped out some debug while making the report, perhaps I broke something. I am currently rebuilding the world, I will apply the reported patch and verify whether it works when I've finished supping/building SMP. For my information, what motherboard are you using? --- > Hint: ALL the APIC ID registers are read/write. > > Can you see it yet? :-) we tried that on the XXPRESS and (Russel, please confirm this) an instant reset of the hardware. My mail notes show: ----------------------------- mail from Russel ------------------------------ > one last experiment. the book says that you can write the local apic id. > so the locore.s I just sent does that. you need to turn OFF "#define XXPRESS" > in i386/include/smpasm.h (and anywhere else you have set it). specifically it > causes the 2nd CPU to change its apic ID from 2 to 1 during its early > bootup code, so we still want CPUNUMBER=0x02000000 for the 1st CPU INIT IPI > code, but after that everything should be in the context of 2nd CPU apic ID=1. I removed my 'options XXPRESS' line and did a 'config SMP' and re-compiled with the new locore.s. Now the machine reboots as soon as I do the sysctl. ------------------------------- end mail notes -------------------------------- We could have had another problem at this point so further experimentation wouldn't hurt. --- >Apart from the existance of the XXPRESS board, the nice easy solution is >to have the boot cpu assign it's ID to zero, and send a broadcast STARTUP >IPI and have the application cpu's fight for a lock, and assign themselves >sequentially increasing ID's, and assign the IO apic's ID to the end of >the list. my databook says: 110( Startup) This delivery mode is used as a special message between two processors ... ie, are you sure you can do broadcast 'Startup IPIs'? as an alternative I would suggest starting to use the MP table: #define MAX_CPU 16 int SMPIDMap[ MAX_CPU ]; int nextCpu( void** fps ) { /* finds next cpu record in MP table @ fps */ } infoFromMPTable() { int mp_ncpus = 0; void* fps = mpfps; while ( nextCpu( &fps ) ) { id = CpuId( fps ) SMPIDMap[ id ] = mp_ncpus++; } } cpunumber() { return SMPIDMap[ GETCPUID() ]; } I've already written code that dumps the entire table, I could produce routines like nextCpu() in quick order. another issue to consider is that when you write the APIC ID register you also write its ARB (arbitration) register. Since I haven't yet deciphered the arbitration stuff I can't say whether this is an issue... --- >One thing I'm not clear about from the IO apic docs yet is whether there >are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on >the APIC bus. my belief is that it is a total of 15. the true 82489 has an 8-bit register, but the IO APIC in 586/686 has a 4-bit field as does the 82379AB and 82093AA. On my mb the IO APIC is numbered 2, while the XXPRESS numbers it 14. >BTW, those .pdf docs you pointed out have done the trick and describe the .IO apic (version 0x11 at least :-) completely, although it is scattered >around different doc files a bit.. did you get the book info I mailed earlier today about finding the true datasheet for the 82489? >As I understand it, the 82371SB PIIX3 chip has an IO apic address decoder, >but no apic. The IOAPIC chip (82093AA) is an optional extra part of the >430HX suite and would only be present on the multi-processor boards. I'm good eye, I didn't see this detail on my first scan of that doc. -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Thu Sep 12 04:02:03 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id EAA28662 for smp-outgoing; Thu, 12 Sep 1996 04:02:03 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id EAA28638 for ; Thu, 12 Sep 1996 04:01:55 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id TAA03569; Thu, 12 Sep 1996 19:00:35 +0800 (WST) Message-Id: <199609121100.TAA03569@spinner.DIALix.COM> X-Mailer: exmh version 1.6.7 5/3/96 To: Steve Passe cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 02:12:56 CST." <199609120812.CAA15528@clem.systemsix.com> Date: Thu, 12 Sep 1996 19:00:34 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Steve Passe wrote: > Hi, > > > Hmm, something seems to have gone wrong, my second CPU isn't found with > > this patch present, so I'll see if I can see what went wrong.. > > I stripped out some debug while making the report, perhaps I broke something. > I am currently rebuilding the world, I will apply the reported patch and > verify whether it works when I've finished supping/building SMP. > For my information, what motherboard are you using? 2xP90 ASUS Neptune PCI/E-P54NP4.. BTW, Some other things we do not do.. We don't set the ERROR LVT to handle a non-delivered or failed message. I noticed you took out the second STARTUP IPI.. The docs I've been reading say "the startup IPI can only be used once after a reset or INIT", and that it only has any effect once, and is ignored on the old 486 APIC, and the example code suggests two STARTUP IPI's. From that, I read that the second one is for insurance in case the first one was missed, and that the second will normally be ignored. I notice that there is no way in the world that we are waiting 10miliseconds after the initial INIT IPI.. My system is probably missing the single startup IPI. Time do do some more accurate times I think. > --- > > Hint: ALL the APIC ID registers are read/write. > > > > Can you see it yet? :-) > > we tried that on the XXPRESS and (Russel, please confirm this) an instant res et > of the hardware. My mail notes show: > ----------------------------- mail from Russel ------------------------------ > > one last experiment. the book says that you can write the local apic id. > > so the locore.s I just sent does that. you need to turn OFF "#define XXPRE SS" > > in i386/include/smpasm.h (and anywhere else you have set it). specifically it > > causes the 2nd CPU to change its apic ID from 2 to 1 during its early > > bootup code, so we still want CPUNUMBER=0x02000000 for the 1st CPU INIT IPI > > code, but after that everything should be in the context of 2nd CPU apic ID =1. > > I removed my 'options XXPRESS' line and did a 'config SMP' and > re-compiled with the new locore.s. Now the machine reboots as > soon as I do the sysctl. > > ------------------------------- end mail notes ------------------------------ -- > We could have had another problem at this point so further experimentation > wouldn't hurt. Some other interesting quotes: "The operating system is responsible for assigning non-conflicting ID's to all the IO APIC's". Is it possible that the APIC's will reset the system if there's a conflict? You probably have an IO APIC in the system, I wonder what the odds are that the BIOS has already set it to ID 1, and when you moved #2 down to ID 1 there was a M.A.D. reset. > >Apart from the existance of the XXPRESS board, the nice easy solution is > >to have the boot cpu assign it's ID to zero, and send a broadcast STARTUP > >IPI and have the application cpu's fight for a lock, and assign themselves > >sequentially increasing ID's, and assign the IO apic's ID to the end of > >the list. > > my databook says: > > 110( Startup) This delivery mode is used as a special message > between two processors ... > > ie, are you sure you can do broadcast 'Startup IPIs'? The pppbios.pdf specifically says: "The BSP sends a StartUp APIC message broadcast......" One of the various other tables in the P5 docs say that startup IPI broadcasts are always edge triggered when used in "all but self" mode, so who knows.. :-) > as an alternative I would suggest starting to use the MP table: > > #define MAX_CPU 16 > int SMPIDMap[ MAX_CPU ]; > > int nextCpu( void** fps ) { /* finds next cpu record in MP table @ fps */ } > > infoFromMPTable() > { > int mp_ncpus = 0; > void* fps = mpfps; > while ( nextCpu( &fps ) ) > { > id = CpuId( fps ) > SMPIDMap[ id ] = mp_ncpus++; > } > } > > cpunumber() > { > return SMPIDMap[ GETCPUID() ]; > } > > I've already written code that dumps the entire table, I could produce routin es > like nextCpu() in quick order. Hmm. I'd rather avoid an extra indirection map if possible. Considering that we are #define'ing things like curproc and some other heavily used variables to lookup the cpuid for array indexes, we would better spend the effort to just deal with a sparse ID set properly. Another thought.. We do not use the timer on the apic. It has a 32 bit read/write register for the "initial count". We could cheat and use that as a 32 bit pointer to a cpu-specific data page with each cpu's scratch area etc, > another issue to consider is that when you write the APIC ID register you als o > write its ARB (arbitration) register. Since I haven't yet deciphered > the arbitration stuff I can't say whether this is an issue... Hmm.. > --- > >One thing I'm not clear about from the IO apic docs yet is whether there > >are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on > >the APIC bus. > > my belief is that it is a total of 15. > > the true 82489 has an 8-bit register, but the IO APIC in 586/686 > has a 4-bit field as does the 82379AB and 82093AA. I've read more info now, I think you're right. the 82489 also has a 4-bit bus and a different wire protocol. I think we can safely ignore it, since it's discontinued(?) and we're very unlikely to see any. We'd be far better off putting in a check for an 82489 apic in the boot code and simply panic if so. Then we can concentrate on the P5 and P6 implementation, and then once it's working on the common hardware we can consider making it run on the 486 theoretical platforms... > On my mb the IO APIC is numbered 2, while the XXPRESS numbers it 14. Ah, ok, well I guess that kills my theory about the IO apic being at #1 then. :-) > >BTW, those .pdf docs you pointed out have done the trick and describe the > .IO apic (version 0x11 at least :-) completely, although it is scattered > >around different doc files a bit.. > > did you get the book info I mailed earlier today about finding the true > datasheet for the 82489? Yes, I think so, but it's well out of my reach being in the USA. I'm happy with the 82093AA docs that you found. At least, there's enough detail there to try out a couple of things that I wanted to try. > >As I understand it, the 82371SB PIIX3 chip has an IO apic address decoder, > >but no apic. The IOAPIC chip (82093AA) is an optional extra part of the > >430HX suite and would only be present on the multi-processor boards. I'm > > good eye, I didn't see this detail on my first scan of that doc. The PIIX3 just provides address decode, it was the only reference in the docs you pointed out. :-) Cheers, -Peter From owner-freebsd-smp Thu Sep 12 11:11:05 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA19994 for smp-outgoing; Thu, 12 Sep 1996 11:11:05 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA19989 for ; Thu, 12 Sep 1996 11:11:04 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id LAA07176; Thu, 12 Sep 1996 11:07:22 -0700 From: Terry Lambert Message-Id: <199609121807.LAA07176@phaeton.artisoft.com> Subject: Re: Intel XXpress - some SMP benchmarks To: peter@spinner.dialix.com (Peter Wemm) Date: Thu, 12 Sep 1996 11:07:22 -0700 (MST) Cc: smp@csn.net, rv@groa.uct.ac.za, freebsd-smp@FreeBSD.org In-Reply-To: <199609120640.OAA02113@spinner.DIALix.COM> from "Peter Wemm" at Sep 12, 96 02:40:41 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk > One thing I'm not clear about from the IO apic docs yet is whether there > are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on > the APIC bus. 1 BP, 31 (AP | IO APIC) (2^5 == 32) > Incidently, in one of the books that I did find when I was looking some > time ago, it mentions that with the MESI caches in action, write-through > was "for speed" (to avoid cache conflict resolution cycles or something) > versus write-back and may be required for correct multiprocessor > operation.. I'm not sure I trust the book on that, as it's only got a > very slim section on the apic and multiprocessing (no detail at all) and > is quite cryptic and (I think) self contradicting in places. Naturally I > cannot find the reference, so I'm denying this comment exists. :-) Depends on whether page updates are propagated to all processors in the writeback case, or only the one that initiated the event (or handled it). Correct hardware should work with writeback. BTW, with MESI, IMO, *only* writeback asures you that you don't have to dump IPI's to the other processor if you are a processor which gets an invalidate. I suppose there is a lot of broken hardware out there, however. Means we should be prepared to handle it in software in all cases, and that writeback should be a non-default option. 8-(. There are ways to test for writeback viability, but they are about as complicated as those you would use to test DMA transfer ranges on EISA or cache notification on Saturn I, Neptune I, Mercury I, and VLB chipsets (ie: complicated and in need of driver mods for all DMA drivers). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Sep 12 11:15:34 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA20172 for smp-outgoing; Thu, 12 Sep 1996 11:15:34 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA20167 for ; Thu, 12 Sep 1996 11:15:32 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id LAA07189; Thu, 12 Sep 1996 11:12:53 -0700 From: Terry Lambert Message-Id: <199609121812.LAA07189@phaeton.artisoft.com> Subject: Re: Intel XXpress - some SMP benchmarks To: smp@csn.net (Steve Passe) Date: Thu, 12 Sep 1996 11:12:53 -0700 (MST) Cc: peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org In-Reply-To: <199609120812.CAA15528@clem.systemsix.com> from "Steve Passe" at Sep 12, 96 02:12:56 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > Hint: ALL the APIC ID registers are read/write. > > > > Can you see it yet? :-) > > we tried that on the XXPRESS and (Russel, please confirm this) an instant > reset of the hardware. Any chance that a write of the ID register acts as an INIT IPI? That's what seems to be implied. I suspect that you will need to inventory the processors, then back-fill the holes for the case where you would get an ID collision during the shuffling -- ie: if I have n processors, all APIC ID's < (n-1) are left alone, and only the remainder are rewritten. I *believe* that the BP is guranteed an APIC ID of 0. You may want to disassemble your MP cold boot BIOS code to see about the ID assignment; clearly it must be happening in BIOS in any case, since the PPRO's are "glueless" and would care about which slots they are put in, otherwise. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Sep 12 11:17:36 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA20323 for smp-outgoing; Thu, 12 Sep 1996 11:17:36 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA20314 for ; Thu, 12 Sep 1996 11:17:32 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id MAA18639; Thu, 12 Sep 1996 12:17:16 -0600 Message-Id: <199609121817.MAA18639@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Peter Wemm cc: freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 19:00:34 +0800." <199609121100.TAA03569@spinner.DIALix.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Sep 1996 12:17:16 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >BTW, Some other things we do not do.. We don't set the ERROR LVT to handle >a non-delivered or failed message. no, but my original code for the apic_startup() had checks on the APIC_ESR register, never saw errors. I will put that back later today. >I noticed you took out the second STARTUP IPI.. The docs I've been >reading say "the startup IPI can only be used once after a reset or INIT", > ... >the second one is for insurance in case the first one was missed, and that >the second will normally be ignored. I notice that there is no way in the I guess the second one couldn't hurt, but I would rather use a better means than "if I do it often enough its gotta work". A greater concern is the INIT/RESET of the 'run bootMP' flavor that the XXPRESS demanded. If we add the correct timings whats to prevent the STARTUP IPI from re-running a CPU once it has already started (via RESET), perhaps double incrementing mp_ncpus? >system is probably missing the single startup IPI. Time do do some more >accurate times I think. I agree. I added a zero to each of the timing loops compared to the original code. Realy need a usleep() of some sort. >The pppbios.pdf specifically says: "The BSP sends a StartUp APIC message >broadcast......" One of the various other tables in the P5 docs say that >startup IPI broadcasts are always edge triggered when used in "all but >self" mode, so who knows.. :-) I see that now, I'm willing to believe it might be doable. >Another thought.. We do not use the timer on the apic. It has a 32 bit >read/write register for the "initial count". We could cheat and use that >as a 32 bit pointer to a cpu-specific data page with each cpu's scratch >area etc, are you certain that we won't want to use it in the future? another issue is that erich claims accessing the APIC registers is relatively expensive time-wise. He pointed out that one should NOT write code like: do_this( cpu_number() ); do_that( cpu_number() ); but instead: id = cpu_number(); do_this( id ); do_that( id ); --- > better off putting in a check for an 82489 apic in the boot code and both mine and Russel's boards show an IO APIC version of 17. His CPUs show APIC vewrsion 16, while mine shows 17 (all numbers taken from the MP table, NOT the APIC version registers). I can't find a # for the 82379AB anywhere in its manual. >From the manuals: 82093AA: 11 82379AB: ?? 82489DX: 01 P5: 1x P6: ?? --- I got supped and "worlded" late last nite. Observations: - we seem to be out of sync with ps: # ps ps: proc size mismatch (13640 total, 632 chunks) - when I tried to halt the code realized it was running on CPU#2, claimed someing about "freezing", then did! It was big red time. the mp_lock was declared to be 01000001, so I guess the 2nd CPU froze while holding the lock. - making all in sys has problem with spl.h: # make all ===> i386/boot ===> i386/boot/biosboot cc -O2 -DDO_BAD144 -DBOOTWAIT=5000 -DTIMEOUT= -DCOMCONSOLE=0x3F8 -DBOOTSEG=0x1000 -DBOOTSTACK=0xFFF0 -c /usr/src/sys/i386/boot/biosboot/io.c In file included from /usr/include/machine/cpufunc.h:425, from /usr/src/sys/i386/boot/biosboot/io.c:31: /usr/include/machine/spl.h:39: opt_smp.h: No such file or directory *** Error code 1 copied opt_smp.h to i386/include: # make all ===> i386/boot ===> i386/boot/biosboot cc -O2 -DDO_BAD144 -DBOOTWAIT=5000 -DTIMEOUT= -DCOMCONSOLE=0x3F8 -DBOOTSEG=0x1000 -DBOOTSTACK=0xFFF0 -c /usr/src/sys/i386/boot/biosboot/io.c In file included from /usr/include/machine/cpufunc.h:425, from /usr/src/sys/i386/boot/biosboot/io.c:31: /usr/include/machine/spl.h:39: opt_smp.h: No such file or directory *** Error code 1 checked current state of i386/include: # grep opt_smp *h pcb.h:#include "opt_smp.h" smp.h:#include "opt_smp.h" spl.h:#include changing the line in spl.h to: #include "opt_smp.h" fixes that. still have a problem in that the makefiles for biosboot etc. don't search the right path, ie opt_smp is only going to be in the kernel compile specific directory, is it not? THis only worked because I copied opt_smp.h to i386/include. --- FYI, Heres my (GA586DX512) MP table: -------------------------------------------------------------------------- MP Floating Pointer Structure: location: BIOS physical address: 0x000f0c80 signature: '_MP_' length: 16 bytes version: 1.1 checksum: 0xf4 mode: Virtual Wire -------------------------------------------------------------------------- MP Config Table Header: physical address: 0x000f0c94 signature: 'PCMP' base table length: 292 version: 1.1 checksum: 0x31 OEM ID: 'OEM00000' Product ID: 'PROD00000000' OEM table pointer: 0x00000000 OEM table size: 0 entry count: 28 local APIC address: 0xfee00000 extended table length: 0 extended table checksum: 0 -------------------------------------------------------------------------- MP Config Base Table Entries: -- Processor apic ID: 0, version: 17 CPU is usable, CPU is the bootstrap processor family: 5, model: 2, stepping: 1 feature flags: 0x000007bf -- Processor apic ID: 1, version: 17 CPU is usable, CPU is NOT the bootstrap processor family: 5, model: 2, stepping: 1 feature flags: 0x000007bf -- Bus bus ID: 0, bus type: ISA -- Bus bus ID: 1, bus type: PCI -- I/O APIC apic ID: 2, version: 17 APIC is usable apic address: 0xfec00000 -- I/O INT INT type: 3, flags: 0x0000 source bus ID: 0, IRQ: 0 destination APIC ID: 2, INT: 0 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 1 destination APIC ID: 2, INT: 1 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 0 destination APIC ID: 2, INT: 2 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 3 destination APIC ID: 2, INT: 3 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 4 destination APIC ID: 2, INT: 4 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 5 destination APIC ID: 2, INT: 5 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 6 destination APIC ID: 2, INT: 6 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 7 destination APIC ID: 2, INT: 7 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 8 destination APIC ID: 2, INT: 8 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 9 destination APIC ID: 2, INT: 9 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 10 destination APIC ID: 2, INT: 10 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 11 destination APIC ID: 2, INT: 11 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 12 destination APIC ID: 2, INT: 12 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 13 destination APIC ID: 2, INT: 13 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 14 destination APIC ID: 2, INT: 14 -- I/O INT INT type: 0, flags: 0x0000 source bus ID: 0, IRQ: 15 destination APIC ID: 2, INT: 15 -- I/O INT INT type: 0, flags: 0x000f source bus ID: 1, IRQ: 32 destination APIC ID: 2, INT: 16 -- I/O INT INT type: 0, flags: 0x000f source bus ID: 1, IRQ: 36 destination APIC ID: 2, INT: 17 -- I/O INT INT type: 0, flags: 0x000f source bus ID: 1, IRQ: 40 destination APIC ID: 2, INT: 18 -- I/O INT INT type: 0, flags: 0x000f source bus ID: 1, IRQ: 48 destination APIC ID: 2, INT: 19 -- I/O INT INT type: 2, flags: 0x0000 source bus ID: 0, IRQ: 0 destination APIC ID: 2, INT: 23 -- Local INT INT type: 3, flags: 0x0005 source bus ID: 0, IRQ: 0 destination APIC ID: 255, INT: 0 -- Local INT INT type: 1, flags: 0x0005 source bus ID: 0, IRQ: 0 destination APIC ID: 255, INT: 1 -------------------------------------------------------------------------- Here's Russel's (Intel XXPRESS): -------------------------------------------------------------------------- MP Floating Pointer Structure: location: BIOS physical address: 0x000f7ba0 signature: '_MP_' length: 16 bytes version: 1.4 checksum: 0x66 mode: Virtual Wire -------------------------------------------------------------------------- MP Config Table Header: physical address: 0x000f7bb0 signature: 'PCMP' base table length: 268 version: 1.4 checksum: 0xdd OEM ID: 'INTEL ' Product ID: 'XXPRESS ' OEM table pointer: 0x00000000 OEM table size: 0 entry count: 25 local APIC address: 0xfee00000 extended table length: 220 extended table checksum: 190 -------------------------------------------------------------------------- MP Config Base Table Entries: -- Processor apic ID: 0, version: 16 CPU is usable, CPU is the bootstrap processor family: 5, model: 2, stepping: 11 feature flags: 0x000003bf -- Processor apic ID: 2, version: 16 CPU is usable, CPU is NOT the bootstrap processor family: 5, model: 2, stepping: 11 feature flags: 0x000003bf -- Bus bus ID: 0, bus type: PCI -- Bus bus ID: 1, bus type: PCI -- Bus bus ID: 18, bus type: XPRESS -- Bus bus ID: 19, bus type: EISA -- I/O APIC apic ID: 14, version: 17 APIC is usable apic address: 0xfec00000 -- I/O INT INT type: 3, flags: 0x0005 source bus ID: 19, IRQ: 0 destination APIC ID: 14, INT: 0 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 1 destination APIC ID: 14, INT: 1 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 0 destination APIC ID: 14, INT: 2 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 3 destination APIC ID: 14, INT: 3 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 4 destination APIC ID: 14, INT: 4 -- I/O INT INT type: 0, flags: 0x000c source bus ID: 19, IRQ: 5 destination APIC ID: 14, INT: 5 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 6 destination APIC ID: 14, INT: 6 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 7 destination APIC ID: 14, INT: 7 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 8 destination APIC ID: 14, INT: 8 -- I/O INT INT type: 0, flags: 0x000c source bus ID: 19, IRQ: 9 destination APIC ID: 14, INT: 9 -- I/O INT INT type: 0, flags: 0x000c source bus ID: 19, IRQ: 10 destination APIC ID: 14, INT: 10 -- I/O INT INT type: 0, flags: 0x000c source bus ID: 19, IRQ: 11 destination APIC ID: 14, INT: 11 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 12 destination APIC ID: 14, INT: 12 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 13 destination APIC ID: 14, INT: 13 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 14 destination APIC ID: 14, INT: 14 -- I/O INT INT type: 0, flags: 0x0005 source bus ID: 19, IRQ: 15 destination APIC ID: 14, INT: 15 -- Local INT INT type: 3, flags: 0x0005 source bus ID: 19, IRQ: 0 destination APIC ID: 255, INT: 0 -- Local INT INT type: 1, flags: 0x0005 source bus ID: 0, IRQ: 0 destination APIC ID: 255, INT: 1 -------------------------------------------------------------------------- MP Config Extended Table Entries: -- bus ID: 0 address type: memory address address base: 0xe8000 address range: 0x4000 -- bus ID: 0 address type: memory address address base: 0xa0000 address range: 0x20000 -- bus ID: 1 address type: prefetch address address base: 0xc0100000 address range: 0x100000 -- bus ID: 1 address type: memory address address base: 0xc0000000 address range: 0x100000 -- bus ID: 1 address type: I/O address address base: 0x7000 address range: 0x1000 -- bus ID: 0 address type: memory address address base: 0x6000000 address range: 0xba000000 -- bus ID: 0 address type: memory address address base: 0xc0200000 address range: 0x3fe00000 -- bus ID: 0 address type: I/O address address base: 0x0 address range: 0x7000 -- bus ID: 0 address type: I/O address address base: 0x8000 address range: 0x8000 -- bus ID: 19 bus info: 0x01 parent bus ID: 0-- bus ID: 0 address modifier: add predefined range: 0x00000000-- bus ID: 0 address modifier: add predefined range: 0x00000001-- bus ID: 1 address modifier: subtract predefined range: 0x00000000-- bus ID: 1 address modifier: subtract predefined range: 0x00000001 -------------------------------------------------------------------------- -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Thu Sep 12 11:21:45 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA20591 for smp-outgoing; Thu, 12 Sep 1996 11:21:45 -0700 (PDT) Received: from groa.uct.ac.za (groa.uct.ac.za [137.158.128.7]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA20584 for ; Thu, 12 Sep 1996 11:21:39 -0700 (PDT) Received: by groa.uct.ac.za via sendmail with stdio id for freebsd-smp@freebsd.org; Thu, 12 Sep 1996 20:17:36 +0200 (SAT) (Smail-3.2 1996-Jul-4 #1 built 1996-Jul-21) Message-Id: From: rv@groa.uct.ac.za (Russell Vincent) Subject: Re: Intel XXpress - some SMP benchmarks To: smp@csn.net (Steve Passe) Date: Thu, 12 Sep 1996 20:17:35 +0200 (SAT) Cc: peter@spinner.dialix.com, freebsd-smp@freebsd.org In-Reply-To: <199609120812.CAA15528@clem.systemsix.com> from "Steve Passe" at Sep 12, 96 02:12:56 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > Hint: ALL the APIC ID registers are read/write. > > > > Can you see it yet? :-) > > we tried that on the XXPRESS and (Russel, please confirm this) an instant reset > of the hardware. My mail notes show: Confirmed. Definately caused the machine to reboot. -Russell From owner-freebsd-smp Thu Sep 12 11:25:53 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA20943 for smp-outgoing; Thu, 12 Sep 1996 11:25:53 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA20931 for ; Thu, 12 Sep 1996 11:25:48 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id MAA18695; Thu, 12 Sep 1996 12:23:59 -0600 Message-Id: <199609121823.MAA18695@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Terry Lambert cc: peter@spinner.dialix.com (Peter Wemm), rv@groa.uct.ac.za, freebsd-smp@FreeBSD.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 11:07:22 PDT." <199609121807.LAA07176@phaeton.artisoft.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Sep 1996 12:23:59 -0600 Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk Hi, >> One thing I'm not clear about from the IO apic docs yet is whether there >> are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on >> the APIC bus. > > 1 BP, 31 (AP | IO APIC) (2^5 == 32) its a four bit register, where do you get 2^5, am I missing something? > You may want to disassemble your MP cold boot BIOS code to see about I've been wondering about the issue of the MP table being in BIOS. Traditionally this is ROM. Do you think they just hardcode all this stuff, or really arbitrate numbers during boot and can 'modify' them in the BIOS area because it is really shadowed into RAM? -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Thu Sep 12 11:41:01 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA22036 for smp-outgoing; Thu, 12 Sep 1996 11:41:01 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id LAA22028 for ; Thu, 12 Sep 1996 11:40:59 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id LAA07343; Thu, 12 Sep 1996 11:38:32 -0700 From: Terry Lambert Message-Id: <199609121838.LAA07343@phaeton.artisoft.com> Subject: Re: Intel XXpress - some SMP benchmarks To: smp@csn.net (Steve Passe) Date: Thu, 12 Sep 1996 11:38:32 -0700 (MST) Cc: terry@lambert.org, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@FreeBSD.org In-Reply-To: <199609121823.MAA18695@clem.systemsix.com> from "Steve Passe" at Sep 12, 96 12:23:59 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk > >> One thing I'm not clear about from the IO apic docs yet is whether there > >> are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on > >> the APIC bus. > > > > 1 BP, 31 (AP | IO APIC) (2^5 == 32) > > its a four bit register, where do you get 2^5, am I missing something? Oh, ugh. Sequent supports 32 processors. I wonder how? Corrected value: 1 BP, 15 (AP | IO APIC) (2^4 == 16) > > You may want to disassemble your MP cold boot BIOS code to see about > > I've been wondering about the issue of the MP table being in BIOS. > Traditionally this is ROM. Do you think they just hardcode all this > stuff, or really arbitrate numbers during boot and can 'modify' them in the > BIOS area because it is really shadowed into RAM? Some of the stuff has to be arbitrary... the cache writeback/writethrough is enough to make me believe that. It changes with CMOS settings. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Sep 12 13:05:01 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id NAA27405 for smp-outgoing; Thu, 12 Sep 1996 13:05:01 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id NAA27370 for ; Thu, 12 Sep 1996 13:04:53 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id OAA19225; Thu, 12 Sep 1996 14:04:27 -0600 Message-Id: <199609122004.OAA19225@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Peter Wemm cc: rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org, Terry Lambert Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 12:17:16 MDT." <199609121817.MAA18639@clem.systemsix.com> Mime-Version: 1.0 Content-Type: text/plain Date: Thu, 12 Sep 1996 14:04:27 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >Hmm. I'd rather avoid an extra indirection map if possible. Considering >that we are #define'ing things like curproc and some other heavily used >variables to lookup the cpuid for array indexes, we would better spend the >effort to just deal with a sparse ID set properly. further thoughts on dense vs sparse ID mapping: the simple indirection to get an int from a table of 16 ints is pretty cheap. I suggest cleanup of any areas of code that repeatedly call macros like curproc. I did a quick grep in /kern and found a few offenders like: curproc->p_flag |= P_PHYSIO; (I wonder if gcc will optimize this to only 1 APIC read?) Most of the code does a 1 time assignment to a struct ptr via curproc and uses that. if we have sparse tables, ie 16 entries (most of which will be empty), code like the following becomes very inefficient: #ifdef SMP for (j = i = 0; i < SPARSE_TABLE_SIZE; i++) { if (p == SMPcurproc[i]) j++; } ... --- of course all the above is assumming my current belief that we can't get away with reprogramming the APIC IDs to a dense sequence. I know that the APICs initially get their IDs from the state of 4 pins during hardware (motherboard) RESET. Whether there are problems when the ID you reprogam to are different I haven't a clue... --- 2nd STARTUP pentium proc. vol 1, 20.1.1.4, par 5: "It is the responsibility of the system software to resend the STARTUP IPI if there is an error... [enable APIC, use LVT3 ERROR vector to check it ] Otherwise the system software would have to poll the delivery status bit ... to determine if IPI is pending ... and resend STARTUP IPI if the IPI remains pending after an appropiate amount of time." since we poll this bit I think that (logically anyways) we should be safe only sending the one STARTUP IPI. From the XXPRESS port we know that if there isn't an APIC sitting there @ that ID the loop hangs forever. If we go to a broadcast scheme then polling may or may not work. I could see it working like a "wired-or", ie stays pending till all CPUs have received it, OR screwing up totally. Does anyone out there reading this list have a board with more than 2 CPUs? We might want to run some tests before we paint ourselves into either corner. --- # of IDs: pentium proc. vol 3, 19.3.1.5, 'Physical Dest Mode': "... A single destination (ID = 0 thru 14) or a broadcast to all (ID = 15) can be specified in the physical mode. ... Note that in this mode, the Pentium Processor ... APIC supports up to 15 agents." So this is probably why the XXPRESS IDs its IO APIC #14, ie at the top end. The board has a 2nd IO APIC for the 2nd PIC bus (we never looked at the MP table while it was enabled) which probably would get ID #13. Ie., I am guessing they start IDing IO APICs at the top and work down. Why they skip ID #1 for CPUs I have not a clue!!! Terry earlier wondered how the Sequent supported 32 CPUs. There is a more complex 'heirarxchial model' that supports up to 60 APICs. This model requires additional custom hardware support. I don't think I wanna go there. --- I'm going to cleanup mptable.c next, and will post it in an hour or 2. It is the user level program that we used to dump the MP tables I published earlier today. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Thu Sep 12 13:38:58 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id NAA00740 for smp-outgoing; Thu, 12 Sep 1996 13:38:58 -0700 (PDT) Received: from critter.tfs.com ([140.145.230.177]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id NAA00731; Thu, 12 Sep 1996 13:38:49 -0700 (PDT) Received: from critter.tfs.com (localhost.tfs.com [127.0.0.1]) by critter.tfs.com (8.7.5/8.7.3) with ESMTP id WAA01692; Thu, 12 Sep 1996 22:38:04 +0200 (MET DST) To: Steve Passe cc: Peter Wemm , rv@groa.uct.ac.za (Russell Vincent), freebsd-smp@freebsd.org, Terry Lambert Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 14:04:27 MDT." <199609122004.OAA19225@clem.systemsix.com> Date: Thu, 12 Sep 1996 22:38:03 +0200 Message-ID: <1690.842560683@critter.tfs.com> From: Poul-Henning Kamp Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk In message <199609122004.OAA19225@clem.systemsix.com>, Steve Passe writes: > >the simple indirection to get an int from a table of 16 ints is pretty cheap. >I suggest cleanup of any areas of code that repeatedly call macros like >curproc. I did a quick grep in /kern and found a few offenders like: > > curproc->p_flag |= P_PHYSIO; > >(I wonder if gcc will optimize this to only 1 APIC read?) Most of the >code does a 1 time assignment to a struct ptr via curproc and uses that. I actually thought of "stealing" one or two vectors in the LAPIC and cache curproc there... -- Poul-Henning Kamp | phk@FreeBSD.ORG FreeBSD Core-team. http://www.freebsd.org/~phk | phk@login.dknet.dk Private mailbox. whois: [PHK] | phk@ref.tfs.com TRW Financial Systems, Inc. Future will arrive by its own means, progress not so. From owner-freebsd-smp Thu Sep 12 14:21:22 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA04424 for smp-outgoing; Thu, 12 Sep 1996 14:21:22 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id OAA04413 for ; Thu, 12 Sep 1996 14:21:14 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id OAA13651; Thu, 12 Sep 1996 14:13:49 -0700 (PDT) Message-Id: <199609122113.OAA13651@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: Terry Lambert , smp@csn.net (Steve Passe) cc: peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) In-reply-to: Your message of "Thu, 12 Sep 1996 11:12:53 PDT." <199609121812.LAA07189@phaeton.artisoft.com> Date: Thu, 12 Sep 1996 14:13:44 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Terry Lambert writes: > > > Hint: ALL the APIC ID registers are read/write. > > > > > > Can you see it yet? :-) > > > > we tried that on the XXPRESS and (Russel, please confirm this) an instant > > reset of the hardware. > > Any chance that a write of the ID register acts as an INIT IPI? That's > what seems to be implied. This is not the case. In particular a RESET and INIT are NOT the same thing. I'm not sure what the problem you're having is (I'll try it tonight), but a known bug with the APIC is that it requires all reads and writes to be 32-bit aligned accesses (or weird things can happen). > I suspect that you will need to inventory the processors, then back-fill > the holes for the case where you would get an ID collision during the > shuffling -- ie: if I have n processors, all APIC ID's < (n-1) are left > alone, and only the remainder are rewritten. > > I *believe* that the BP is guranteed an APIC ID of 0. No. The rules are: "default configuration": CPUs must be numbered consecutively starting with 0, in any order. configurations defined in the MP tables: at least one CPU must have id #0, and a vague warning that if you use id numbers too large for other CPUs then the addressing mechanism won't be compatible with all versions of the APIC (i.e. dont use anything more than an 8-bit address can find). The above rules (numbering is arbitrary) have to be following by the OS, or many machines simply aren't going to work. Many Pentium Pro machines using 2 CPUs have the boot CPU as APIC id #1. Also realize that when a processor goes through INIT, the APIC id will be reset to the hardware id. This means if your OS is in the middle of doing a bunch of things and then sends an INIT to one of the CPUs, you have a big potential problem there. To solve this, you either don't reassign APIC ids, or never send an INIT when the system is up and running. All this stuff is in the MPS spec 1.4 and Pentium errata (the 32-bit access thing might not be, I forget if it is a Pentium Pro issue). > You may want to disassemble your MP cold boot BIOS code to see about > the ID assignment; clearly it must be happening in BIOS in any case, > since the PPRO's are "glueless" and would care about which slots they > are put in, otherwise. Quite a bit is determined by the hardware slot it is in. I'm pretty sure that the Pentium and discrete APICs were hard-wired based on wires on the board. I think even the Pentium Pros are, but this is based on a set of wires in the bus protocol in their case, that's why it's "glueless". In a later message, Terry Lambert wrote: > Oh, ugh. Sequent supports 32 processors. I wonder how? I'm pretty sure they don't use anything like the Intel SMP architecture. Steve Passe writes: > >BTW, Some other things we do not do.. We don't set the ERROR LVT to handle > >a non-delivered or failed message. > no, but my original code for the apic_startup() had checks on the APIC_ESR > register, never saw errors. I will put that back later today. Before your operation, write a zero to the APIC_ESR register (I think a read before the write is even necessary... I'll dredge up my notes which tried to make sense of all the eratta sometime tonight or tomorrow). > >I noticed you took out the second STARTUP IPI.. The docs I've been > >reading say "the startup IPI can only be used once after a reset or INIT", > > ... > >the second one is for insurance in case the first one was missed, and that > >the second will normally be ignored. I notice that there is no way in the > > I guess the second one couldn't hurt, but I would rather use a better > means than "if I do it often enough its gotta work". A greater concern > is the INIT/RESET of the 'run bootMP' flavor that the XXPRESS demanded. > If we add the correct timings whats to prevent the STARTUP IPI from > re-running a CPU once it has already started (via RESET), perhaps > double incrementing mp_ncpus? It's not a problem of the first STARTUP being "missed", the problem (which is only present on the Pentium integrated APIC) is as follows: When the Pentium is in the state waiting for a STARTUP, an INIT doesn't get processed, but it is latched. The STARTUP then allows the INIT to be executed, which effectively kills the STARTUP message (this is where the time-delays in the recommended INIT/STARTUP/STARTUP process come from, to allow the CPU to finish the INIT process), so you need another STARTUP message. The problem with trying to just do an INIT, then if that doesn't work, a STARTUP and INIT, is that you're never sure what the state the BIOS puts CPUs in on a lot of the Pentium boxes (you'd also have to wait to see if it responded to each part of the sequence). I spent a lot of time finding out that the ENTIRE sequence was required on even machines that I thought didn't need it (the XXPress box). The Intel folks spent a lot of time generating a single guaranteed sequence that covered all the bases. It's best to simply use the recommended startup sequence and leave it at that, if you wish to support Pentium SMP boxes reliably (again, Pentium Pros dont need this). > >The pppbios.pdf specifically says: "The BSP sends a StartUp APIC message > >broadcast......" One of the various other tables in the P5 docs say that > >startup IPI broadcasts are always edge triggered when used in "all but > >self" mode, so who knows.. :-) > > I see that now, I'm willing to believe it might be doable. I'm not quite sure what you're saying here. If this is from the SMP bootup sections of the various manuals, I remember it talking a lot about such things that were entirely referring to inter-APIC messages done as part of the APIC bus protocol which had no bearing on how the OS was to do CPU startup/etc. > >Another thought.. We do not use the timer on the apic. It has a 32 bit > >read/write register for the "initial count". We could cheat and use that > >as a 32 bit pointer to a cpu-specific data page with each cpu's scratch > >area etc, > > are you certain that we won't want to use it in the future? > another issue is that erich claims accessing the APIC registers is relatively > expensive time-wise. Yes. An APIC read/write is an *uncached* access, which is prety slow. The reason for this is that the APIC is effectively a hardware device, and you're working with it's control ports. I suppose it could have been optimized further, but other things accessed much more often took precidence. On the Pentium Pro, an uncached access has to drain many of the bus pipelines, and is not terribly fast. An L2 hit will certainly be a heck of a lot faster (an order of magnitude in some cases), no matter what CPU you use. Basically, get the APIC id, then calculate everything from there, even if you need to do a small table lookup, it's still going to be faster. > > better off putting in a check for an 82489 apic in the boot code and > > both mine and Russel's boards show an IO APIC version of 17. His CPUs > show APIC vewrsion 16, while mine shows 17 (all numbers taken from the > MP table, NOT the APIC version registers). I can't find a # for the > 82379AB anywhere in its manual. An integrated APIC (or more to the point, any APIC with support for the STARTUP IPI) will have a version of 16 or greater. > >From the manuals: > > 82093AA: 11 > 82379AB: ?? > 82489DX: 01 > > P5: 1x > P6: ?? >From the manual I have, the P6 is hex 11. .... > fixes that. still have a problem in that the makefiles for biosboot etc. > don't search the right path, ie opt_smp is only going to be in the kernel > compile specific directory, is it not? THis only worked because I copied > opt_smp.h to i386/include. Interesting, is it coincidence, or does the biosboot actually need to know about SMP information? -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" From owner-freebsd-smp Thu Sep 12 14:45:09 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA05459 for smp-outgoing; Thu, 12 Sep 1996 14:45:09 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id OAA05450 for ; Thu, 12 Sep 1996 14:45:02 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id OAA07599; Thu, 12 Sep 1996 14:42:51 -0700 From: Terry Lambert Message-Id: <199609122142.OAA07599@phaeton.artisoft.com> Subject: Re: Intel XXpress - some SMP benchmarks To: smp@csn.net (Steve Passe) Date: Thu, 12 Sep 1996 14:42:50 -0700 (MST) Cc: peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org, terry@lambert.org In-Reply-To: <199609122004.OAA19225@clem.systemsix.com> from "Steve Passe" at Sep 12, 96 02:04:27 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > if we have sparse tables, ie 16 entries (most of which will be empty), code > like the following becomes very inefficient: > > #ifdef SMP > for (j = i = 0; i < SPARSE_TABLE_SIZE; i++) { > if (p == SMPcurproc[i]) > j++; > } > ... Use it like a hash: #ifdef SPARSE int cpuid[ MAX_CPUID]; /* filled out by probe sequence*/ #define CPUID(x) (cpuid[ x]) #else /* !SPARSE*/ #define CPUID(x) (x) #endif /* !SPARSE*/ Obviously, this would save one indirection per reference, which shouldn't be too frequent anyway. > Does anyone out there reading this list have a board with more than 2 CPUs? > We might want to run some tests before we paint ourselves into either corner. Erich has more than 2 CPU's; he also wrote the Linux SMP startup code. > Terry earlier wondered how the Sequent supported 32 CPUs. There is a more > complex 'heirarxchial model' that supports up to 60 APICs. This model > requires additional custom hardware support. I don't think I wanna go there. Too bad; my Alma Mater has two sequent boxes that were just donated to them. 8-). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Sep 12 14:55:18 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA05973 for smp-outgoing; Thu, 12 Sep 1996 14:55:18 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id OAA05967 for ; Thu, 12 Sep 1996 14:55:14 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id OAA07647; Thu, 12 Sep 1996 14:51:42 -0700 From: Terry Lambert Message-Id: <199609122151.OAA07647@phaeton.artisoft.com> Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) To: erich@uruk.org Date: Thu, 12 Sep 1996 14:51:41 -0700 (MST) Cc: terry@lambert.org, smp@csn.net, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org In-Reply-To: <199609122113.OAA13651@uruk.org> from "erich@uruk.org" at Sep 12, 96 02:13:44 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > I suspect that you will need to inventory the processors, then back-fill > > the holes for the case where you would get an ID collision during the > > shuffling -- ie: if I have n processors, all APIC ID's < (n-1) are left > > alone, and only the remainder are rewritten. > > > > I *believe* that the BP is guranteed an APIC ID of 0. > > No. The rules are: > > "default configuration": > CPUs must be numbered consecutively starting with 0, in any order. The XXPRESS fails this, then, since it goes 0, 2 (not 0, 1). Does consecutively mean "insreasing" as opposed to "increasing monotonically"? > Also realize that when a processor goes through INIT, the APIC id will be > reset to the hardware id. This means if your OS is in the middle of doing > a bunch of things and then sends an INIT to one of the CPUs, you have a big > potential problem there. To solve this, you either don't reassign APIC ids, > or never send an INIT when the system is up and running. Or you can reassign ID's, as long as you back fill holes instead of shifting them down. That way an ID can never collide with a reassigned ID following an INIT. You just have to be prepared to deal with INIT by taking the ID and remapping it back into the backfill location. The reason we are talking about backfilling or renumbering at all is because the XXPRESS is known to fail the "consecutiveness" test already... Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Sep 12 16:09:45 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id QAA10641 for smp-outgoing; Thu, 12 Sep 1996 16:09:45 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id QAA10631 for ; Thu, 12 Sep 1996 16:09:41 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id RAA20132; Thu, 12 Sep 1996 17:07:15 -0600 Message-Id: <199609122307.RAA20132@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: erich@uruk.org cc: terry@lambert.org, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) In-reply-to: Your message of "Thu, 12 Sep 1996 14:13:44 PDT." <199609122113.OAA13651@uruk.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Sep 1996 17:07:15 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, erich, thanx for helping! --- > Also realize that when a processor goes through INIT, the APIC id will be > reset to the hardware id. This means if your OS is in the middle of doing pentium processor manual vol3, 19.3.1.11.3 'Initialization Reset (INIT)': "INIT is a software reset, and is delivered as a bus message. INIT has the same effect on the Local APIC as the power-up Reset, except that the APIC ID and the Arb ID registers are not affected." --- > Before your operation, write a zero to the APIC_ESR register (I think a > read before the write is even necessary... I'll dredge up my notes > which tried to make sense of all the eratta sometime tonight or tomorrow). I followed your code from the linux project for accessing the ESR register, including the clear of the register to start. --------------------------------------------- Everyone please bear with me here, this gets ugly, but I think we are getting close to resolution. --- > It's not a problem of the first STARTUP being "missed", the problem > (which is only present on the Pentium integrated APIC) is as follows: > When the Pentium is in the state waiting for a STARTUP, an INIT "in the state waiting for a STARTUP", is this a state reached by hardware RESET? or perhaps I'm asking what this state is. Is it as if a 'hlt' instruction had been executed? > doesn't get processed, but it is latched. The STARTUP then allows the > INIT to be executed, which effectively kills the STARTUP message (this is > where the time-delays in the recommended INIT/STARTUP/STARTUP process > come from, to allow the CPU to finish the INIT process), so you need > another STARTUP message. so if I understand this we have: 2nd CPU is waiting for a STARTUP IPI 1st CPU does INIT, 1st CPU waits for INIT to run 2nd CPU latches INIT IPI, DOESN'T process it yet. 1st CPU finishes wait 1st CPU does STARTUP IPI 2nd CPU catches STARTUP IPI, starts to vector thru supplied STARTUP vector 2nd CPU immediately catches INIT IPI, aborting STARTUP IPI INIT causes vector thru warm-boot to bootMP() bootMP() gets 2nd CPU initialized and running 2nd CPU ignores second STARTUP IPI as it only accepts ONE STARTUP IPI after RESET/INIT to get the XXPRESS box working I HAD to preceed the INIT IPI with a setup of the BIOS warm-start vector to the bootMP() code. pointing it at a 'hlt' instruction, then allowing the STARTUP IPI to provide the vector to bootMP() fails. so it would appear that BOTH of the STARTUP IPIs (both vectoring to bootMP()) fail and/or are ignored by the XXPRESS. I'm sure (he says with confidence) that the INIT IPI actually travels thru the vector as we replaced the 'hlt' instruction with an NMI and we got spontanious reboot. > The problem with trying to just do an INIT, then if that doesn't work, a > STARTUP and INIT, is that you're never sure what the state the BIOS > puts CPUs in on a lot of the Pentium boxes (you'd also have to wait > to see if it responded to each part of the sequence). I spent a lot of > time finding out that the ENTIRE sequence was required on even machines > that I thought didn't need it (the XXPress box). The Intel folks > spent a lot of time generating a single guaranteed sequence that > covered all the bases. I'm getting a headache... SO at this point I think I buy the double INIT IPI followed by 2 STARTUP IPS, with appropriate timings in-between as the cure for all hardware, in all the "brokenness" we are likely to find. I'm just not clear on the appropriate thing to do with the warm-start vector. I believe its needed, and that the XXPRESS for one needs it to point to the actual 2nd CPU boot code. Can we expect to be able to vector to the boot code on all hardware, or will we encounter boxes that run the boot code via the INIT IPI warm-start vector, then re-run it via one of the following STARTUP IPIs??????? is there a definitive code sample that encompasses all this. in the MP spec they have the pseudo code with 2 STARTUP IPIs, but no actual code. the pentium manual shows some real code with timings, but NO double STARTUP sequence.... Peter, if this is whats happening to you with the new apic_startup() I submitted, cloning the STARTUP+delay+read pending section to a 2nd one should fix it (I think). -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Thu Sep 12 16:09:56 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id QAA10663 for smp-outgoing; Thu, 12 Sep 1996 16:09:56 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id QAA10655 for ; Thu, 12 Sep 1996 16:09:53 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id RAA20140; Thu, 12 Sep 1996 17:07:31 -0600 Message-Id: <199609122307.RAA20140@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Terry Lambert cc: erich@uruk.org, smp@csn.net, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) In-reply-to: Your message of "Thu, 12 Sep 1996 14:51:41 PDT." <199609122151.OAA07647@phaeton.artisoft.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Sep 1996 17:07:30 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >> No. The rules are: >> >> "default configuration": >> CPUs must be numbered consecutively starting with 0, in any order. > >The XXPRESS fails this, then, since it goes 0, 2 (not 0, 1). it isn't a 'default' configuration as it contains a complete MP table describing its configuration. (not to say that the XXPRESS doesn't "fail", the jury is still out on that issue!) -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Thu Sep 12 17:05:35 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA14202 for smp-outgoing; Thu, 12 Sep 1996 17:05:35 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id RAA14197 for ; Thu, 12 Sep 1996 17:05:32 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id SAA20483; Thu, 12 Sep 1996 18:05:21 -0600 Message-Id: <199609130005.SAA20483@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Chuck Robey cc: freebsd-smp@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Thu, 12 Sep 1996 19:51:16 EDT." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Sep 1996 18:05:21 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >> Pentium® Pro Processor BIOS Writer's Guide V2.0 >> http://www.intel.com/IAL/processr/p6/pppbios.pdf > >Hi Steve. That above wasn't there anymore when I went looking for it. >Any chance I could download it from you? Opps, sorry, my fault: ftp://ftp.intel.com/pub/IAL/p6/pppbios.pdf -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Thu Sep 12 17:14:01 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA14691 for smp-outgoing; Thu, 12 Sep 1996 17:14:01 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id RAA14675 for ; Thu, 12 Sep 1996 17:13:52 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id IAA05873; Fri, 13 Sep 1996 08:13:10 +0800 (WST) Message-Id: <199609130013.IAA05873@spinner.DIALix.COM> X-Mailer: exmh version 1.6.7 5/3/96 To: Steve Passe cc: Chuck Robey , freebsd-smp@freebsd.org Subject: Re: 82489 data books In-reply-to: Your message of "Thu, 12 Sep 1996 18:05:21 CST." <199609130005.SAA20483@clem.systemsix.com> Date: Fri, 13 Sep 1996 08:13:09 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Steve Passe wrote: > Hi, > > >> Pentium® Pro Processor BIOS Writer's Guide V2.0 > >> http://www.intel.com/IAL/processr/p6/pppbios.pdf > > > >Hi Steve. That above wasn't there anymore when I went looking for it. > >Any chance I could download it from you? > > Opps, sorry, my fault: > > ftp://ftp.intel.com/pub/IAL/p6/pppbios.pdf Also, beware that this is a P6 doc. It has comments in there about early boot that suggest that the P6 has special MPSPEC support built in that probably isn't in the P5. For example, the negotiation about which cpu is the BSP, ID numbers etc. Really, that's none of our concern though, but there may be other minor differences we need to be aware of. Cheers, -Peter From owner-freebsd-smp Thu Sep 12 17:26:55 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA15262 for smp-outgoing; Thu, 12 Sep 1996 17:26:55 -0700 (PDT) Received: from po1.glue.umd.edu (po1.glue.umd.edu [129.2.128.44]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id RAA15257 for ; Thu, 12 Sep 1996 17:26:53 -0700 (PDT) Received: from uplink.eng.umd.edu (uplink.eng.umd.edu [129.2.98.181]) by po1.glue.umd.edu (8.7.5/8.7.3) with ESMTP id UAA22709; Thu, 12 Sep 1996 20:26:50 -0400 (EDT) Received: from localhost (chuckr@localhost) by uplink.eng.umd.edu (8.7.5/8.7.3) with SMTP id UAA03673; Thu, 12 Sep 1996 20:26:49 -0400 (EDT) X-Authentication-Warning: uplink.eng.umd.edu: chuckr owned process doing -bs Date: Thu, 12 Sep 1996 20:26:49 -0400 (EDT) From: Chuck Robey X-Sender: chuckr@uplink.eng.umd.edu To: Steve Passe cc: freebsd-smp@freebsd.org Subject: Re: 82489 data books In-Reply-To: <199609130005.SAA20483@clem.systemsix.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Thu, 12 Sep 1996, Steve Passe wrote: > Hi, > > >> Pentium. Pro Processor BIOS Writer's Guide V2.0 > >> http://www.intel.com/IAL/processr/p6/pppbios.pdf > > > >Hi Steve. That above wasn't there anymore when I went looking for it. > >Any chance I could download it from you? > > Opps, sorry, my fault: > > ftp://ftp.intel.com/pub/IAL/p6/pppbios.pdf That one works. Thanks to everyone on this list, providing a wonderful tutorial on smp! I'm not commenting, but I sure am enjoying this! ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@eng.umd.edu | communications topic, C programming, and Unix. 9120 Edmonston Ct #302 | Greenbelt, MD 20770 | I run Journey2 and n3lxx, both FreeBSD (301) 220-2114 | version 2.2 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Thu Sep 12 17:45:22 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA16373 for smp-outgoing; Thu, 12 Sep 1996 17:45:22 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id RAA16366 for ; Thu, 12 Sep 1996 17:45:15 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id SAA20723; Thu, 12 Sep 1996 18:45:11 -0600 Message-Id: <199609130045.SAA20723@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: freebsd-smp@freebsd.org cc: peter@spinner.dialix.com Subject: mptable.c Mime-Version: 1.0 Content-Type: text/plain Date: Thu, 12 Sep 1996 18:45:11 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, here's the tool to parse out your MP table from your motherboard. It uses /dev/kmem so it must be built and run as root. Keywords: MP Spec, Configuration Table ----------------------------------- cut --------------------------------------- /* * mptable.c */ #define MP_SIG 0x5f504d5f /* _MP_ */ #define EXTENDED_PROCESSING_READY #define OEM_PROCESSING_READY_NOT /** cheat for now, hardcode address */ #define CHEATING_NOT #if 0 /* XXPRESS */ #define CHEAT_ADDRESS 0x000f7ba0 #else /* GA586DX */ #define CHEAT_ADDRESS 0x000f0c80 #endif #include #include #include #include #include #if 0 #include #include #include what else? and is it worth it? #else #define KERNBASE ((vm_offset_t)0xf0000000) #endif #define BIOS_BASE 0xf0000 #define BIOS_SIZE 0x10000 #define ONE_KBYTE 1024 #define PROCENTRY_FLAG_EN 0x01 #define PROCENTRY_FLAG_BP 0x02 #define IOAPICENTRY_FLAG_EN 0x01 #define MAXPNSTR 132 char* whereStrings[] = { "Extended BIOS Data Area", "@ top of memory", "BIOS" }; typedef struct TABLE_ENTRY { u_char type; u_char length; char name[ 32 ]; } tableEntry; tableEntry basetableEntryTypes[] = { { 0, 20, "Processor" }, { 1, 8, "Bus" }, { 2, 8, "I/O APIC" }, { 3, 8, "I/O INT" }, { 4, 8, "Local INT" } }; tableEntry extendedtableEntryTypes[] = { { 128, 20, "System Address Space" }, { 129, 8, "Bus Heirarchy" }, { 130, 8, "Compatibility Bus Address" } }; /* MP Floating Pointer Structure */ typedef struct MPFPS { char signature[ 4 ]; void* pap; u_char length; u_char spec_rev; u_char checksum; u_char mpfb1; u_char mpfb2; u_char mpfb3; u_char mpfb4; u_char mpfb5; } mpfps_t; /* MP Configuration Table Header */ typedef struct MPCTH { char signature[ 4 ]; u_short base_table_length; u_char spec_rev; u_char checksum; u_char oem_id[ 8 ]; u_char product_id[ 12 ]; void* oem_table_pointer; u_short oem_table_size; u_short entry_count; void* apic_address; u_short extended_table_length; u_char extended_table_checksum; u_char reserved; } mpcth_t; typedef struct PROCENTRY { u_char type; u_char apicID; u_char apicVersion; u_char cpuFlags; u_long cpuSignature; u_long featureFlags; u_long reserved1; u_long reserved2; } ProcEntry; typedef struct BUSENTRY { u_char type; u_char busID; char busType[ 6 ]; } BusEntry; typedef struct IOAPICENTRY { u_char type; u_char apicID; u_char apicVersion; u_char apicFlags; void* apicAddress; } IOApicEntry; typedef struct INTENTRY { u_char type; u_char intType; u_short intFlags; u_char srcBusID; u_char srcBusIRQ; u_char dstApicID; u_char dstApicINT; } IntEntry; /* * extended entry type structures */ typedef struct SASENTRY { u_char type; u_char length; u_char busID; u_char addressType; u_int64_t addressBase; u_int64_t addressLength; } SasEntry; typedef struct BHDENTRY { u_char type; u_char length; u_char busID; u_char busInfo; u_char busParent; u_char reserved[ 3 ]; } BhdEntry; typedef struct CBASMENTRY { u_char type; u_char length; u_char busID; u_char addressMod; u_int predefinedRange; } CbasmEntry; static void apic_probe( vm_offset_t* paddr, int* where ); static void MPConfigDefault( int featureByte ); static int MPFloatingPointer( vm_offset_t paddr, int where, mpfps_t* mpfps ); static void MPConfigTableHeader( void* pap ); static int readType( void ); static void seekEntry( vm_offset_t addr ); static void readEntry( void* entry, int size ); static void processorEntry( void ); static void busEntry( void ); static void ioApicEntry( void ); static void intEntry( void ); static void sasEntry( void ); static void bhdEntry( void ); static void cbasmEntry( void ); static void pnstr( char* s, int c ); /* global data */ int kfd; /* * */ int main( int argc, char *argv[] ) { vm_offset_t paddr; int where; mpfps_t mpfps; int defaultConfig; /* open kernel memory for access to MP structures */ if ( (kfd = open( "/dev/kmem", O_RDONLY )) < 0 ) { perror( "kmem open" ); exit( 1 ); } /* probe for MP structures */ apic_probe( &paddr, &where ); if ( where <= 0 ) { fprintf( stderr, "\n MP Table NOT found!!!\n\n" ); return 1; } printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); printf( "\nFound MP Table in %s, physical addr: 0x%08x\n", whereStrings[ where - 1 ], paddr ); printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); /* analyze the MP Floating Pointer Structure */ MPFloatingPointer( paddr, where, &mpfps ); printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); /* check whether an MP config table exists */ if ( defaultConfig = mpfps.mpfb1 ) { MPConfigDefault( defaultConfig ); } else { MPConfigTableHeader( mpfps.pap ); } printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); return 0; } /* * set PHYSICAL address of MP floating pointer structure */ static void apic_probe( vm_offset_t* paddr, int* where ) { #if defined( CHEATING ) /** cheat for now, hardcode address */ *paddr = (vm_offset_t)CHEAT_ADDRESS; /** cheat again, where we found it */ *where = 3; #else /** CHEATING */ /* * c rewrite of apic_probe() by Jack F. Vogel */ int x; unsigned short segment; vm_offset_t target; unsigned int buffer[ 16384 ]; if ( 1 ) { /* why can't I access kmem below 0xf0010000? */ fprintf( stderr, "\nWarning: EBDA support is BROKEN!!!\n" ); } else { /* EBDA is @ 40:0e in real-mode terms */ seekEntry( (vm_offset_t)0x040e + KERNBASE ); readEntry( &segment, 2 ); if ( segment ) /* search EBDA */ { target = (vm_offset_t)segment << 4; seekEntry( target + KERNBASE ); readEntry( buffer, ONE_KBYTE ); for ( x = 0; x < ONE_KBYTE / sizeof ( unsigned int ); ++x ) { if ( buffer[ x ] == MP_SIG ) { *where = 1; *paddr = (x * sizeof( unsigned int )) + target; return; } } } } # if 0 /** we should read CMOS for real top of mem, for now: */ # else /* base of the last 1K of 640K */ target = 0x9fc00; # endif seekEntry( target + KERNBASE); readEntry( buffer, ONE_KBYTE ); for ( x = 0; x < ONE_KBYTE / sizeof ( unsigned int ); ++x ) { if ( buffer[ x ] == MP_SIG ) { *where = 2; *paddr = (x * sizeof( unsigned int )) + target; return; } } /* search the BIOS */ seekEntry( BIOS_BASE + KERNBASE ); readEntry( buffer, BIOS_SIZE ); for ( x = 0; x < BIOS_SIZE / sizeof( unsigned int ); ++x ) { if ( buffer[ x ] == MP_SIG ) { *where = 3; *paddr = (x * sizeof( unsigned int )) + BIOS_BASE; return; } } *where = 0; *paddr = (vm_offset_t)0; #endif /** CHEATING */ } /* * */ static int MPFloatingPointer( vm_offset_t paddr, int where, mpfps_t* mpfps ) { vm_offset_t vaddr; /* convert physical address to a virtual address */ vaddr = paddr + (vm_offset_t)KERNBASE; /* read in mpfps structure*/ seekEntry( vaddr ); readEntry( mpfps, sizeof( mpfps_t ) ); /* show its contents */ printf( "MP Floating Pointer Structure:\n\n" ); printf( " location:\t\t\t", where ); switch ( where ) { case 0: printf( "NOT found!\n" ); exit( 1 ); case 1: printf( "EBDA\n" ); break; case 2: printf( "base memory\n" ); break; case 3: printf( "BIOS\n" ); break; default: printf( "BOGUS!\n" ); exit( 1 ); } printf( " physical address:\t\t0x%08x\n", paddr ); printf( " signature:\t\t\t'" ); pnstr( mpfps->signature, 4 ); printf( "'\n" ); printf( " length:\t\t\t%d bytes\n", mpfps->length * 16 ); printf( " version:\t\t\t1.%1d\n", mpfps->spec_rev ); printf( " checksum:\t\t\t0x%02x\n", mpfps->checksum ); /* bits 0:6 are RESERVED */ if ( mpfps->mpfb2 & 0x7f ) { printf( " warning, MP feature byte 2: 0x%02x\n" ); } /* bit 7 is IMCRP */ printf( " mode:\t\t\t\t%s\n", (mpfps->mpfb2 & 0x80) ? "PIC" : "Virtual Wire" ); /* MP feature bytes 3-5 are expected to be ZERO */ if ( mpfps->mpfb3 ) printf( " warning, MP feature byte 3 NONZERO!\n" ); if ( mpfps->mpfb4 ) printf( " warning, MP feature byte 4 NONZERO!\n" ); if ( mpfps->mpfb5 ) printf( " warning, MP feature byte 5 NONZERO!\n" ); } /* * */ static void MPConfigDefault( int featureByte ) { printf( " MP default config type: %d\n\n", featureByte ); switch ( featureByte ) { case 1: printf( " bus: ISA, APIC: 82489DX\n" ); break; case 2: printf( " bus: EISA, APIC: 82489DX\n" ); break; case 3: printf( " bus: EISA, APIC: 82489DX\n" ); break; case 4: printf( " bus: MCA, APIC: 82489DX\n" ); break; case 5: printf( " bus: ISA+PCI, APIC: Integrated\n" ); break; case 6: printf( " bus: EISA+PCI, APIC: Integrated\n" ); break; case 7: printf( " bus: MCA+PCI, APIC: Integrated\n" ); break; default: printf( " future type\n" ); break; } } /* * */ static void MPConfigTableHeader( void* pap ) { vm_offset_t vaddr; mpcth_t cth; int totalSize; int count; int type; vm_offset_t voemtp; void* oemdata; if ( pap == 0 ) { printf( "MP Configuration Table Header MISSING!\n" ); exit( 1 ); } /* convert physical address to virtual address */ vaddr = (vm_offset_t)pap + (vm_offset_t)KERNBASE; /* read in cth structure */ seekEntry( vaddr ); readEntry( &cth, sizeof( cth ) ); printf( "MP Config Table Header:\n\n" ); printf( " physical address:\t\t0x%08x\n", pap ); printf( " signature:\t\t\t'" ); pnstr( cth.signature, 4 ); printf( "'\n" ); printf( " base table length:\t\t%d\n", cth.base_table_length ); printf( " version:\t\t\t1.%1d\n", cth.spec_rev ); printf( " checksum:\t\t\t0x%02x\n", cth.checksum ); printf( " OEM ID:\t\t\t'" ); pnstr( cth.oem_id, 8 ); printf( "'\n" ); printf( " Product ID:\t\t\t'" ); pnstr( cth.product_id, 12 ); printf( "'\n" ); printf( " OEM table pointer:\t\t0x%08x\n", cth.oem_table_pointer ); printf( " OEM table size:\t\t%d\n", cth.oem_table_size ); printf( " entry count:\t\t\t%d\n", cth.entry_count ); printf( " local APIC address:\t\t0x%08x\n", cth.apic_address ); printf( " extended table length:\t%d\n", cth.extended_table_length ); printf( " extended table checksum:\t%d\n", cth.extended_table_checksum ); totalSize = cth.base_table_length - sizeof( struct MPCTH ); count = cth.entry_count; printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); printf( "MP Config Base Table Entries:\n\n" ); while ( count-- ) { switch ( type = readType() ) { case 0: processorEntry(); break; case 1: busEntry(); break; case 2: ioApicEntry(); break; case 3: intEntry(); break; case 4: intEntry(); break; default: printf( "Base Table HOSED!\n" ); exit( 1 ); } totalSize -= basetableEntryTypes[ type ].length; } #if defined( EXTENDED_PROCESSING_READY ) /* process any extended data */ if ( totalSize = cth.extended_table_length ) { printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); printf( "MP Config Extended Table Entries:\n\n" ); while ( totalSize > 0 ) { switch ( type = readType() ) { case 128: sasEntry(); break; case 129: bhdEntry(); break; case 130: cbasmEntry(); break; default: printf( "Extended Table HOSED!\n" ); exit( 1 ); } totalSize -= extendedtableEntryTypes[ type-128 ].length; } } #endif /* EXTENDED_PROCESSING_READY */ /* process any OEM data */ if ( cth.oem_table_pointer && (cth.oem_table_size > 0) ) { #if defined( OEM_PROCESSING_READY ) # error your on your own here! /* convert OEM table pointer to virtual address */ voemtp = (vm_offset_t)cth.oem_table_pointer + (vm_offset_t)KERNBASE; /* read in oem table structure */ if ( (oemdata = (void*)malloc( cth.oem_table_size )) == NULL ) { perror( "oem malloc" ); exit( 1 ); } seekEntry( voemtp ); readEntry( oemdata, cth.oem_table_size ); /** process it */ free( oemdata ); #else printf( "\nyou need to modify the source to handle OEM data!\n\n" ); #endif /* OEM_PROCESSING_READY */ } } /* * */ static int readType( void ) { u_char type; if ( read( kfd, &type, sizeof( u_char ) ) != sizeof( u_char ) ) { perror( "type read" ); exit( 1 ); } if ( lseek( kfd, -1, SEEK_CUR ) < 0 ) { perror( "type seek" ); exit( 1 ); } return (int)type; } /* * */ static void seekEntry( vm_offset_t addr ) { if ( lseek( kfd, (off_t)addr, SEEK_SET ) < 0 ) { fprintf( stderr, "\nvaddr: 0x%08x\n", addr ); perror( "kmem seek" ); exit( 1 ); } } /* * */ static void readEntry( void* entry, int size ) { if ( read( kfd, entry, size ) != size ) { perror( "readEntry" ); exit( 1 ); } } static void processorEntry( void ) { ProcEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " apic ID: %d", entry.apicID ); printf( ", version: %d\n", entry.apicVersion ); printf( " CPU %s usable, CPU %s the bootstrap processor\n", (entry.cpuFlags & PROCENTRY_FLAG_EN) ? "is" : "is NOT", (entry.cpuFlags & PROCENTRY_FLAG_BP) ? "is" : "is NOT" ); printf( " family: %d, model: %d, stepping: %d\n", (entry.cpuSignature >> 8) & 0x0f, (entry.cpuSignature >> 4) & 0x0f, entry.cpuSignature & 0x0f ); printf( " feature flags: 0x%08x\n", entry.featureFlags ); } static void busEntry( void ) { BusEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( ", bus type: " ); pnstr( entry.busType, 6 ); printf( "\n" ); } static void ioApicEntry( void ) { IOApicEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " apic ID: %d", entry.apicID ); printf( ", version: %d\n", entry.apicVersion ); printf( " APIC %s usable\n", (entry.apicFlags & IOAPICENTRY_FLAG_EN) ? "is" : "is NOT" ); printf( " apic address: 0x%x\n", entry.apicAddress ); } static void intEntry( void ) { IntEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " INT type: %d", (int)entry.intType ); printf( ", flags: 0x%04x\n", (int)entry.intFlags ); printf( " source bus ID: %d", (int)entry.srcBusID ); printf( ", IRQ: %d\n", (int)entry.srcBusIRQ ); printf( " destination APIC ID: %d", (int)entry.dstApicID ); printf( ", INT: %d\n", (int)entry.dstApicINT ); } static void sasEntry( void ) { SasEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", extendedtableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( " address type: " ); switch ( entry.addressType ) { case 0: printf( "I/O address\n" ); break; case 1: printf( "memory address\n" ); break; case 2: printf( "prefetch address\n" ); break; default: printf( "UNKNOWN type\n" ); break; } printf( " address base: 0x%qx\n", entry.addressBase ); printf( " address range: 0x%qx\n", entry.addressLength ); } static void bhdEntry( void ) { BhdEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", extendedtableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( " bus info: 0x%02x", entry.busInfo ); printf( " parent bus ID: %d", entry.busParent ); } static void cbasmEntry( void ) { CbasmEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", extendedtableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( " address modifier: %s\n", (entry.addressMod & 0x01) ? "subtract" : "add" ); printf( " predefined range: 0x%08x", entry.predefinedRange ); } /* * */ static void pnstr( char* s, int c ) { char string[ MAXPNSTR + 1 ]; if ( c > MAXPNSTR ) c = MAXPNSTR; strncpy( string, s, c ); string[ c ] = '\0'; printf( "%s", string ); } ----------------------------------- cut --------------------------------------- -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Thu Sep 12 19:19:56 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA20803 for smp-outgoing; Thu, 12 Sep 1996 19:19:56 -0700 (PDT) Received: from GndRsh.aac.dev.com (GndRsh.aac.dev.com [198.145.92.241]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id TAA20798 for ; Thu, 12 Sep 1996 19:19:52 -0700 (PDT) Received: (from rgrimes@localhost) by GndRsh.aac.dev.com (8.6.12/8.6.12) id TAA25436; Thu, 12 Sep 1996 19:19:42 -0700 From: "Rodney W. Grimes" Message-Id: <199609130219.TAA25436@GndRsh.aac.dev.com> Subject: Re: mptable.c To: smp@csn.net (Steve Passe) Date: Thu, 12 Sep 1996 19:19:42 -0700 (PDT) Cc: freebsd-smp@freebsd.org, peter@spinner.dialix.com In-Reply-To: <199609130045.SAA20723@clem.systemsix.com> from Steve Passe at "Sep 12, 96 06:45:11 pm" X-Mailer: ELM [version 2.4ME+ PL11 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Hi, > > here's the tool to parse out your MP table from your motherboard. It > uses /dev/kmem so it must be built and run as root. > > Keywords: MP Spec, Configuration Table > > ----------------------------------- cut --------------------------------------- > /* > * mptable.c > */ ... I did not see this code, I can not see this code until it has a public release copyright on it... either GPL or BSD style. -- Rod Grimes rgrimes@gndrsh.aac.dev.com Accurate Automation Company Reliable computers for FreeBSD From owner-freebsd-smp Thu Sep 12 19:41:18 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA21992 for smp-outgoing; Thu, 12 Sep 1996 19:41:18 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id TAA21981 for ; Thu, 12 Sep 1996 19:41:09 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id UAA21341 for ; Thu, 12 Sep 1996 20:41:06 -0600 Message-Id: <199609130241.UAA21341@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol From: Steve Passe To: freebsd-smp@freebsd.org Subject: new mptable.c Mime-Version: 1.0 Content-Type: text/plain Date: Thu, 12 Sep 1996 20:41:06 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, here's the new version, someone requested changes... Keywords: MP Spec, Configuration Table ----------------------------------- cut --------------------------------------- /* * Copyright (c) 1996, by Steve Passe * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. The name of the developer may NOT be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * mptable.c */ #define MP_SIG 0x5f504d5f /* _MP_ */ #define EXTENDED_PROCESSING_READY #define OEM_PROCESSING_READY_NOT /** cheat for now, hardcode address */ #define CHEATING_NOT #if 0 /* XXPRESS */ #define CHEAT_ADDRESS 0x000f7ba0 #else /* GA586DX */ #define CHEAT_ADDRESS 0x000f0c80 #endif #include #include #include #include #include #if 0 #include #include #include what else? and is it worth it? #else #define KERNBASE ((vm_offset_t)0xf0000000) #endif #define BIOS_BASE 0xf0000 #define BIOS_SIZE 0x10000 #define ONE_KBYTE 1024 #define PROCENTRY_FLAG_EN 0x01 #define PROCENTRY_FLAG_BP 0x02 #define IOAPICENTRY_FLAG_EN 0x01 #define MAXPNSTR 132 char* whereStrings[] = { "Extended BIOS Data Area", "@ top of memory", "BIOS" }; typedef struct TABLE_ENTRY { u_char type; u_char length; char name[ 32 ]; } tableEntry; tableEntry basetableEntryTypes[] = { { 0, 20, "Processor" }, { 1, 8, "Bus" }, { 2, 8, "I/O APIC" }, { 3, 8, "I/O INT" }, { 4, 8, "Local INT" } }; tableEntry extendedtableEntryTypes[] = { { 128, 20, "System Address Space" }, { 129, 8, "Bus Heirarchy" }, { 130, 8, "Compatibility Bus Address" } }; /* MP Floating Pointer Structure */ typedef struct MPFPS { char signature[ 4 ]; void* pap; u_char length; u_char spec_rev; u_char checksum; u_char mpfb1; u_char mpfb2; u_char mpfb3; u_char mpfb4; u_char mpfb5; } mpfps_t; /* MP Configuration Table Header */ typedef struct MPCTH { char signature[ 4 ]; u_short base_table_length; u_char spec_rev; u_char checksum; u_char oem_id[ 8 ]; u_char product_id[ 12 ]; void* oem_table_pointer; u_short oem_table_size; u_short entry_count; void* apic_address; u_short extended_table_length; u_char extended_table_checksum; u_char reserved; } mpcth_t; typedef struct PROCENTRY { u_char type; u_char apicID; u_char apicVersion; u_char cpuFlags; u_long cpuSignature; u_long featureFlags; u_long reserved1; u_long reserved2; } ProcEntry; typedef struct BUSENTRY { u_char type; u_char busID; char busType[ 6 ]; } BusEntry; typedef struct IOAPICENTRY { u_char type; u_char apicID; u_char apicVersion; u_char apicFlags; void* apicAddress; } IOApicEntry; typedef struct INTENTRY { u_char type; u_char intType; u_short intFlags; u_char srcBusID; u_char srcBusIRQ; u_char dstApicID; u_char dstApicINT; } IntEntry; /* * extended entry type structures */ typedef struct SASENTRY { u_char type; u_char length; u_char busID; u_char addressType; u_int64_t addressBase; u_int64_t addressLength; } SasEntry; typedef struct BHDENTRY { u_char type; u_char length; u_char busID; u_char busInfo; u_char busParent; u_char reserved[ 3 ]; } BhdEntry; typedef struct CBASMENTRY { u_char type; u_char length; u_char busID; u_char addressMod; u_int predefinedRange; } CbasmEntry; static void apic_probe( vm_offset_t* paddr, int* where ); static void MPConfigDefault( int featureByte ); static int MPFloatingPointer( vm_offset_t paddr, int where, mpfps_t* mpfps ); static void MPConfigTableHeader( void* pap ); static int readType( void ); static void seekEntry( vm_offset_t addr ); static void readEntry( void* entry, int size ); static void processorEntry( void ); static void busEntry( void ); static void ioApicEntry( void ); static void intEntry( void ); static void sasEntry( void ); static void bhdEntry( void ); static void cbasmEntry( void ); static void pnstr( char* s, int c ); /* global data */ int kfd; /* * */ int main( int argc, char *argv[] ) { vm_offset_t paddr; int where; mpfps_t mpfps; int defaultConfig; /* open kernel memory for access to MP structures */ if ( (kfd = open( "/dev/kmem", O_RDONLY )) < 0 ) { perror( "kmem open" ); exit( 1 ); } /* probe for MP structures */ apic_probe( &paddr, &where ); if ( where <= 0 ) { fprintf( stderr, "\n MP Table NOT found!!!\n\n" ); return 1; } printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); printf( "\nFound MP Table in %s, physical addr: 0x%08x\n", whereStrings[ where - 1 ], paddr ); printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); /* analyze the MP Floating Pointer Structure */ MPFloatingPointer( paddr, where, &mpfps ); printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); /* check whether an MP config table exists */ if ( defaultConfig = mpfps.mpfb1 ) { MPConfigDefault( defaultConfig ); } else { MPConfigTableHeader( mpfps.pap ); } printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); return 0; } /* * set PHYSICAL address of MP floating pointer structure */ static void apic_probe( vm_offset_t* paddr, int* where ) { #if defined( CHEATING ) /** cheat for now, hardcode address */ *paddr = (vm_offset_t)CHEAT_ADDRESS; /** cheat again, where we found it */ *where = 3; #else /** CHEATING */ /* * c rewrite of apic_probe() by Jack F. Vogel */ int x; unsigned short segment; vm_offset_t target; unsigned int buffer[ 16384 ]; if ( 1 ) { /* why can't I access kmem below 0xf0010000? */ fprintf( stderr, "\nWarning: EBDA support is BROKEN!!!\n" ); } else { /* EBDA is @ 40:0e in real-mode terms */ seekEntry( (vm_offset_t)0x040e + KERNBASE ); readEntry( &segment, 2 ); if ( segment ) /* search EBDA */ { target = (vm_offset_t)segment << 4; seekEntry( target + KERNBASE ); readEntry( buffer, ONE_KBYTE ); for ( x = 0; x < ONE_KBYTE / sizeof ( unsigned int ); ++x ) { if ( buffer[ x ] == MP_SIG ) { *where = 1; *paddr = (x * sizeof( unsigned int )) + target; return; } } } } # if 0 /** we should read CMOS for real top of mem, for now: */ # else /* base of the last 1K of 640K */ target = 0x9fc00; # endif seekEntry( target + KERNBASE); readEntry( buffer, ONE_KBYTE ); for ( x = 0; x < ONE_KBYTE / sizeof ( unsigned int ); ++x ) { if ( buffer[ x ] == MP_SIG ) { *where = 2; *paddr = (x * sizeof( unsigned int )) + target; return; } } /* search the BIOS */ seekEntry( BIOS_BASE + KERNBASE ); readEntry( buffer, BIOS_SIZE ); for ( x = 0; x < BIOS_SIZE / sizeof( unsigned int ); ++x ) { if ( buffer[ x ] == MP_SIG ) { *where = 3; *paddr = (x * sizeof( unsigned int )) + BIOS_BASE; return; } } *where = 0; *paddr = (vm_offset_t)0; #endif /** CHEATING */ } /* * */ static int MPFloatingPointer( vm_offset_t paddr, int where, mpfps_t* mpfps ) { vm_offset_t vaddr; /* convert physical address to a virtual address */ vaddr = paddr + (vm_offset_t)KERNBASE; /* read in mpfps structure*/ seekEntry( vaddr ); readEntry( mpfps, sizeof( mpfps_t ) ); /* show its contents */ printf( "MP Floating Pointer Structure:\n\n" ); printf( " location:\t\t\t", where ); switch ( where ) { case 0: printf( "NOT found!\n" ); exit( 1 ); case 1: printf( "EBDA\n" ); break; case 2: printf( "base memory\n" ); break; case 3: printf( "BIOS\n" ); break; default: printf( "BOGUS!\n" ); exit( 1 ); } printf( " physical address:\t\t0x%08x\n", paddr ); printf( " signature:\t\t\t'" ); pnstr( mpfps->signature, 4 ); printf( "'\n" ); printf( " length:\t\t\t%d bytes\n", mpfps->length * 16 ); printf( " version:\t\t\t1.%1d\n", mpfps->spec_rev ); printf( " checksum:\t\t\t0x%02x\n", mpfps->checksum ); /* bits 0:6 are RESERVED */ if ( mpfps->mpfb2 & 0x7f ) { printf( " warning, MP feature byte 2: 0x%02x\n" ); } /* bit 7 is IMCRP */ printf( " mode:\t\t\t\t%s\n", (mpfps->mpfb2 & 0x80) ? "PIC" : "Virtual Wire" ); /* MP feature bytes 3-5 are expected to be ZERO */ if ( mpfps->mpfb3 ) printf( " warning, MP feature byte 3 NONZERO!\n" ); if ( mpfps->mpfb4 ) printf( " warning, MP feature byte 4 NONZERO!\n" ); if ( mpfps->mpfb5 ) printf( " warning, MP feature byte 5 NONZERO!\n" ); } /* * */ static void MPConfigDefault( int featureByte ) { printf( " MP default config type: %d\n\n", featureByte ); switch ( featureByte ) { case 1: printf( " bus: ISA, APIC: 82489DX\n" ); break; case 2: printf( " bus: EISA, APIC: 82489DX\n" ); break; case 3: printf( " bus: EISA, APIC: 82489DX\n" ); break; case 4: printf( " bus: MCA, APIC: 82489DX\n" ); break; case 5: printf( " bus: ISA+PCI, APIC: Integrated\n" ); break; case 6: printf( " bus: EISA+PCI, APIC: Integrated\n" ); break; case 7: printf( " bus: MCA+PCI, APIC: Integrated\n" ); break; default: printf( " future type\n" ); break; } } /* * */ static void MPConfigTableHeader( void* pap ) { vm_offset_t vaddr; mpcth_t cth; int totalSize; int count; int type; vm_offset_t voemtp; void* oemdata; if ( pap == 0 ) { printf( "MP Configuration Table Header MISSING!\n" ); exit( 1 ); } /* convert physical address to virtual address */ vaddr = (vm_offset_t)pap + (vm_offset_t)KERNBASE; /* read in cth structure */ seekEntry( vaddr ); readEntry( &cth, sizeof( cth ) ); printf( "MP Config Table Header:\n\n" ); printf( " physical address:\t\t0x%08x\n", pap ); printf( " signature:\t\t\t'" ); pnstr( cth.signature, 4 ); printf( "'\n" ); printf( " base table length:\t\t%d\n", cth.base_table_length ); printf( " version:\t\t\t1.%1d\n", cth.spec_rev ); printf( " checksum:\t\t\t0x%02x\n", cth.checksum ); printf( " OEM ID:\t\t\t'" ); pnstr( cth.oem_id, 8 ); printf( "'\n" ); printf( " Product ID:\t\t\t'" ); pnstr( cth.product_id, 12 ); printf( "'\n" ); printf( " OEM table pointer:\t\t0x%08x\n", cth.oem_table_pointer ); printf( " OEM table size:\t\t%d\n", cth.oem_table_size ); printf( " entry count:\t\t\t%d\n", cth.entry_count ); printf( " local APIC address:\t\t0x%08x\n", cth.apic_address ); printf( " extended table length:\t%d\n", cth.extended_table_length ); printf( " extended table checksum:\t%d\n", cth.extended_table_checksum ); totalSize = cth.base_table_length - sizeof( struct MPCTH ); count = cth.entry_count; printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); printf( "MP Config Base Table Entries:\n\n" ); while ( count-- ) { switch ( type = readType() ) { case 0: processorEntry(); break; case 1: busEntry(); break; case 2: ioApicEntry(); break; case 3: intEntry(); break; case 4: intEntry(); break; default: printf( "Base Table HOSED!\n" ); exit( 1 ); } totalSize -= basetableEntryTypes[ type ].length; } #if defined( EXTENDED_PROCESSING_READY ) /* process any extended data */ if ( totalSize = cth.extended_table_length ) { printf( "\n-------------------------------------" ); printf( "-------------------------------------\n" ); printf( "MP Config Extended Table Entries:\n\n" ); while ( totalSize > 0 ) { switch ( type = readType() ) { case 128: sasEntry(); break; case 129: bhdEntry(); break; case 130: cbasmEntry(); break; default: printf( "Extended Table HOSED!\n" ); exit( 1 ); } totalSize -= extendedtableEntryTypes[ type-128 ].length; } } #endif /* EXTENDED_PROCESSING_READY */ /* process any OEM data */ if ( cth.oem_table_pointer && (cth.oem_table_size > 0) ) { #if defined( OEM_PROCESSING_READY ) # error your on your own here! /* convert OEM table pointer to virtual address */ voemtp = (vm_offset_t)cth.oem_table_pointer + (vm_offset_t)KERNBASE; /* read in oem table structure */ if ( (oemdata = (void*)malloc( cth.oem_table_size )) == NULL ) { perror( "oem malloc" ); exit( 1 ); } seekEntry( voemtp ); readEntry( oemdata, cth.oem_table_size ); /** process it */ free( oemdata ); #else printf( "\nyou need to modify the source to handle OEM data!\n\n" ); #endif /* OEM_PROCESSING_READY */ } } /* * */ static int readType( void ) { u_char type; if ( read( kfd, &type, sizeof( u_char ) ) != sizeof( u_char ) ) { perror( "type read" ); exit( 1 ); } if ( lseek( kfd, -1, SEEK_CUR ) < 0 ) { perror( "type seek" ); exit( 1 ); } return (int)type; } /* * */ static void seekEntry( vm_offset_t addr ) { if ( lseek( kfd, (off_t)addr, SEEK_SET ) < 0 ) { fprintf( stderr, "\nvaddr: 0x%08x\n", addr ); perror( "kmem seek" ); exit( 1 ); } } /* * */ static void readEntry( void* entry, int size ) { if ( read( kfd, entry, size ) != size ) { perror( "readEntry" ); exit( 1 ); } } static void processorEntry( void ) { ProcEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " apic ID: %d", entry.apicID ); printf( ", version: %d\n", entry.apicVersion ); printf( " CPU %s usable, CPU %s the bootstrap processor\n", (entry.cpuFlags & PROCENTRY_FLAG_EN) ? "is" : "is NOT", (entry.cpuFlags & PROCENTRY_FLAG_BP) ? "is" : "is NOT" ); printf( " family: %d, model: %d, stepping: %d\n", (entry.cpuSignature >> 8) & 0x0f, (entry.cpuSignature >> 4) & 0x0f, entry.cpuSignature & 0x0f ); printf( " feature flags: 0x%08x\n", entry.featureFlags ); } static void busEntry( void ) { BusEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( ", bus type: " ); pnstr( entry.busType, 6 ); printf( "\n" ); } static void ioApicEntry( void ) { IOApicEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " apic ID: %d", entry.apicID ); printf( ", version: %d\n", entry.apicVersion ); printf( " APIC %s usable\n", (entry.apicFlags & IOAPICENTRY_FLAG_EN) ? "is" : "is NOT" ); printf( " apic address: 0x%x\n", entry.apicAddress ); } static void intEntry( void ) { IntEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", basetableEntryTypes[ entry.type ].name ); printf( " INT type: %d", (int)entry.intType ); printf( ", flags: 0x%04x\n", (int)entry.intFlags ); printf( " source bus ID: %d", (int)entry.srcBusID ); printf( ", IRQ: %d\n", (int)entry.srcBusIRQ ); printf( " destination APIC ID: %d", (int)entry.dstApicID ); printf( ", INT: %d\n", (int)entry.dstApicINT ); } static void sasEntry( void ) { SasEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", extendedtableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( " address type: " ); switch ( entry.addressType ) { case 0: printf( "I/O address\n" ); break; case 1: printf( "memory address\n" ); break; case 2: printf( "prefetch address\n" ); break; default: printf( "UNKNOWN type\n" ); break; } printf( " address base: 0x%qx\n", entry.addressBase ); printf( " address range: 0x%qx\n", entry.addressLength ); } static void bhdEntry( void ) { BhdEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", extendedtableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( " bus info: 0x%02x", entry.busInfo ); printf( " parent bus ID: %d", entry.busParent ); } static void cbasmEntry( void ) { CbasmEntry entry; /* read it into local memory */ readEntry( &entry, sizeof( entry ) ); printf( "--\n%s\n", extendedtableEntryTypes[ entry.type ].name ); printf( " bus ID: %d", entry.busID ); printf( " address modifier: %s\n", (entry.addressMod & 0x01) ? "subtract" : "add" ); printf( " predefined range: 0x%08x", entry.predefinedRange ); } /* * */ static void pnstr( char* s, int c ) { char string[ MAXPNSTR + 1 ]; if ( c > MAXPNSTR ) c = MAXPNSTR; strncpy( string, s, c ); string[ c ] = '\0'; printf( "%s", string ); } ----------------------------------- cut --------------------------------------- -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Thu Sep 12 21:21:22 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id VAA02834 for smp-outgoing; Thu, 12 Sep 1996 21:21:22 -0700 (PDT) Received: from mx.serv.net (mx.serv.net [199.201.191.10]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id VAA02778 for ; Thu, 12 Sep 1996 21:21:17 -0700 (PDT) Received: from MindBender.serv.net by mx.serv.net (8.7.5/SERV Revision: 2.30 † id VAA07294; Thu, 12 Sep 1996 21:21:21 -0700 (PDT) Received: from localhost.HeadCandy.com (michaelv@localhost.HeadCandy.com [127.0.0.1]) by MindBender.serv.net (8.7.5/8.7.3) with SMTP id VAA01364; Thu, 12 Sep 1996 21:20:56 -0700 (PDT) Message-Id: <199609130420.VAA01364@MindBender.serv.net> X-Authentication-Warning: MindBender.serv.net: Host michaelv@localhost.HeadCandy.com [127.0.0.1] didn't use HELO protocol To: Terry Lambert cc: smp@csn.net (Steve Passe), peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of Thu, 12 Sep 96 11:38:32 -0700. <199609121838.LAA07343@phaeton.artisoft.com> Date: Thu, 12 Sep 1996 21:20:51 -0700 From: "Michael L. VanLoon -- HeadCandy.com" Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >> >> One thing I'm not clear about from the IO apic docs yet is whether there >> >> are 15 cpu's and 15 IO apic's, or whether there's a limit of 15 devices on >> >> the APIC bus. >> > 1 BP, 31 (AP | IO APIC) (2^5 == 32) >> its a four bit register, where do you get 2^5, am I missing something? >Oh, ugh. Sequent supports 32 processors. I wonder how? I don't believe Sequent relies on a whole lot of Intel-brand glue... ----------------------------------------------------------------------------- Michael L. VanLoon michaelv@MindBender.serv.net --< Free your mind and your machine -- NetBSD free un*x >-- NetBSD working ports: 386+PC, Mac 68k, Amiga, Atari 68k, HP300, Sun3, Sun4/4c/4m, DEC MIPS, DEC Alpha, PC532, VAX, MVME68k, arm32... NetBSD ports in progress: PICA, others... ----------------------------------------------------------------------------- From owner-freebsd-smp Thu Sep 12 23:41:59 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id XAA25154 for smp-outgoing; Thu, 12 Sep 1996 23:41:59 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id XAA25142 for ; Thu, 12 Sep 1996 23:41:50 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id XAA14603; Thu, 12 Sep 1996 23:39:33 -0700 (PDT) Message-Id: <199609130639.XAA14603@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: Terry Lambert , smp@csn.net (Steve Passe) cc: peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: Intel XXpress - some SMP benchmarks In-reply-to: Your message of "Thu, 12 Sep 1996 14:42:50 PDT." <199609122142.OAA07599@phaeton.artisoft.com> Date: Thu, 12 Sep 1996 23:39:32 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Terry Lambert writes: > > if we have sparse tables, ie 16 entries (most of which will be empty), code > > like the following becomes very inefficient: > > > > #ifdef SMP > > for (j = i = 0; i < SPARSE_TABLE_SIZE; i++) { > > if (p == SMPcurproc[i]) > > j++; > > } > > ... > > Use it like a hash: > > #ifdef SPARSE > int cpuid[ MAX_CPUID]; /* filled out by probe sequence*/ > #define CPUID(x) (cpuid[ x]) > #else /* !SPARSE*/ > #define CPUID(x) (x) > #endif /* !SPARSE*/ > > Obviously, this would save one indirection per reference, which shouldn't > be too frequent anyway. I use a virtual CPU mapping scheme which is: -- BSP -> virtual number 0 -- others mapped consecutively from there (doesn't really matter which order, but I use APIC id order). Well, the way I'm doing it now in the code I'm writing is something like: int apic_to_virtual[MAX_APIC_ID]; int virtual_to_apic[MAX_CPUS]; #define CPUNUM(x) (cpunum[x]) #define CUR_CPUNUM() (cpunum[cur_apicid()]) The APIC register reference takes so long that the array reference is simply absorbed in the overhead. I always get the virtual number (and possibly the APIC id, if necessary), then operate from there. Walking a sparse array takes up enough other overhead (checking for whether you're done as well as not using non-existent entries) that translating back to the actual APIC ID is OK. If anything it simplifies the code to walk only consecutive numbers. -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" From owner-freebsd-smp Thu Sep 12 23:48:50 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id XAA25623 for smp-outgoing; Thu, 12 Sep 1996 23:48:50 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id XAA25609 for ; Thu, 12 Sep 1996 23:48:37 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id XAA14621; Thu, 12 Sep 1996 23:45:58 -0700 (PDT) Message-Id: <199609130645.XAA14621@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: Terry Lambert cc: smp@csn.net, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) In-reply-to: Your message of "Thu, 12 Sep 1996 14:51:41 PDT." <199609122151.OAA07647@phaeton.artisoft.com> Date: Thu, 12 Sep 1996 23:45:57 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Terry Lambert writes: > > > I *believe* that the BP is guranteed an APIC ID of 0. > > > > No. The rules are: > > > > "default configuration": > > CPUs must be numbered consecutively starting with 0, in any order. > > The XXPRESS fails this, then, since it goes 0, 2 (not 0, 1). > > Does consecutively mean "insreasing" as opposed to "increasing > monotonically"? There were two sections to the "rules" I mentioned. One was the above, for "default" configurations which have only 2 CPUs and a fixed layout. The second was when the MP Configuration Table is used. The only rule it had was that one of the CPUs must have APIC id #0. > > Also realize that when a processor goes through INIT, the APIC id will be > > reset to the hardware id. This means if your OS is in the middle of doing > > a bunch of things and then sends an INIT to one of the CPUs, you have a big > > potential problem there. To solve this, you either don't reassign APIC ids, > > or never send an INIT when the system is up and running. > > Or you can reassign ID's, as long as you back fill holes instead of > shifting them down. That way an ID can never collide with a reassigned > ID following an INIT. > > You just have to be prepared to deal with INIT by taking the ID and > remapping it back into the backfill location. Sure. The problem I mentioned was an exclusive condition. Either: -- Only send a startup sequence (which resets a CPU's APIC id to the hardware id) at system startup time, or -- Don't reassign APIC ids. If you do both, then the temporary confusion when a CPU is restarted and before it can boot up far enough to set the APIC id to the different value is a serious race condition in interprocessor interrupts sent to that APIC id number. -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" From owner-freebsd-smp Fri Sep 13 00:19:32 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id AAA29327 for smp-outgoing; Fri, 13 Sep 1996 00:19:32 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id AAA29321 for ; Fri, 13 Sep 1996 00:19:25 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id AAA14688; Fri, 13 Sep 1996 00:16:19 -0700 (PDT) Message-Id: <199609130716.AAA14688@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: Steve Passe cc: terry@lambert.org, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) In-reply-to: Your message of "Thu, 12 Sep 1996 17:07:15 MDT." <199609122307.RAA20132@clem.systemsix.com> Date: Fri, 13 Sep 1996 00:16:18 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Steve Passe writes: > > Also realize that when a processor goes through INIT, the APIC id will be > > reset to the hardware id. This means if your OS is in the middle of doing > > pentium processor manual vol3, 19.3.1.11.3 'Initialization Reset (INIT)': > > "INIT is a software reset, and is delivered as a bus message. INIT has > the same effect on the Local APIC as the power-up Reset, except that the > APIC ID and the Arb ID registers are not affected." This is *different* for different parts. *don't* rely on it not resetting the APIC state! NOTE that the "Pentium" manual refers to the Pentium APIC. The 82489DX manual is necessary for other important information. > Everyone please bear with me here, this gets ugly, > but I think we are getting close to resolution. > > --- > > It's not a problem of the first STARTUP being "missed", the problem > > (which is only present on the Pentium integrated APIC) is as follows: > > When the Pentium is in the state waiting for a STARTUP, an INIT > > "in the state waiting for a STARTUP", is this a state reached by hardware > RESET? or perhaps I'm asking what this state is. Is it as if a 'hlt' > instruction had been executed? It is the state after an INIT, when the Pentium or Pentium Pro thinks it is not the "boot processor". The essential issue is that you *don't* *know* what state it is in!!! It might have been intercepted by the BIOS at boot time, it might be using an architecture where the "secondary processor" internal state bit isn't being set, etc... This allowed much more flexibility for them to built fault-tolerant hardware based on the APIC stuff built into the Pentium CPU (it was really designed to minimize hardware for the 2-CPU case). The Pentium Pro is again much more flexible along these lines. > > doesn't get processed, but it is latched. The STARTUP then allows the > > INIT to be executed, which effectively kills the STARTUP message (this is > > where the time-delays in the recommended INIT/STARTUP/STARTUP process > > come from, to allow the CPU to finish the INIT process), so you need > > another STARTUP message. > > so if I understand this we have: > > 2nd CPU is waiting for a STARTUP IPI > 1st CPU does INIT, > 1st CPU waits for INIT > to run > 2nd CPU latches INIT IPI, DOESN'T process it yet. > 1st CPU finishes wait > 1st CPU does STARTUP IPI > 2nd CPU catches STARTUP IPI, > starts to vector thru supplied STARTUP vector > 2nd CPU immediately catches INIT IPI, > aborting STARTUP IPI Up to here is correct. The real finish is: 1st CPU send second STARTUP IPI 2nd CPU responds as it is supposed to, and jumps to startup vector. That sequence is only valid for when the "secondary processor" flag is set. In many cases (such as with the XXPRESS), this flag isn't set, and an INIT is REALLY a RESET, and it doesn't halt in the weird state waiting for the STARTUP IPI. In theory, with the XXPRESS, all you have to do is set the warm-boot vector and send it an INIT, so you following comments should be valid: > INIT causes vector thru warm-boot to bootMP() > bootMP() gets 2nd CPU initialized and running Again, it's simpler to just use the working sequence in all instances. > to get the XXPRESS box working I HAD to preceed the INIT IPI with > a setup of the BIOS warm-start vector to the bootMP() code. Yes, the XXPRESS architecture doesn't use the "secondary process" flags at all (again, fault tolerance stuff... so you can yank any CPU and have it work). > pointing it at a 'hlt' instruction, then allowing the STARTUP IPI to > provide the vector to bootMP() fails. so it would appear that BOTH > of the STARTUP IPIs (both vectoring to bootMP()) fail and/or are > ignored by the XXPRESS. I'm sure (he says with confidence) that the INIT > IPI actually travels thru the vector as we replaced the 'hlt' instruction > with an NMI and we got spontanious reboot. See above. > > The problem with trying to just do an INIT, then if that doesn't work, a > > STARTUP and INIT, is that you're never sure what the state the BIOS > > puts CPUs in on a lot of the Pentium boxes (you'd also have to wait > > to see if it responded to each part of the sequence). I spent a lot of > > time finding out that the ENTIRE sequence was required on even machines > > that I thought didn't need it (the XXPress box). The Intel folks > > spent a lot of time generating a single guaranteed sequence that > > covered all the bases. > > I'm getting a headache... SO at this point I think I buy the double > INIT IPI followed by 2 STARTUP IPS, with appropriate timings in-between > as the cure for all hardware, in all the "brokenness" we are likely to find. -- set warm-boot vector -- single INIT IPI -- wait a bit -- single STARTUP IPI -- wait a bit -- single STARTUP IPI -- wait for CPU's response. > I'm just not clear on the appropriate thing to do with the warm-start vector. > I believe its needed, and that the XXPRESS for one needs it to point to > the actual 2nd CPU boot code. Can we expect to be able to vector to > the boot code on all hardware, or will we encounter boxes that run the > boot code via the INIT IPI warm-start vector, then re-run it via one > of the following STARTUP IPIs??????? Some use the warm-boot vector, and some the vector from the STARTUP IPIs. All Pentium Pro machines I've seen, for example, use the STARTUP IPI vector. > is there a definitive code sample that encompasses all this. in the > MP spec they have the pseudo code with 2 STARTUP IPIs, but no actual code. Pretty much. The Linux-SMP startup code, or the code in my bootloader GRUB, are pretty universal. I'm still tweaking the one I'm going to place on my web site as a "canonical" startup sequence. I found some definition issues that won't likely be seen in practice, but are technically wrong with the Linux and FreeBSD startup codes. > the pentium manual shows some real code with timings, but NO double > STARTUP sequence.... The pentium manual is out-of-date, and doesn't include what in the MP spec 1.4 is really an algorithm that is a workaround to account for all the problems that have been encountered. Use the 1.4 spec!!!! Don't even refer to the 1.1 spec, as some parts of it (for example the bus hierarchy stuff) is simply inconsistent. -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" From owner-freebsd-smp Fri Sep 13 01:14:17 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id BAA01758 for smp-outgoing; Fri, 13 Sep 1996 01:14:17 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id BAA01753 for ; Fri, 13 Sep 1996 01:14:13 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id CAA23090; Fri, 13 Sep 1996 02:02:57 -0600 Message-Id: <199609130802.CAA23090@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: erich@uruk.org cc: terry@lambert.org, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org Subject: Re: (long) Intel SMP info (was -> Re: Intel XXpress - some SMP benchmarks) In-reply-to: Your message of "Fri, 13 Sep 1996 00:16:18 PDT." <199609130716.AAA14688@uruk.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 13 Sep 1996 02:02:57 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, its all starting to make sense... > That sequence is only valid for when the "secondary processor" flag > is set. In many cases (such as with the XXPRESS), this flag isn't > set, and an INIT is REALLY a RESET, and it doesn't halt in the is there anyway to programmatically determine the state of this flag? > Yes, the XXPRESS architecture doesn't use the "secondary process" flags at > all (again, fault tolerance stuff... so you can yank any CPU and have is this because of the mb design, ie what pins are strapped during RESET, or is it accomplished by the BIOS doing something? -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Fri Sep 13 10:49:48 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id KAA04820 for smp-outgoing; Fri, 13 Sep 1996 10:49:48 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id KAA04807 for ; Fri, 13 Sep 1996 10:49:44 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id KAA09209; Fri, 13 Sep 1996 10:46:36 -0700 From: Terry Lambert Message-Id: <199609131746.KAA09209@phaeton.artisoft.com> Subject: Re: Intel XXpress - some SMP benchmarks To: erich@uruk.org Date: Fri, 13 Sep 1996 10:46:36 -0700 (MST) Cc: terry@lambert.org, smp@csn.net, peter@spinner.dialix.com, rv@groa.uct.ac.za, freebsd-smp@freebsd.org In-Reply-To: <199609130639.XAA14603@uruk.org> from "erich@uruk.org" at Sep 12, 96 11:39:32 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > The APIC register reference takes so long that the array reference is > simply absorbed in the overhead. I always get the virtual number (and > possibly the APIC id, if necessary), then operate from there. Doesn't it gall you anyway? 8-) 8-). Guess I'm just a computational nanosecond kind of guy -- it's people like me what cause unrest (and research into quantum computing). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Fri Sep 13 12:00:25 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id MAA08840 for smp-outgoing; Fri, 13 Sep 1996 12:00:25 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id MAA08832 for ; Fri, 13 Sep 1996 12:00:21 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id NAA26181; Fri, 13 Sep 1996 13:00:17 -0600 Message-Id: <199609131900.NAA26181@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: freebsd-smp@freebsd.org cc: peter@spinner.dialix.com Subject: writing new apic_startup Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 13 Sep 1996 13:00:16 -0600 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, I'm starting to re-code apic_startup() armed with the knowqledge we've gained the last several days. One of the things I want to do is use real timing loops. Is there a kernel facility available at the point apic_startup() is called (middle of init386() in machdep.c) that will give me blocking delays in the order of 10 usec to 10 msec? Failing that does anyone see any reason why I shouldn't use the APIC timer? Is it currently used on the boot CPU for anything else? -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Fri Sep 13 12:56:07 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id MAA12543 for smp-outgoing; Fri, 13 Sep 1996 12:56:07 -0700 (PDT) Received: from GndRsh.aac.dev.com (GndRsh.aac.dev.com [198.145.92.241]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id MAA12499 for ; Fri, 13 Sep 1996 12:55:58 -0700 (PDT) Received: (from rgrimes@localhost) by GndRsh.aac.dev.com (8.6.12/8.6.12) id MAA01166; Fri, 13 Sep 1996 12:55:44 -0700 From: "Rodney W. Grimes" Message-Id: <199609131955.MAA01166@GndRsh.aac.dev.com> Subject: Re: writing new apic_startup To: smp@csn.net (Steve Passe) Date: Fri, 13 Sep 1996 12:55:44 -0700 (PDT) Cc: freebsd-smp@freebsd.org, peter@spinner.dialix.com In-Reply-To: <199609131900.NAA26181@clem.systemsix.com> from Steve Passe at "Sep 13, 96 01:00:16 pm" X-Mailer: ELM [version 2.4ME+ PL11 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Hi, > > I'm starting to re-code apic_startup() armed with the knowqledge we've gained > the last several days. One of the things I want to do is use real timing > loops. > > Is there a kernel facility available at the point apic_startup() > is called (middle of init386() in machdep.c) that will give me > blocking delays in the order of 10 usec to 10 msec? > > Failing that does anyone see any reason why I shouldn't use the APIC timer? > Is it currently used on the boot CPU for anything else? The very reason Intel added a timer to the APIC was because of the need to do these timings during SMP initilization. Please do write your code to use the APIC timer. Please do write a set of acquire_apic_timer(), release_apic_timer() lock functions as well as a set_apic_timer() and read_apic_timer() so that we can insure exclusive use of the timer: if (acquire_apic_timer()) panic("APIC timer in use when attempting to aquire"); /* Do what you need to do with timing */ if (release_apic_timer()) panic("APIC timer was already released"); The panics could be initially imbeded in the acquire/release, but that would not allow for more generized use of the apic timer for other purposes that can deal with the fact that it is already in use (ie, block the process/thread). -- Rod Grimes rgrimes@gndrsh.aac.dev.com Accurate Automation Company Reliable computers for FreeBSD From owner-freebsd-smp Fri Sep 13 19:31:38 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA08630 for smp-outgoing; Fri, 13 Sep 1996 19:31:38 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id TAA08611 for ; Fri, 13 Sep 1996 19:31:24 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id TAA16417; Fri, 13 Sep 1996 19:31:46 -0700 (PDT) Message-Id: <199609140231.TAA16417@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: "Rodney W. Grimes" cc: smp@csn.net (Steve Passe), smp@freebsd.org, peter@spinner.dialix.com Subject: Re: writing new apic_startup In-reply-to: Your message of "Fri, 13 Sep 1996 12:55:44 PDT." <199609131955.MAA01166@GndRsh.aac.dev.com> Date: Fri, 13 Sep 1996 19:31:46 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk "Rodney W. Grimes" writes: > > Failing that does anyone see any reason why I shouldn't use the APIC > > timer? Is it currently used on the boot CPU for anything else? > > The very reason Intel added a timer to the APIC was because of the need to > do these timings during SMP initilization. Please do write your code to use > the APIC timer. I'd *highly* suggest that anybody doing this kind of stuff look at the Pentium eratta. I remember at least one bug in some versions about the APIC timer being somewhat off in some circumstances. -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" From owner-freebsd-smp Fri Sep 13 20:07:11 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id UAA10958 for smp-outgoing; Fri, 13 Sep 1996 20:07:11 -0700 (PDT) Received: from GndRsh.aac.dev.com (GndRsh.aac.dev.com [198.145.92.241]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id UAA10947 for ; Fri, 13 Sep 1996 20:07:04 -0700 (PDT) Received: (from rgrimes@localhost) by GndRsh.aac.dev.com (8.6.12/8.6.12) id UAA01724; Fri, 13 Sep 1996 20:06:34 -0700 From: "Rodney W. Grimes" Message-Id: <199609140306.UAA01724@GndRsh.aac.dev.com> Subject: Re: writing new apic_startup To: erich@uruk.org Date: Fri, 13 Sep 1996 20:06:34 -0700 (PDT) Cc: smp@csn.net, smp@freebsd.org, peter@spinner.dialix.com In-Reply-To: <199609140231.TAA16417@uruk.org> from "erich@uruk.org" at "Sep 13, 96 07:31:46 pm" X-Mailer: ELM [version 2.4ME+ PL11 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > "Rodney W. Grimes" writes: > > > > Failing that does anyone see any reason why I shouldn't use the APIC > > > timer? Is it currently used on the boot CPU for anything else? > > > > The very reason Intel added a timer to the APIC was because of the need to > > do these timings during SMP initilization. Please do write your code to use > > the APIC timer. > > I'd *highly* suggest that anybody doing this kind of stuff look at the > Pentium eratta. I remember at least one bug in some versions about the > APIC timer being somewhat off in some circumstances. The only reference I find in my old May 1995 Pentium Errata book (242480-004) with respect to this is Specification Clarification number 5. Basically timer interval 0 is of 1 PIC clock duration, where as the other timer intervals are of ``divisor'' length. A future version of the processor (this book only covers upto stepping C5 == CPUID xx5) will totally eliminate interval zero (the timer will count down 5,4,3,2,1,5,4,3,2,1) and the interrupt will occur at the end of interval 1. So those writing this code should infact read this errata, I will fax it to whomever is undertaking this task. If Erich knows of some other errata about the APIC timer I have over looked I'll obtain a later spec update and send that out as well. -- Rod Grimes rgrimes@gndrsh.aac.dev.com Accurate Automation Company Reliable computers for FreeBSD From owner-freebsd-smp Sat Sep 14 15:49:03 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id PAA13866 for smp-outgoing; Sat, 14 Sep 1996 15:49:03 -0700 (PDT) Received: from po2.glue.umd.edu (po2.glue.umd.edu [129.2.128.45]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id PAA13860 for ; Sat, 14 Sep 1996 15:49:00 -0700 (PDT) Received: from protocol.eng.umd.edu (protocol.eng.umd.edu [129.2.98.180]) by po2.glue.umd.edu (8.7.5/8.7.3) with ESMTP id SAA27179 for ; Sat, 14 Sep 1996 18:48:57 -0400 (EDT) Received: from localhost (chuckr@localhost) by protocol.eng.umd.edu (8.7.5/8.7.3) with SMTP id SAA05465 for ; Sat, 14 Sep 1996 18:48:57 -0400 (EDT) X-Authentication-Warning: protocol.eng.umd.edu: chuckr owned process doing -bs Date: Sat, 14 Sep 1996 18:48:54 -0400 (EDT) From: Chuck Robey X-Sender: chuckr@protocol.eng.umd.edu To: FreeBSD-smp@FreeBSD.org Subject: Caching Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk Can someone explain the added intricacies of caching in a multiprocessor environment a moment? I know this can be a really complicated subject if all the details are extruciatingly covered (write-thru, write-back, etc), but my question is really pointed at making me understand why it costs so much for a Pentium p6-200 if you include 512K of cache (it ups the price over 600 bucks per chip). Does the old P5 incur the same penalty? Does not getting the p6 with cache make me incur a large penalty? Would a p5 with cache be equivalently quicker, mhz for mhz, than a p6 without it? Thats the kind of stuff I'm flailing around with, in trying to determine what kind of smp platform to buy. Help! ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@eng.umd.edu | communications topic, C programming, and Unix. 9120 Edmonston Ct #302 | Greenbelt, MD 20770 | I run Journey2 and n3lxx, both FreeBSD (301) 220-2114 | version 2.2 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Sat Sep 14 16:26:00 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id QAA17089 for smp-outgoing; Sat, 14 Sep 1996 16:26:00 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id QAA17081 for ; Sat, 14 Sep 1996 16:25:56 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id RAA06693; Sat, 14 Sep 1996 17:25:48 -0600 Message-Id: <199609142325.RAA06693@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Chuck Robey cc: FreeBSD-smp@FreeBSD.org Subject: Re: Caching In-reply-to: Your message of "Sat, 14 Sep 1996 18:48:54 EDT." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 14 Sep 1996 17:25:48 -0600 Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk Hi, > Can someone explain the added intricacies of caching in a multiprocessor > ... > but my question is really pointed at making me understand why it costs so > much for a Pentium p6-200 if you include 512K of cache (it ups the price > over 600 bucks per chip). one reason: supply and demand, ie they feel they can sell about as many as they produce at that price... it will drop. > not getting the p6 with cache make me incur a large penalty? Would a p5 p6 comes with 256K cache minimum. I would NOT give Intel the blood money for the extra cache. > with cache be equivalently quicker, mhz for mhz, than a p6 without it? > Thats the kind of stuff I'm flailing around with, in trying to determine > what kind of smp platform to buy. Help! I would NOT get a dual p6 with the orion chipset, which most still use. does anyone know whats available with the newer natoma? Be aware that you will be on the cutting edge if buying this sort of hardware today. IMHO, any decision could end up being bad, things just haven't shaken out yet. economy: dual P5 with tritonII deluxe: dual P6 with natoma For these reasons I went the economy path: GA586DX512 (dual P5, 512k cache): $350 133mHz P5: 2 x $210: $420 The basic board benches well as a uniproc machine. It includes onboard adaptec 2940UW, saving you $200+. I figured that if it didn't work out I could strip it of CPUs & memory and not loose much money. --- this document compares various cache schemes: X86 Multiprocessing Basics: http://www.chips.ibm.com/products/x86/appnote/40208.ps -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Sat Sep 14 16:45:33 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id QAA18446 for smp-outgoing; Sat, 14 Sep 1996 16:45:33 -0700 (PDT) Received: from mx.serv.net (mx.serv.net [199.201.191.10]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id QAA18438 for ; Sat, 14 Sep 1996 16:45:31 -0700 (PDT) Received: from MindBender.serv.net by mx.serv.net (8.7.5/SERV Revision: 2.30 † id QAA18995; Sat, 14 Sep 1996 16:44:43 -0700 (PDT) Received: from localhost.HeadCandy.com (michaelv@localhost.HeadCandy.com [127.0.0.1]) by MindBender.serv.net (8.7.5/8.7.3) with SMTP id QAA08975; Sat, 14 Sep 1996 16:43:56 -0700 (PDT) Message-Id: <199609142343.QAA08975@MindBender.serv.net> X-Authentication-Warning: MindBender.serv.net: Host michaelv@localhost.HeadCandy.com [127.0.0.1] didn't use HELO protocol To: Chuck Robey cc: FreeBSD-smp@freebsd.org Subject: Re: Caching In-reply-to: Your message of Sat, 14 Sep 96 18:48:54 -0400. Date: Sat, 14 Sep 1996 16:43:54 -0700 From: "Michael L. VanLoon -- HeadCandy.com" Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >Can someone explain the added intricacies of caching in a multiprocessor >environment a moment? I'm not sure why you bring up multiprocessor, since none of your questions specifically address that. >I know this can be a really complicated subject if >all the details are extruciatingly covered (write-thru, write-back, etc), >but my question is really pointed at making me understand why it costs so >much for a Pentium p6-200 if you include 512K of cache (it ups the price >over 600 bucks per chip). Actually, I don't think you're paying for the extra 256K of cache. You're paying for supply and demand. Intel makes very few 512K chips; they cost more. >Does the old P5 incur the same penalty? Does >not getting the p6 with cache make me incur a large penalty? Would a p5 >with cache be equivalently quicker, mhz for mhz, than a p6 without it? Lots of questions, some which can't be answered with any real authority. I'm not sure what you mean by the P5 incuring the same penalty, when the P5 can never be in the same circumstance (since it has no L2 cache on the chip). The P6 still has room to grow, where the P5 has topped out, specifically because of the L2 cache being on the chip. The P5 L2 cache, being on the motherboard side of the bus, can run a maximum of 66MHz. It also must block if a cache miss occurs (so it can fetch the memory and satisfy the read request). The P6 cache, being in the chip package, runs at the full speed of the processor (200MHz, for example). Plus, the P6 cache is non-blocking -- it can miss up to four requests before it blocks. Since the P6 is capable of doing speculative and out-of-order execution, it can continue to process instructions that are in the cache, if a previous instruction caused a cache miss. Read the Intel web site if you want more background on all this. Would a P5 that had on-chip L2 cache be faster than a P6 of the same speed without on-chip cache? Hard to call. But I don't think it's a very realistic question since the chances are almost zero that Intel will make a P5 with on-chip L2 cache. Their future is all P6, and the P5 is now only a secondary market for them. >Thats the kind of stuff I'm flailing around with, in trying to determine >what kind of smp platform to buy. Help! A P6 SMP platform is almost guaranteed to be faster than a P5 SMP platform. And, even if it isn't significantly faster now, you will be able to buy 300MHz, and maybe even 400MHz P6 chips to upgrade, somewhere down the road. The Pentium has topped out. 200MHz is pretty much the end of the line for it. Sure, maybe AMD or Cyrix will eventually bring out a 300MHz Pentium-like chip. But, its performance improvements, like the 133MHz 486 (5x86), will be dubious and slight. ----------------------------------------------------------------------------- Michael L. VanLoon michaelv@MindBender.serv.net --< Free your mind and your machine -- NetBSD free un*x >-- NetBSD working ports: 386+PC, Mac 68k, Amiga, Atari 68k, HP300, Sun3, Sun4/4c/4m, DEC MIPS, DEC Alpha, PC532, VAX, MVME68k, arm32... NetBSD ports in progress: PICA, others... ----------------------------------------------------------------------------- From owner-freebsd-smp Sat Sep 14 16:49:58 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id QAA18725 for smp-outgoing; Sat, 14 Sep 1996 16:49:58 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id QAA18718 for ; Sat, 14 Sep 1996 16:49:55 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id RAA06853; Sat, 14 Sep 1996 17:49:44 -0600 Message-Id: <199609142349.RAA06853@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Steve Passe cc: Chuck Robey , FreeBSD-smp@FreeBSD.org Subject: Re: Caching In-reply-to: Your message of "Sat, 14 Sep 1996 17:25:48 MDT." <199609142325.RAA06693@clem.systemsix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 14 Sep 1996 17:49:43 -0600 Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk Hi, I said: >I would NOT get a dual p6 with the orion chipset, which most still use. >does anyone know whats available with the newer natoma? browsing another list I just saw that Tyan have 2 PPro/natoma boards available: http://www.tyan.com/s1662.htm (baby AT) http://www.tyan.com/s1668.htm (ATX) The ATX would be my choice if all others things were equal. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Sat Sep 14 20:21:14 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id UAA25889 for smp-outgoing; Sat, 14 Sep 1996 20:21:14 -0700 (PDT) Received: from po2.glue.umd.edu (po2.glue.umd.edu [129.2.128.45]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id UAA25884 for ; Sat, 14 Sep 1996 20:21:11 -0700 (PDT) Received: from thurston.eng.umd.edu (thurston.eng.umd.edu [129.2.103.25]) by po2.glue.umd.edu (8.7.5/8.7.3) with ESMTP id XAA29401; Sat, 14 Sep 1996 23:21:08 -0400 (EDT) Received: from localhost (chuckr@localhost) by thurston.eng.umd.edu (8.7.5/8.7.3) with SMTP id XAA23766; Sat, 14 Sep 1996 23:21:07 -0400 (EDT) X-Authentication-Warning: thurston.eng.umd.edu: chuckr owned process doing -bs Date: Sat, 14 Sep 1996 23:21:07 -0400 (EDT) From: Chuck Robey X-Sender: chuckr@thurston.eng.umd.edu To: Steve Passe cc: FreeBSD-smp@FreeBSD.org Subject: Re: Caching In-Reply-To: <199609142349.RAA06853@clem.systemsix.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk On Sat, 14 Sep 1996, Steve Passe wrote: > Hi, > > I said: > >I would NOT get a dual p6 with the orion chipset, which most still use. > >does anyone know whats available with the newer natoma? > > browsing another list I just saw that Tyan have 2 PPro/natoma boards available: > > http://www.tyan.com/s1662.htm (baby AT) > http://www.tyan.com/s1668.htm (ATX) > > The ATX would be my choice if all others things were equal. According to what I read from Michael's posting (between yours and Michael's, it's been real helpful) then I ought to consider maybe a slower PPro, rather than a faster Pentium, on the theory that I could upgrade. This sounds reasonable. The reason I wanted to go into cache, I consider that in an smp environment, contention for the memory bus might be somewhat more strained, which might mean that cache was much more important. I was indeed already looking at the Tyan product, both because of the chipset, and because I already have a Tomcat in my other machine. OK, I'm happy, time to find out who's got the best prices, thanks for the assist. BTW, I have using the names (Orion, Natoma, whatever), I like the numbers. Tyan is asking folks about getting the last of their 430 based boards, because they say Intel isn't going to make the 430 series anymore. I haven't seen anything real using the 450K/G X yet, so I guess everthing is going to be 440. If Tyan is to be believed, that is. > > -- > Steve Passe | powered by > smp@csn.net | FreeBSD > > ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@eng.umd.edu | communications topic, C programming, and Unix. 9120 Edmonston Ct #302 | Greenbelt, MD 20770 | I run Journey2 and n3lxx, both FreeBSD (301) 220-2114 | version 2.2 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Sat Sep 14 21:33:51 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id VAA01965 for smp-outgoing; Sat, 14 Sep 1996 21:33:51 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id VAA01954 for ; Sat, 14 Sep 1996 21:33:48 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id WAA08200; Sat, 14 Sep 1996 22:33:39 -0600 Message-Id: <199609150433.WAA08200@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Chuck Robey cc: FreeBSD-smp@FreeBSD.org Subject: Re: Caching In-reply-to: Your message of "Sat, 14 Sep 1996 23:21:07 EDT." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 14 Sep 1996 22:33:39 -0600 Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk Hi, >BTW, I have using the names (Orion, Natoma, whatever), I like the numbers. >Tyan is asking folks about getting the last of their 430 based boards, >because they say Intel isn't going to make the 430 series anymore. I they must mean the original triton, ie the 430FX, the 430HX will be around awhile. >haven't seen anything real using the 450K/G X yet, so I guess everthing is >going to be 440. If Tyan is to be believed, that is. that's a flavor of orion, I believe still with certain basic flaws. ASUS P/E-P6RP7D uses it. --- >From the intel website: The phrase Orion DT is not an Intel product name. The correct product name is Intel 450KX PCIset. The phrase Orion ST is not an Intel product name. The correct product name is Intel 450GX PCIset. The phrase Triton FX is not an Intel product name. The correct product name is Intel 430FX PCIset. The phrase Triton II is not an Intel product name. The correct product name is Intel 430HX PCIset. The phrase Triton VX is not an Intel product name. The correct product name is Intel 430VX PCIset. ( some people like to call this one the triton III ) ( I couldn't find the name 'natoma' but it refers to the 440FX ) --- Also a 440fx PPro board: http://www-cs.intel.com/oem_developer/motherbd/pr_ds.htm -- Steve Passe | powered by smp@csn.net | FreeBSD -----BEGIN PGP PUBLIC KEY BLOCK----- Version: 2.6.2 mQCNAzHe7tEAAAEEAM274wAEEdP+grIrV6UtBt54FB5ufifFRA5ujzflrvlF8aoE 04it5BsUPFi3jJLfvOQeydbegexspPXL6kUejYt2OeptHuroIVW5+y2M2naTwqtX WVGeBP6s2q/fPPAS+g+sNZCpVBTbuinKa/C4Q6HJ++M9AyzIq5EuvO0a8Rr9AAUR tBlTdGV2ZSBQYXNzZSA8c21wQGNzbi5uZXQ+ =ds99 -----END PGP PUBLIC KEY BLOCK----- From owner-freebsd-smp Sat Sep 14 21:52:17 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id VAA03697 for smp-outgoing; Sat, 14 Sep 1996 21:52:17 -0700 (PDT) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id VAA03692 for ; Sat, 14 Sep 1996 21:52:15 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id WAA08310; Sat, 14 Sep 1996 22:52:08 -0600 Message-Id: <199609150452.WAA08310@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Chuck Robey cc: FreeBSD-smp@FreeBSD.org Subject: Re: Caching In-reply-to: Your message of "Sat, 14 Sep 1996 23:21:07 EDT." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 14 Sep 1996 22:52:08 -0600 Sender: owner-smp@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk Hi, >According to what I read from Michael's posting (between yours and >Michael's, it's been real helpful) then I ought to consider maybe a slower >PPro, rather than a faster Pentium, on the theory that I could upgrade. If by upgrade you mean replace the CPUs I disagree. I don't know what 150mHz PPros go for but I tend to think your better off buying whatever speed you expect to live with for the life of the board. If you spend $500 for the board, plus $400 x 2 for 2 PPro150, it doesn't make alot of economic sense to 'shelve' $800 worth of CPU, and add another $1000 of PPro266 to keep a $500 (probably by then $300 new) board up to date. Unless PPro150's are CONSIDERABLY cheaper, go with the 200s. -- Steve Passe | powered by smp@csn.net | FreeBSD