From owner-freebsd-smp Wed Jul 3 23:27:24 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id XAA04686 for smp-outgoing; Wed, 3 Jul 1996 23:27:24 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id XAA04679 for ; Wed, 3 Jul 1996 23:27:18 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id XAA09982; Wed, 3 Jul 1996 23:28:40 -0700 (PDT) Message-Id: <199607040628.XAA09982@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: freebsd-smp@freebsd.org cc: erich@uruk.org Subject: *Finally* got a build/run environment for FreeBSD-SMP set up Date: Wed, 03 Jul 1996 23:28:40 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi all. I just finished getting a build/run environment for FreeBSD-SMP set up for my test SMP box (it was a pain in the rear, I'll tell you). First of all, I tried building the SMP source tree without SMP enabled. The generic kernel would *not* boot where a normal FreeBSD generic kernel would. I had to remove a lot of devices to get it to work. The problem was that after all the devices were probed, it just said: panic: double fault syncing disks... and hung. :-( The next problem is to note that some presumptions are being made which aren't good (this is other than I've mentioned before). -- "APIC_BASE" is *not* a constant. The MPS table will tell the kernel where it is. Note that all Pentium Pro machines I've tried use a different address than the one used for Pentium machines (in the Pentium it was hardwired). This will effect the boot/startup code. -- The CPU tables (lots of static data) are replicated on the order of "NCPUS". This will really bloat the static data area. Can someone comment on what's really needed here ?? Linux used such a define, but it didn't add a huge amount of overhead. I have to finish reading through the code... stay tuned! I want to make quick progress now that I have the environment set up. -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" This is my home system, so I'm speaking only for myself, not for Intel. From owner-freebsd-smp Thu Jul 4 11:42:47 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id LAA10380 for smp-outgoing; Thu, 4 Jul 1996 11:42:47 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id LAA10371 for ; Thu, 4 Jul 1996 11:42:44 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id LAA11426 for ; Thu, 4 Jul 1996 11:44:12 -0700 (PDT) Message-Id: <199607041844.LAA11426@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: freebsd-smp@freebsd.org Subject: Running SMP Date: Thu, 04 Jul 1996 11:44:12 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk After hacking the hard-coded base address of the APIC, I tried running some kernel compiles to see what kind of performance I was getting. Frankly, it was pretty rotten. Here's what I got (multiprocessor 150MHz Pentium Pro w/256K caches): FreeBSD SMP 1 CPU : 4:36 -j 2 : 4:15 FreeBSD SMP 2 CPU : 4:28 -j 2 : 4:28 This was all compiling the SMP kernel with almost no devices. I almost couldn't tell the other CPU was activated, except that "sysctl" told me in big bold letters that it was, and I kept getting kernel error messages about "calcru: negative time: ..." and such after doing it. Linux-SMP would give me at least a 1.5x speedup with a full kernel spinlock and badly parallelized makefile. What's the deal here? -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" This is my home system, so I'm speaking only for myself, not for Intel. From owner-freebsd-smp Thu Jul 4 13:16:19 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id NAA18747 for smp-outgoing; Thu, 4 Jul 1996 13:16:19 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id NAA18738 for ; Thu, 4 Jul 1996 13:16:16 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id NAA13471; Thu, 4 Jul 1996 13:13:28 -0700 From: Terry Lambert Message-Id: <199607042013.NAA13471@phaeton.artisoft.com> Subject: Re: Running SMP To: erich@uruk.org Date: Thu, 4 Jul 1996 13:13:28 -0700 (MST) Cc: freebsd-smp@freebsd.org In-Reply-To: <199607041844.LAA11426@uruk.org> from "erich@uruk.org" at Jul 4, 96 11:44:12 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > After hacking the hard-coded base address of the APIC, I tried running > some kernel compiles to see what kind of performance I was getting. > > Frankly, it was pretty rotten. Here's what I got (multiprocessor > 150MHz Pentium Pro w/256K caches): > > FreeBSD SMP 1 CPU : 4:36 > -j 2 : 4:15 > > FreeBSD SMP 2 CPU : 4:28 > -j 2 : 4:28 > > This was all compiling the SMP kernel with almost no devices. I almost > couldn't tell the other CPU was activated, except that "sysctl" told > me in big bold letters that it was, and I kept getting kernel error > messages about "calcru: negative time: ..." and such after doing it. > > Linux-SMP would give me at least a 1.5x speedup with a full kernel spinlock > and badly parallelized makefile. What's the deal here? There is a scheduler bug in the current SMP code which causes there to be an interspersal problem exiting the kernel as a scheduled entity. You can best see this by starting a kernel build on a remote system in an xterm, and then hitting return on the console. The scheduling is interleaved, and the console will be on the second processor, and the system will toggle between them. This is quite broken, and it should be ovious from the original patches I sent you that this is an attempt to preserve cache locality (the Jack Vogel patches lost cache locality by anonymously scheduling processes without regard to "preferred processor"). Any input you have on how to correct this would be welcome; I've beem dealing with Lite2 integration and porting issues, and wanted to ignore the scheduler and page mapping if I could. 8-). If not, I'm sure Peter or Poul or one of the others will get to it (or you could), before I have a chance. I have a PPC milestone commitment coming due soon, or I'd jump on it. 8-(. Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Jul 4 17:11:48 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA21277 for smp-outgoing; Thu, 4 Jul 1996 17:11:48 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id RAA21262 for ; Thu, 4 Jul 1996 17:11:41 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id RAA12069; Thu, 4 Jul 1996 17:11:48 -0700 (PDT) Message-Id: <199607050011.RAA12069@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: Terry Lambert cc: freebsd-smp@freebsd.org Subject: Re: Running SMP In-reply-to: Your message of "Thu, 04 Jul 1996 13:13:28 PDT." <199607042013.NAA13471@phaeton.artisoft.com> Date: Thu, 04 Jul 1996 17:11:47 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Terry Lambert writes: ... > This is quite broken, and it should be ovious from the original patches I > sent you that this is an attempt to preserve cache locality (the Jack > Vogel patches lost cache locality by anonymously scheduling processes > without regard to "preferred processor"). > > Any input you have on how to correct this would be welcome; I've beem > dealing with Lite2 integration and porting issues, and wanted to > ignore the scheduler and page mapping if I could. 8-). The order which I was going to try to fix things was: -- startup code -- generalization to more than 2 CPUs -- scheduler I could munge the scheduler first easy enough. More on this later. It's still important to get the details on FreeBSD-SMP's model of how it uses the x86 data structures such as the TSS. Getting an accurate image of this is particularly important to how the > 2 CPU generalization goes (i.e. what static data is really necessary here... allocating something upwards of 64K per CPU statically at compile-time would be very annoying). I still haven't looked through that part yet, admittedly. Linux-SMP has one GDT with one TSS per process, simply guaranteeing that no more than one processor will access any particular TSS at a time. Could someone comment on how FreeBSD currently does this in more detail ? -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" This is my home system, so I'm speaking only for myself, not for Intel. From owner-freebsd-smp Thu Jul 4 17:35:23 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA23951 for smp-outgoing; Thu, 4 Jul 1996 17:35:23 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id RAA23943 for ; Thu, 4 Jul 1996 17:35:19 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id RAA13959; Thu, 4 Jul 1996 17:32:29 -0700 From: Terry Lambert Message-Id: <199607050032.RAA13959@phaeton.artisoft.com> Subject: Re: Running SMP To: erich@uruk.org Date: Thu, 4 Jul 1996 17:32:29 -0700 (MST) Cc: terry@lambert.org, freebsd-smp@freebsd.org In-Reply-To: <199607050011.RAA12069@uruk.org> from "erich@uruk.org" at Jul 4, 96 05:11:47 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > It's still important to get the details on FreeBSD-SMP's model of how > it uses the x86 data structures such as the TSS. Getting an accurate > image of this is particularly important to how the > 2 CPU generalization > goes (i.e. what static data is really necessary here... allocating > something upwards of 64K per CPU statically at compile-time would be > very annoying). I still haven't looked through that part yet, admittedly. > > Linux-SMP has one GDT with one TSS per process, simply guaranteeing that > no more than one processor will access any particular TSS at a time. > > Could someone comment on how FreeBSD currently does this in more detail ? David? John? Pohl? Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Thu Jul 4 18:58:10 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id SAA00874 for smp-outgoing; Thu, 4 Jul 1996 18:58:10 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id SAA00866 for ; Thu, 4 Jul 1996 18:58:05 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id JAA18293; Fri, 5 Jul 1996 09:57:41 +0800 (WST) Message-Id: <199607050157.JAA18293@spinner.DIALix.COM> X-Mailer: exmh version 1.6.6 3/24/96 To: erich@uruk.org cc: freebsd-smp@freebsd.org Subject: Re: Running SMP In-reply-to: Your message of "Thu, 04 Jul 1996 11:44:12 MST." <199607041844.LAA11426@uruk.org> Date: Fri, 05 Jul 1996 09:57:40 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >>>erich@uruk.org said: > > After hacking the hard-coded base address of the APIC, I tried running > some kernel compiles to see what kind of performance I was getting. > > Frankly, it was pretty rotten. Here's what I got (multiprocessor > 150MHz Pentium Pro w/256K caches): > > FreeBSD SMP 1 CPU : 4:36 > -j 2 : 4:15 > > FreeBSD SMP 2 CPU : 4:28 > -j 2 : 4:28 > > This was all compiling the SMP kernel with almost no devices. I almost > couldn't tell the other CPU was activated, except that "sysctl" told > me in big bold letters that it was, and I kept getting kernel error > messages about "calcru: negative time: ..." and such after doing it. > > Linux-SMP would give me at least a 1.5x speedup with a full kernel spinlock > and badly parallelized makefile. What's the deal here? Yes, the FreeBSD 'make -j' does nothing much. I (and so does phk) use the NetBSD make, which truely does run parallel jobs. However, the NetBSD make is no longer compatable with our Makefile tree since Jordan changed the obj dir stuff... it can only be used on the kernel now.... If you use that, and set COPTS="-O -pipe" in /etc/make.conf, and use somewhere around 'make -j3', you should see up to 180% or so of the speed of the single-job make on a SMP kernel. However, the single-job SMP kernel runs things slower than non-smp (probably because of the scheduling, which is partly worked around in the code that I have on my checked out copy of the source). 'make -j3' was between 130% and 150% of the speed of the uniprocessor, generic kernel though, even with our bogus scheduling. Cheers, -Peter From owner-freebsd-smp Thu Jul 4 19:43:42 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA03480 for smp-outgoing; Thu, 4 Jul 1996 19:43:42 -0700 (PDT) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id TAA03471 for ; Thu, 4 Jul 1996 19:43:35 -0700 (PDT) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.7.5/8.7.3) with ESMTP id KAA18450; Fri, 5 Jul 1996 10:42:24 +0800 (WST) Message-Id: <199607050242.KAA18450@spinner.DIALix.COM> X-Mailer: exmh version 1.6.6 3/24/96 To: Terry Lambert cc: erich@uruk.org, freebsd-smp@freebsd.org Subject: Re: Running SMP In-reply-to: Your message of "Thu, 04 Jul 1996 17:32:29 MST." <199607050032.RAA13959@phaeton.artisoft.com> Date: Fri, 05 Jul 1996 10:42:23 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >>>Terry Lambert said: > > It's still important to get the details on FreeBSD-SMP's model of how > > it uses the x86 data structures such as the TSS. Getting an accurate > > image of this is particularly important to how the > 2 CPU generalization > > goes (i.e. what static data is really necessary here... allocating > > something upwards of 64K per CPU statically at compile-time would be > > very annoying). I still haven't looked through that part yet, admittedly. > > > > Linux-SMP has one GDT with one TSS per process, simply guaranteeing that > > no more than one processor will access any particular TSS at a time. > > > > Could someone comment on how FreeBSD currently does this in more detail ? > > David? John? Pohl? We have one tss structure per process, it is stored at the beginning of the user area in a fixed virtual address. The GDT has a single slot pointing to the virtual address of the tss, and all processes use this same slot number in their task register. I want to change this so that we have a hybrid dynamic GDT (ie: some fixed entries, and an extentable component) and have the 'struct i386tss' in kernel space at a floating address (referenced by proc->p_addr rather than the fixed virtual address). This means doing a 'ltr' at task switch time, which is no big deal. This simply causes the process context to be saved in the new location, it's not hardware (slow) task switching. A dynamic GDT means we can *really* do VM86 properly, as well as support the dos emulator and the willows stuff properly. However, Bruce Evans is/was working in a different direction. He wants to remove the 'struct i386tss' from the process context and use a much smaller structure. The structure that holds the user process context at interrupt would be instead be in a static place and itself context switched on traps etc. If I understand what he's doing, this will make SMP support harder, and makes VM86, user_ldt etc stuff harder too, because some processes will need a full i386tss, while others will use the shared one etc, and as well, for SMP, you need a shared tss per cpu, and need to figure out which one you are using at trap/context switch time etc. I'm not sold on Bruce's idea yet, I've got my "wait and see" hat on, to see how much work it'll cause us on SMP. What I've implemented, is remove the per-process kernel stack and 'struct user' pages from the process address space, and cleaning up the code as a result. This will allow pure address space sharing between processes for the libc_r pthread support, since when one process changes it's VM space, it automatically appears in the others by virtue of the fact that they have the *same* page tables, vm_map and vmspace. This can only be completed once the user pages are completely gone. As for SMP, I've left things on the back burner for an uncomfortably long time now, I must get back to it. What I've written (but not polished/tested/debugged yet) is: - MP config table parsing - Safe boot straps of the non-booting cpus (ie: generate a PTD with the 0MB P==V mapping) - this PTD is used on a per-cpu basis for the idle loop (yes, it's back!) - the idle procs are gone. - As far as I can tell, it'll support N cpus cleanly. - a hack sysctl handler that takes string writes and parses them, theoretically it should allow fine-grained halt/start/on/offline etc of cpus, but for now it just understands "boot" - code to do message passing via IPI (eg: force reschedule of non-primary processors, this is suboptimal as there will be a thundering herd trying to enter the kernel. This would probably be better done with the timer in the local apic) - move most of the remaining initialisation code out of locore and into a seperate module. There is very little SMP activity at boot, and it should be possible to run almost (but not quite) identically as a non-smp kernel would. (curproc runtime etc would still be intercepted). The boot code does not do much other than locate the MP block and preserve it to try and prevent the VM system from trashing it if it's below 640K, and reserve some space for the trampoline code. I've thought of a few ways (and talked to phk about it some time ago) to do scheduling biasing to try and get processes on the same cpu where possible to get the benefit of the on-chip cache. I think this will be critical for P6 support as the 256K/512K L2 cache will generate a lot of MESI traffic for invalidations when we miss the cpu. Does anybody have documentation on the IO apic? I've got enough detail on the local apic, but the IO apic is a real problem. We're stuck in the painful "dumb" mode where all cpus get all interrupts in parallel with each other until we can get some details on the IO apic. (somebody pointed me to something from intel, but I can't find the reference anymore) > Terry Lambert > terry@lambert.org > --- > Any opinions in this posting are my own and not those of my present > or previous employers. Cheers, -Peter From owner-freebsd-smp Fri Jul 5 12:01:20 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id MAA26235 for smp-outgoing; Fri, 5 Jul 1996 12:01:20 -0700 (PDT) Received: from uruk.org (uruk.org [198.145.95.253]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id MAA26219 for ; Fri, 5 Jul 1996 12:01:16 -0700 (PDT) From: erich@uruk.org Received: from loopback (loopback [127.0.0.1]) by uruk.org (8.7.4/8.7.3) with SMTP id MAA13894; Fri, 5 Jul 1996 12:02:22 -0700 (PDT) Message-Id: <199607051902.MAA13894@uruk.org> X-Authentication-Warning: uruk.org: Host loopback [127.0.0.1] didn't use HELO protocol To: Peter Wemm cc: freebsd-smp@freebsd.org Subject: Re: Running SMP In-reply-to: Your message of "Fri, 05 Jul 1996 09:57:40 +0800." <199607050157.JAA18293@spinner.DIALix.COM> Date: Fri, 05 Jul 1996 12:02:16 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Peter Wemm writes: > Yes, the FreeBSD 'make -j' does nothing much. I (and so does phk) use the > NetBSD make, which truely does run parallel jobs. However, the NetBSD > make is no longer compatable with our Makefile tree since Jordan changed > the obj dir stuff... it can only be used on the kernel now.... > > If you use that, and set COPTS="-O -pipe" in /etc/make.conf, and use > somewhere around 'make -j3', you should see up to 180% or so of the speed > of the single-job make on a SMP kernel. > > However, the single-job SMP kernel runs things slower than non-smp > (probably because of the scheduling, which is partly worked around in the > code that I have on my checked out copy of the source). 'make -j3' was > between 130% and 150% of the speed of the uniprocessor, generic kernel > though, even with our bogus scheduling. Well, I don't have easy access to the NetBSD make, so since GNU make is used on so many things, I grabbed GCC and GNU make, then tried compiling GCC. SMP 1 CPU : gmake : 6:43 gmake -j 2 : 5:23 After I activated the second CPU, the system becomes fairly unstable. Each time I ran "gmake", I had at least one "Memory fault: core dump" for some program, and it would always crash the system by locking up hard when it tried to run GCC's "enquire" program. -- Erich Stefan Boleyn \_ E-mail (preferred): Mad Genius wanna-be, CyberMuffin \__ (finger me for other stats) Web: http://www.uruk.org/~erich/ Motto: "I'll live forever or die trying" This is my home system, so I'm speaking only for myself, not for Intel. From owner-freebsd-smp Fri Jul 5 12:06:38 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id MAA27005 for smp-outgoing; Fri, 5 Jul 1996 12:06:38 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id MAA26999 for ; Fri, 5 Jul 1996 12:06:35 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id MAA15216; Fri, 5 Jul 1996 12:02:49 -0700 From: Terry Lambert Message-Id: <199607051902.MAA15216@phaeton.artisoft.com> Subject: Re: Running SMP To: peter@spinner.dialix.com (Peter Wemm) Date: Fri, 5 Jul 1996 12:02:49 -0700 (MST) Cc: terry@lambert.org, erich@uruk.org, freebsd-smp@freebsd.org In-Reply-To: <199607050242.KAA18450@spinner.DIALix.COM> from "Peter Wemm" at Jul 5, 96 10:42:23 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > However, Bruce Evans is/was working in a different direction. He wants to > remove the 'struct i386tss' from the process context and use a much > smaller structure. The structure that holds the user process context at > interrupt would be instead be in a static place and itself context > switched on traps etc. If I understand what he's doing, this will make SMP > support harder, and makes VM86, user_ldt etc stuff harder too, because > some processes will need a full i386tss, while others will use the shared > one etc, and as well, for SMP, you need a shared tss per cpu, and need to > figure out which one you are using at trap/context switch time etc. I'm > not sold on Bruce's idea yet, I've got my "wait and see" hat on, to see > how much work it'll cause us on SMP. On the other hand, it will make it portable to non-TSS-using systems. I'm really torn on this, because I want real VM86 support. 8-(. > - Safe boot straps of the non-booting cpus (ie: generate a PTD with the > 0MB P==V mapping) > - this PTD is used on a per-cpu basis for the idle loop (yes, it's back!) With a real halt? > Does anybody have documentation on the IO apic? I've got enough detail on > the local apic, but the IO apic is a real problem. We're stuck in the > painful "dumb" mode where all cpus get all interrupts in parallel with > each other until we can get some details on the IO apic. (somebody > pointed me to something from intel, but I can't find the reference anymore) I thought I had pointed you to the www.intel.com page for the IO chip, and there's now a 1.2 MP document on their site, which goes into more detail on virtual wire (the mode I think we want to use to anonymize which procesor has to take which interrupt). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Fri Jul 5 12:09:37 1996 Return-Path: owner-smp Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id MAA27388 for smp-outgoing; Fri, 5 Jul 1996 12:09:37 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.5/8.7.3) with SMTP id MAA27377 for ; Fri, 5 Jul 1996 12:09:34 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id MAA15231; Fri, 5 Jul 1996 12:05:51 -0700 From: Terry Lambert Message-Id: <199607051905.MAA15231@phaeton.artisoft.com> Subject: Re: Running SMP To: peter@spinner.dialix.com (Peter Wemm) Date: Fri, 5 Jul 1996 12:05:50 -0700 (MST) Cc: erich@uruk.org, freebsd-smp@freebsd.org In-Reply-To: <199607050157.JAA18293@spinner.DIALix.COM> from "Peter Wemm" at Jul 5, 96 09:57:40 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > After hacking the hard-coded base address of the APIC, I tried running > > some kernel compiles to see what kind of performance I was getting. > > > > Frankly, it was pretty rotten. Here's what I got (multiprocessor > > 150MHz Pentium Pro w/256K caches): > > > > FreeBSD SMP 1 CPU : 4:36 > > -j 2 : 4:15 > > > > FreeBSD SMP 2 CPU : 4:28 > > -j 2 : 4:28 > > > > This was all compiling the SMP kernel with almost no devices. I almost > > couldn't tell the other CPU was activated, except that "sysctl" told > > me in big bold letters that it was, and I kept getting kernel error > > messages about "calcru: negative time: ..." and such after doing it. > > > > Linux-SMP would give me at least a 1.5x speedup with a full kernel spinlock > > and badly parallelized makefile. What's the deal here? > > Yes, the FreeBSD 'make -j' does nothing much. I (and so does phk) use the > NetBSD make, which truely does run parallel jobs. However, the NetBSD > make is no longer compatable with our Makefile tree since Jordan changed > the obj dir stuff... it can only be used on the kernel now.... > > If you use that, and set COPTS="-O -pipe" in /etc/make.conf, and use > somewhere around 'make -j3', you should see up to 180% or so of the speed > of the single-job make on a SMP kernel. > > However, the single-job SMP kernel runs things slower than non-smp > (probably because of the scheduling, which is partly worked around in the > code that I have on my checked out copy of the source). 'make -j3' was > between 130% and 150% of the speed of the uniprocessor, generic kernel > though, even with our bogus scheduling. Actually, I got my (better than UP) numbers by increasing the activation cycle for the kernel entrancy unlocking. Basically, I ran the parallel make in an xterm window, then ...uh, then set a textbook on the keypad enter key on the console keyboard. 8-) 8-) 8-). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers.