From owner-freebsd-smp  Thu Dec  5 22:41:49 1996
Return-Path: <owner-smp>
Received: (from root@localhost)
          by freefall.freebsd.org (8.8.4/8.8.4) id WAA00797
          for smp-outgoing; Thu, 5 Dec 1996 22:41:49 -0800 (PST)
Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67])
          by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id WAA00788
          for <smp@freebsd.org>; Thu, 5 Dec 1996 22:41:41 -0800 (PST)
Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1])
          by spinner.DIALix.COM (8.8.4/8.8.4) with ESMTP id OAA13185;
          Fri, 6 Dec 1996 14:41:13 +0800 (WST)
Message-Id: <199612060641.OAA13185@spinner.DIALix.COM>
To: "J.M. Chuang" <smp@bluenose.na.tuns.ca>
cc: Tor.Egge@idt.ntnu.no (Tor Egge), smp@freebsd.org
Subject: Re: last major problem 
In-reply-to: Your message of "Fri, 06 Dec 1996 01:21:53 -0400."
             <199612060521.BAA24385@bluenose.na.tuns.ca> 
Date: Fri, 06 Dec 1996 14:41:12 +0800
From: Peter Wemm <peter@spinner.dialix.com>
Sender: owner-smp@freebsd.org
X-Loop: FreeBSD.org
Precedence: bulk

"J.M. Chuang" wrote:
> > 
> > I wrote:
> > >   2. I'll report back when I've tried the following patch:
> > 
> > [ patch deleted ]
> > 
> > I still get the same crash, with APIC_IO and SMP_INVLTLB defined :-(.
> > 
> I tried your patch, no luck for Titan pro as well!
> 
> BTW, I found that in the `make depend' (mkdep) for compiling a kernel,
> the system crashes.
> 
> Jim

BTW, we discovered that there are some differences between the P5 and P6
class cpu's in the prefetch behavior that have bitten us before.

If anybody has ever wondered what pmap_bootstrap2() was for, this is it.

Under the P5 cpu's, the prefetch queue was filled, but under the P6, the
prefetch queue or pipeline or whatever seems to be flushed on a load
of %cr3.

We were doing this sort of thing while running in low memory at 1MB:
  movl $_IdlePTD,%eax
  movl %eax,%cr3
  pushl $MPbegin
  ret
MPbegin:
  .. rest of boot code...

The problem was that %eip was low, and loading %cr3 caused it to be
running from unmapped memory.  On the P5, there were enough instructions
in the pipeline/prefetch queue to push the address and jump up to the
high memory address without faulting.  On the P6, the next instruction
after the load of %cr3 faults immediately.

I wonder if we're running into something like this?  Things seem to be
working with the P5 cpu's...  I must look over the crash reports again to
see if I can spot the reason why...

Cheers,
-Peter