From owner-freebsd-smp  Sat Dec 14 07:04:04 1996
Return-Path: <owner-smp>
Received: (from root@localhost)
          by freefall.freebsd.org (8.8.4/8.8.4) id HAA05521
          for smp-outgoing; Sat, 14 Dec 1996 07:04:04 -0800 (PST)
Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67])
          by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id HAA05483
          for <smp@freebsd.org>; Sat, 14 Dec 1996 07:03:59 -0800 (PST)
Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1])
          by spinner.DIALix.COM (8.8.4/8.8.4) with ESMTP id XAA17454;
          Sat, 14 Dec 1996 23:03:51 +0800 (WST)
Message-Id: <199612141503.XAA17454@spinner.DIALix.COM>
To: smp@freebsd.org
cc: Mike Haertel <haertel@ichips.intel.com>
Subject: Re: some questions concerning TLB shootdowns in FreeBSD 
In-reply-to: Your message of "Fri, 13 Dec 1996 23:05:04 PST."
             <199612140705.XAA00954@root.com> 
Date: Sat, 14 Dec 1996 23:03:51 +0800
From: Peter Wemm <peter@spinner.dialix.com>
Sender: owner-smp@freebsd.org
X-Loop: FreeBSD.org
Precedence: bulk

David Greenman wrote:
> >I'm curious how/when people are doing TLB shootdowns.
> >Obviously when reducing permission or unmapping pages.
> >How about for manipulations of the dirty/accessed bits?
> >(Does FreeBSD use these?)
> 
>    Speaking of the uni-processor case, FreeBSD does the access/modify bit
> changes in the pmap_changebit() function which does a TLB flush if anything
> is actually changed.
> 
> -DG
> 
> David Greenman
> Core-team/Principal Architect, The FreeBSD Project

Also, since I wrote the initial TLB flushing on top of Steve's IPI code,
I can freely admit that what I wrote is pretty sub-standard and does
not go far enough.

There are a couple of major shortfalls:
1: It's async.  it does not syncronise the remote processors as it must do,
or they can get out of sync, slave processors can do updates on stale
data, etc.
2: It does too much work.  There are a lot of cases where a global flush
is done for the local user process on the local cpu.   I am not 100% sure
whether this is needed or not.  I can imagine that APTD accesses might
present a problem if we try to avoid global flushes here.
3: We have no way of doing a local-only tlb flush with the trivial hack
that I did to test the theory.
4: the CADDR/APTD hacks are potential problems from many angles.  As long
as we only have one cpu in the kernel "proper" at present, this shouldn't
be too much of an issue yet - but it will be one of the things waiting to
bite us later on.

There was the query about the possibility of speculative execution
on the PPro being the problem that is breaking the kernel. The
scenario sounds plausable, but my initial reaction to that was that
we are doing this from an _interrupt handler_, and I would be very
suprised if speculative execution from the original code thread
isn't wound up before going into the interrupt...  If not, do we
need some strategic nop's?

I'm still digesting it,  I am almost worried that we might (shudder!)
be forced into doing an IPI to stop all the cpu's *before* the
current cpu changes the page tables, then letting them do the tlb
flush and letting them proceed.  If this actually is a real problem
this means a much bigger code impact.

Cheers,
-Peter