From owner-freebsd-arch@FreeBSD.ORG  Sat Jun  9 08:36:58 2007
Return-Path: <owner-freebsd-arch@FreeBSD.ORG>
X-Original-To: arch@freebsd.org
Delivered-To: freebsd-arch@FreeBSD.ORG
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 5B0FA16A400;
	Sat,  9 Jun 2007 08:36:58 +0000 (UTC)
	(envelope-from dillon@apollo.backplane.com)
Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2])
	by mx1.freebsd.org (Postfix) with ESMTP id 358BF13C447;
	Sat,  9 Jun 2007 08:36:58 +0000 (UTC)
	(envelope-from dillon@apollo.backplane.com)
Received: from apollo.backplane.com (localhost [127.0.0.1])
	by apollo.backplane.com (8.13.8/8.13.7) with ESMTP id l598Pv2q020258;
	Sat, 9 Jun 2007 01:25:57 -0700 (PDT)
Received: (from dillon@localhost)
	by apollo.backplane.com (8.13.8/8.13.4/Submit) id l598Pval020257;
	Sat, 9 Jun 2007 01:25:57 -0700 (PDT)
Date: Sat, 9 Jun 2007 01:25:57 -0700 (PDT)
From: Matthew Dillon <dillon@apollo.backplane.com>
Message-Id: <200706090825.l598Pval020257@apollo.backplane.com>
To: Jeff Roberson <jroberson@chesapeake.net>
References: <20070604220649.E606@10.0.0.1>
X-Mailman-Approved-At: Sat, 09 Jun 2007 11:23:15 +0000
Cc: kmacy@freebsd.org, benno@freebsd.org, marius@freebsd.org, marcl@freebsd.org,
	arch@freebsd.org, jake@freebsd.org, tmm@freebsd.org,
	cognet@freebsd.org, grehan@freebsd.org
Subject: Re: New cpu_switch() and cpu_throw().
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
	<mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
	<mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 09 Jun 2007 08:36:58 -0000

     I haven't read your code to see what you've done exactly but I
     had a similar issue in DragonFly and I solved it by having a bit
     of code that set a bit in the previous thread's flags AFTER having
     switched to the new thread.

     e.g. the switch restore code for the new thread (%eax) is also
     responsible for cleaning up the old thread (%ebx):

        /*
         * Clear TDF_RUNNING flag in old thread only after cleaning up
         * %cr3.  The target thread is already protected by being TDF_RUNQ
         * so setting TDF_RUNNING isn't as big a deal.
         */
        andl    $~TDF_RUNNING,TD_FLAGS(%ebx)
        orl     $TDF_RUNNING,TD_FLAGS(%eax)

	(from /usr/src/sys/platform/pc32/i386/swtch.s on DragonFly)

    The exit code can then interlock on TDF_RUNNING without there being
    a race against the old thread's stack.  The condition occured so rarely
    I didn't even bother using a lock... I just test it in exit (well,
    really the reaper) and sleep for one tick in a loop if the race was
    detected.  The same feature is used for thread migration between cpu's...
    the target cpu gets an IPI message with the thread being migrated, and
    spins waiting for TDF_RUNNING to clear, indicating that the thread has
    been completely switched out by the originating cpu.

    FreeBSD uses a more complex arrangement so it might not be applicable,
    but it is still a nice trick.

						-Matt