From owner-freebsd-arch@FreeBSD.ORG Tue Jun 5 14:30:58 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A15D816A468; Tue, 5 Jun 2007 14:30:58 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id ECB3213C447; Tue, 5 Jun 2007 14:30:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l55EUsTY028249; Tue, 5 Jun 2007 10:30:55 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Tue, 5 Jun 2007 10:12:17 -0400 User-Agent: KMail/1.9.6 References: <20070604220649.E606@10.0.0.1> In-Reply-To: <20070604220649.E606@10.0.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200706051012.18864.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Tue, 05 Jun 2007 10:30:55 -0400 (EDT) X-Virus-Scanned: ClamAV 0.88.3/3360/Tue Jun 5 00:32:46 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx X-Mailman-Approved-At: Tue, 05 Jun 2007 14:33:00 +0000 Cc: kmacy@freebsd.org, benno@freebsd.org, marius@freebsd.org, marcl@freebsd.org, arch@freebsd.org, jake@freebsd.org, tmm@freebsd.org, cognet@freebsd.org, grehan@freebsd.org Subject: Re: New cpu_switch() and cpu_throw(). X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jun 2007 14:30:58 -0000 On Tuesday 05 June 2007 01:32:46 am Jeff Roberson wrote: > For every architecture we need to support a new features in cpu_switch() > and cpu_throw() before they can support per-cpu schedlock. I'll describe > those below. I'm soliciting help or advice in implementing these on > platforms other than x86, and amd64, especially on ia64 where things are > implemented in C! > > I checked in the new version of cpu_switch() for amd64 today after > threadlock went in. Basically, we have to release a thread's lock when > it's switched out and acquire a lock when it's switched in. > > The release must happen after we're totally done with the stack and > vmspace of the thread to be switched out. On amd64 this meant after we > clear the active bits for tlb shootdown. The release actually makes use > of a new 'mtx' argument to cpu_switch() and sets the td_lock pointer to > this argument rather than unlocking a real lock. td_lock has previously > been set to the blocked lock, which is always blocked. Threads > spinning in thread_lock() will notice the td_lock pointer change and > acquire the new lock. So this is simple, just a non-atomic store with a > pointer passed as an argument. On amd64: > > movq %rdx, TD_LOCK(%rdi) /* Release the old thread */ > > The acquire part is slightly more complicated and involves a little loop. > We don't actually have to spin trying to lock the thread. We just spin > until it's no longer set to the blocked lock. The switching thread > already owns the per-cpu scheduler lock for the current cpu. If we're > switching into a thread that is set to the blocked_lock another cpu is > about to set it to our current cpu's lock via the mtx argument mentioned > above. On amd64 we have: > > /* Wait for the new thread to become unblocked */ > movq $blocked_lock, %rdx > 1: > movq TD_LOCK(%rsi),%rcx > cmpq %rcx, %rdx > je 1b If this is to handle a thread migrating from one CPU to the next (and there's no interlock to control migration, otherwise you wouldn't have to spin here) then you will need memory barriers on the first write (i.e. the first write above should be an atomic_store_rel()) and the equivalent of an _acq barrier here. -- John Baldwin