From owner-freebsd-current@FreeBSD.ORG Wed Oct 10 21:31:31 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E26E116A418 for ; Wed, 10 Oct 2007 21:31:31 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail6.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 78A5513C48D for ; Wed, 10 Oct 2007 21:31:31 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8p) with ESMTP id 213856274-1834499 for multiple; Wed, 10 Oct 2007 17:29:30 -0400 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l9ALVLAf017700; Wed, 10 Oct 2007 17:31:24 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-current@freebsd.org Date: Wed, 10 Oct 2007 16:53:53 -0400 User-Agent: KMail/1.9.6 References: <20070928184659.GB57646@cicely12.cicely.de> <46FDD3AB.3050305@bitfreak.org> In-Reply-To: <46FDD3AB.3050305@bitfreak.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200710101653.54019.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Wed, 10 Oct 2007 17:31:25 -0400 (EDT) X-Virus-Scanned: ClamAV 0.88.3/4523/Wed Oct 10 14:30:26 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Darren Pilgrim Subject: Re: Non-sequential AP starts [Was: Re: 8-core server] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Oct 2007 21:31:32 -0000 On Saturday 29 September 2007 12:25:15 am Darren Pilgrim wrote: > Bernd Walter wrote: > > On Fri, Sep 28, 2007 at 07:56:49AM -0700, Darren Pilgrim wrote: > >> Jeremy Chadwick wrote: > >>> I think what you're not noticing is that the cores are being launched in > >>> non-sequential order. 1, 2, 3, 4, 7, 5, 6. This isn't a problem. > >> Why they wouldn't be launched sequentially? > > > > First SMP rule: > > Don't expect a specific execution order from things running parallel. > > > > I don't know the code, but would assume that they are started > > sequentially and each core prints it's own line, so they get > > disordered. > > This question bugged me long enough for me to go read some source to > find the answer, but I ended up with more questions. I guess that's > what I get for reading boot code. :) I know it's basically just a race > condition, but where does the race actually occur? In each core's > execution of startup instructions leading up to the printf() in > init_secondary() or in the assignment of the value returned by > PCPU_GET(cpudid)? Where is the data behind PCPU_GET() anyway? I > couldn't find it. Is AP launch order random or are there CPU > characteristics that result in reproducible non-sequential ordering? The lauch is random. The boot processor sets a flag to tell the APs (which are already running, but spinnin in a loop) "ok, go run". The APs then compete on a spin lock to serialize their final initialization, and they "launch" in the order they acquire the spin lock, which is quite indeterminate. -- John Baldwin