From owner-freebsd-mips@FreeBSD.ORG Fri Jan 29 05:57:37 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B326106566B for ; Fri, 29 Jan 2010 05:57:36 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-px0-f183.google.com (mail-px0-f183.google.com [209.85.216.183]) by mx1.freebsd.org (Postfix) with ESMTP id 5CA548FC17 for ; Fri, 29 Jan 2010 05:57:36 +0000 (UTC) Received: by pxi13 with SMTP id 13so1147928pxi.3 for ; Thu, 28 Jan 2010 21:57:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=h/YoQw8brMRw+qug79xUFmsfSbeAyQvjq4C/WZh2oDM=; b=h4cXTZZawo19zbdLvY225qS/pYbWBbwwJjhE+cfgFHB/376oIqj5SS2yC5m2xMbQXe 9w97eZF2vuwzc5I41+yRRNfH2m77Dot1JiccRUlpHXoE6PlrLgCTJy6zWHaHtM5OuEo3 CLqmp2Oe9wnUJg9f3p9q2p7zgdYjYjVqOY74o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=r38e+bdR8NC1i0zauqvb5w/X99iP4iecnAA5bdSKXcuLLBWUzyDtmxAQcX71z0zDb2 YLeoIuszxB9FFVsUN15v5mfnaHQKeduEuZSjwwoh6wrEknlIAn3WfgwE6ePNdHmJrXuw sQeadtiqG+Jd+iRbDD8CB6sgqRO0aALETi9v0= MIME-Version: 1.0 Received: by 10.141.15.9 with SMTP id s9mr244458rvi.221.1264743041603; Thu, 28 Jan 2010 21:30:41 -0800 (PST) In-Reply-To: References: <20100128.132114.1004138037722505681.imp@bsdimp.com> <66207A08-F691-4603-A6C5-9C675414C91E@lakerest.net> Date: Fri, 29 Jan 2010 11:00:41 +0530 Message-ID: <98a59be81001282130n1776b31bn3f6995b6ef136ff0@mail.gmail.com> From: "C. Jayachandran" To: Juli Mallett Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org, Neel Natu Subject: Re: Code review: groundwork for SMP X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2010 05:57:37 -0000 I'm new to this list, joined yesterday.=C2=A0 I work at RMI (now Netlogic), and did part of our internal port of FreeBSD 6.4 to XLR/XLS processors. > So on your systems threads share the TLB? =C2=A0Wired TLB entries can't b= e > pulled out (in the case of the kernel stack it's basically > catastrophic for that to happen.) =C2=A0A compromise if your TLB entries > are really at a premium is to use a single large entry (using, say, a > single 32k page) that contains both PCPU and the kernel stack, or a > page which has pointers to pcpu data, the kernel stack, etc. =C2=A0I seem > to recall seeing a port of FreeBSD that used the same storage for the > kernel stack and PCPU data, but I could be mistaken. Our cpus can be configured in a way that they share the 64 TLB entries among the 4 threads in the core. You could also configure the threads so that they have 16 independent entires each. But 16 is too less for running FreeBSD, so by default we used the shared TLB mode. > There are other trade-offs available, of course. =C2=A0If we don't use th= e > gp for accessing small data, we can keep a pointer to the pcpu data of > a CPU in gp whenever the kernel is running, and then PCPU accesses are > just a madder of loading from offset+gp, which is very quick =E2=80=94 fa= ster > than the wired TLB entry mechanism, unless you use a virtual address > for the pcpu in which case it can be painful. =C2=A0As there are more > things like VIMAGE, the amount of small global data in the kernel is > going to fall and making gp a pcpu pointer makes more sense. =C2=A0My old > port used -G0 and I still disable use of the gp in my non-FreeBSD MIPS > work =E2=80=94 I think NetBSD used to but I haven't noticed what FreeBSD = does. Again on XLR processors, there are per-thread scratch registers in COP0. So our preferred way of doing this was to have the per-cpu pointer in one of these scratch registers. We can also get the TLB out of the way for some of these by reserving KSEG0 region on startup for these and for stack. I agree with Randall here, the preferred way is to avoid wiring the TLB entries. Can't we reserve some area for this at start-up and keep the pointer in a platform-specific macro? > More curiosity than anything (since I don't seem to be able to get an > RMI system to develop on): if the threads are sharing the TLB, how are > updates to TLB-related fields synchronized? =C2=A0How do you atomically > increase the wired count of the TLB? =C2=A0How does 'tlbwr' work? =C2=A0D= o you > have to use special instructions when you're sharing the TLB that are > XLR-specific? Each thread has its own COP0 registers, but they update the core's TLB, so there are no special TLB instructions. Regards, JC. -- C. Jayachandran =C2=A0 =C2=A0c.jayachandran@gmail.com