From owner-cvs-src@FreeBSD.ORG Sun Aug 7 09:02:49 2005 Return-Path: X-Original-To: cvs-src@FreeBSD.org Delivered-To: cvs-src@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8306216A422; Sun, 7 Aug 2005 09:02:49 +0000 (GMT) (envelope-from dfr@nlsystems.com) Received: from itchy.rabson.org (mailgate.nlsystems.com [80.177.232.242]) by mx1.FreeBSD.org (Postfix) with ESMTP id 081E94429F; Sun, 7 Aug 2005 08:41:41 +0000 (GMT) (envelope-from dfr@nlsystems.com) Received: from herring.rabson.org (herring [10.0.0.2]) by itchy.rabson.org (8.13.3/8.13.3) with ESMTP id j778fZnY095368; Sun, 7 Aug 2005 09:41:35 +0100 (BST) (envelope-from dfr@nlsystems.com) From: Doug Rabson To: Marcel Moolenaar Date: Sun, 7 Aug 2005 09:41:32 +0100 User-Agent: KMail/1.8 References: <200508062028.j76KSJtM019032@repoman.freebsd.org> In-Reply-To: <200508062028.j76KSJtM019032@repoman.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200508070941.33821.dfr@nlsystems.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (itchy.rabson.org [80.177.232.242]); Sun, 07 Aug 2005 09:41:35 +0100 (BST) X-Spam-Status: No, score=-2.8 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on itchy.rabson.org X-Virus-Scanned: ClamAV 0.83/1006/Sat Aug 6 21:02:09 2005 on itchy.rabson.org X-Virus-Status: Clean Cc: cvs-src@FreeBSD.org, src-committers@FreeBSD.org, cvs-all@FreeBSD.org Subject: Re: cvs commit: src/sys/ia64/ia64 exception.S interrupt.c machdep.c mp_machdep.c pmap.c trap.c vm_machdep.c src/sys/ia64/include proc.h smp.h X-BeenThere: cvs-src@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CVS commit messages for the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2005 09:02:50 -0000 Excellent! When trying to think about per-cpu VHPT in the past, I could never quite see how to handle the collision chains sanely. The solution described below seems ideal. On Saturday 06 August 2005 21:28, Marcel Moolenaar wrote: > marcel 2005-08-06 20:28:19 UTC > > FreeBSD src repository > > Modified files: > sys/ia64/ia64 exception.S interrupt.c machdep.c > mp_machdep.c pmap.c trap.c vm_machdep.c > sys/ia64/include proc.h smp.h > Log: > Improve SMP support: > o Allocate a VHPT per CPU. The VHPT is a hash table that the CPU > uses to look up translations it can't find in the TLB. As such, > the VHPT serves as a level 1 cache (the TLB being a level 0 > cache) and best results are obtained when it's not shared between > CPUs. The collision chain (i.e. the hash bucket) is shared between > CPUs, as all buckets together constitute our collection of PTEs. To > achieve this, the collision chain does not point to the first PTE in > the list anymore, but to a hash bucket head structure. The head > structure contains the pointer to the first PTE in the list, as well > as a mutex to lock the bucket. Thus, each bucket is locked > independently of each other. With at least 1024 buckets in the VHPT, > this provides for sufficiently finei-grained locking to make the > ssolution scalable to large SMP machines. > o Add synchronisation to the lazy FP context switching. We do this > with a seperate per-thread lock. On SMP machines the lazy high > FP context switching without synchronisation caused inconsistent > state, which resulted in a panic. Since the use of the high FP > registers is not common, it's possible that races exist. The ia64 > package build has proven to be a good stress test, so this will get > plenty of exercise in the near future. > o Don't use the local ID of the processor we want to send the IPI > to as the argument to ipi_send(). use the struct pcpu pointer > instead. The reason for this is that IPI delivery is unreliable. It > has been observed that sending an IPI to a CPU causes it to receive a > stray external interrupt. As such, we need a way to make the delivery > reliable. The intended solution is to queue requests in the target > CPU's per-CPU structure and use a single IPI to inform the CPU that > there's a new entry in the queue. If that IPI gets lost, the CPU can > check it's queue at any convenient time (such as for each clock > interrupt). This also allows us to send requests to a CPU without > interrupting it, if such would be beneficial. > > With these changes SMP is almost working. There are still some > random process crashes and the machine can hang due to having the IPI > lost that deals with the high FP context switch. > > The overhead of introducing the hash bucket head structure results > in a performance degradation of about 1% for UP (extra pointer > indirection). This is surprisingly small and is offset by gaining > reasonably/good scalable SMP support. > > Revision Changes Path > 1.57 +8 -0 src/sys/ia64/ia64/exception.S > 1.50 +5 -0 src/sys/ia64/ia64/interrupt.c > 1.201 +30 -13 src/sys/ia64/ia64/machdep.c > 1.56 +29 -25 src/sys/ia64/ia64/mp_machdep.c > 1.161 +227 -272 src/sys/ia64/ia64/pmap.c > 1.114 +12 -7 src/sys/ia64/ia64/trap.c > 1.91 +1 -0 src/sys/ia64/ia64/vm_machdep.c > 1.15 +2 -1 src/sys/ia64/include/proc.h > 1.10 +4 -2 src/sys/ia64/include/smp.h