From owner-freebsd-arch@FreeBSD.ORG Wed Aug 10 22:21:07 2005 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3728516A41F; Wed, 10 Aug 2005 22:21:07 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id C898043D53; Wed, 10 Aug 2005 22:21:06 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (localhost.village.org [127.0.0.1] (may be forged)) by harmony.bsdimp.com (8.13.3/8.13.3) with ESMTP id j7AMK4Da076366; Wed, 10 Aug 2005 16:20:04 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 10 Aug 2005 16:20:06 -0600 (MDT) Message-Id: <20050810.162006.48492066.imp@bsdimp.com> To: jhb@freebsd.org From: "M. Warner Losh" In-Reply-To: <200508100911.50004.jhb@FreeBSD.org> References: <42F9ECF2.8080809@freebsd.org> <200508100911.50004.jhb@FreeBSD.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Wed, 10 Aug 2005 16:20:04 -0600 (MDT) Cc: andre@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Special schedulers, one CPU only kernel, one only userland X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2005 22:21:07 -0000 In message: <200508100911.50004.jhb@FreeBSD.org> John Baldwin writes: : First, you might want to go find a copy of "Unix Systems for Modern : Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers" : by Curt Schimmel as he describes this model as one of the early SMP models : for UNIX kernels before fine-grained locking was used. (It's known as a : master-slave system IIRC). I think this is the model that BSD/OS employed : for SMP in its 4.x series before they did their version of SMPng. Solbourne's OS/MP version 4.0[ABC] used this approach. It was called at the time 'Assymetric Multi Processing' or ASMP. All kernel traffic went to CPU0, while user jobs were scheduled for the additional CPUs. Since it was BSD 4.2 based kernel, it used spl locking as well as having a big kernel lock for CPU0. There was no other locking in the kernel than at the entry/exit to the kernel. OS/MP 4.1 and later used true SMP locking. 4.0D, iirc, used a modified version of ASMP locking that would allow only one CPU into the whole kernel at a time, but that CPU floated around. The engineer doing the work said it gave better performance for higher cpu work loads because CPU0 wouldn't have to be interrupted all the time. OS/MP 4.1 performed much better than OS/MP 4.0, and 4.0D better than 4.0C for what we would call today a 'world stone' test. Especially on the 12 CPU machines. You'd be trading locking overhead for latency if you push the ideas that you have too far. If only one CPU can be used for the kernel, then if the other CPU wants to use kernel services, it has to queue the request to cpu0 and then sit idle. Maybe this isn't a big deal for your workload, but for many workloads it results in lower performance. Warner