From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 19 21:36:51 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5876106566B for ; Wed, 19 Mar 2008 21:36:51 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (student.mired.org [66.92.153.77]) by mx1.freebsd.org (Postfix) with ESMTP id 83B8B8FC1D for ; Wed, 19 Mar 2008 21:36:51 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 27744 invoked by uid 1001); 19 Mar 2008 17:36:09 -0400 Received: from bhuda.mired.org (192.168.195.1) by bhuda.mired.org (tmda-ofmipd) with ESMTP; Wed, 19 Mar 2008 17:36:08 -0400 Date: Wed, 19 Mar 2008 17:36:08 -0400 To: Chuck Robey Message-ID: <20080319173608.558a3704@bhuda.mired.org> In-Reply-To: <47E16514.7090203@chuckr.org> References: <47DF1045.6050202@chuckr.org> <20080318082816.GA74218@eos.sc1.parodius.com> <47E146F9.5060105@chuckr.org> <20080319172213.GA28075@eos.sc1.parodius.com> <47E1558A.2030107@chuckr.org> <20080319184244.GA29838@eos.sc1.parodius.com> <47E16514.7090203@chuckr.org> Organization: Meyer Consulting X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.8; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: FreeBSD-Hackers , Jeremy Chadwick Subject: Re: remote operation or admin X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2008 21:36:52 -0000 On Wed, 19 Mar 2008 15:10:12 -0400 Chuck Robey wrote: > Not completely yet (I tend to be stubborn, if I carry this too far, tell me > in private mail and I will politely drop it). Your use cases show me the > differences in size, and *because* of the size, the differences in how > you'd use them, and that part I did already know. I'm perfectly well aware > of the current differences in size, but what I'm after is what are the real > differences, ignoring size, in what they actually accomplish, and how they > go about doing it. I'm thinking of the possibility of perhaps finding it > it might be possible to find some way to extend the work domain of an smp > system to stretch across machine lines, to jump across motherboards. Maybe > not to be global (huge latencies scare me away), but what about just going > 3 feet, on a very high speed bus, like maybe a private pci bus? Not what > is, what could be? What you're describing is a classic multi-cpu system. From the software point of view, it's just a another variant on the whole multiprocessor thing. I believe most modern multi-cpu systems share all the resources, and are all SMP. You could build one with memory that wasn't equally accessible to all CPUs in the system, but in general you don't want to do that unless the CPUs in question are doing different things, such as a GPU, an I/O processor of some kind, etc. So the answer to your two central questions "what are the real differences between what they actually accomplish and how they go about doing it" - at least for tightly coupled, everything shared multi-cpu boxes - is "there aren't any." There's been lots of work done in these areas, dating back to the 60s. Multics, Tandem and V are the ones that come to my mind, but there are others. There are schools of concurrent software design that hold that this should be true across the entire scale: that you design your system to work the same way on a single box with a quad-core CPU as you would on single-cpu boxes one each in Sao Paulo, Anchorage, Melbourne and Trondheim, as the differences are "implementation details". In real life, those difference are huge. In particular, if you're staying inside the same room, the critical one is going from "all memory shared" to "some memory shared", as the former means you can have a single memory location representing an external resource, and once you lock it you're set. With the latter, you can't do that - you have to bounce messages back and forth between the system, and get them all agree to let you have it before you work on it. This can easily push timing things out to as bad as the next level of sharing. For instance, I build ETL systems that shove thousands of multi-megabyte files/hour between boxes. It's actually faster to transfer the data over the network and let each box use an unshared file system for them than to rename the files on a shared disk from one directory to another, because doing the file rename requires locking both directories on every system that has access to them - which in this case means dozens of them. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information.