From owner-freebsd-virtualization@FreeBSD.ORG Mon Nov 15 07:26:54 2010 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39FD2106566B for ; Mon, 15 Nov 2010 07:26:54 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from out-0.mx.aerioconnect.net (outx.internet-mail-service.net [216.240.47.247]) by mx1.freebsd.org (Postfix) with ESMTP id 17B248FC13 for ; Mon, 15 Nov 2010 07:26:53 +0000 (UTC) Received: from idiom.com (postfix@mx0.idiom.com [216.240.32.160]) by out-0.mx.aerioconnect.net (8.13.8/8.13.8) with ESMTP id oAF7QqWT021876; Sun, 14 Nov 2010 23:26:52 -0800 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (h-67-100-89-137.snfccasy.static.covad.net [67.100.89.137]) by idiom.com (Postfix) with ESMTP id 4C6152D6013; Sun, 14 Nov 2010 23:26:51 -0800 (PST) Message-ID: <4CE0E0BD.1040707@freebsd.org> Date: Sun, 14 Nov 2010 23:26:53 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: <4CDEFC2D.4090908@freebsd.org> <4CDF0D87.70700@freebsd.org> <4CDF1F7E.8000601@freebsd.org> <20101114042427.V78896@maildrop.int.zabbadoz.net> In-Reply-To: <20101114042427.V78896@maildrop.int.zabbadoz.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 216.240.47.51 Cc: virtualization@freebsd.org, Jamie Gritton Subject: Re: limitations on jail style virtualization X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Nov 2010 07:26:54 -0000 On 11/13/10 8:27 PM, Bjoern A. Zeeb wrote: > On Sat, 13 Nov 2010, Julian Elischer wrote: > >> On 11/13/10 2:13 PM, Julian Elischer wrote: >>> On 11/13/10 1:55 PM, Brandon Gooch wrote: >>>> On Sat, Nov 13, 2010 at 2:59 PM, Julian >>>> Elischer wrote: >>>> >>>> Was this brought up in any of the discussions? >>>> >>>> http://www.7he.at/freebsd/vps/ >>> >>> no it was not brought up.. >>> it was an unofficial non-planned discussion that errupted pretty much >>> spontaneously an a couple of couches so it was not prepared in >>> advance.. >>> >>> Since I don't know much about that project I didn't bring it up. >>> However I should take the time to read it.. >>> thanks for bringing it up! >> >> replying to myself: >> >> Now that I have looked at the reference above I realise that I did >> look >> at it before but during a particularly hectic pereiod of my life so >> I didn't >> really take it in as much as I should have. >> >> I really do like this and I seen no reason why it can not be all >> linked together. >> Since I last heard about it, it appears that they have advanced >> somewhat. >> They are already integrating the VIMAGE code into their project, >> and they have >> extended the VIMAGE mechanism to suite their own purposes. >> >> It would be nice if they would show up in -virtualization sometimes >> so we could >> include them into our discussions. >> >>> >>>> I'm not sure if the VPS project pertains directly to what you're >>>> talking about, but perhaps some of the code or ideas from the >>>> project >>>> might? >>>> >>>> Even if it doesn't, it's still an exciting project that adds a >>>> ton of >>>> value to FreeBSD's light-weight virtualization strategy. What do >>>> think >>>> about the VPS concept in relation to the current virtualization >>>> effort >>>> being put in to jails? It seems to me that recent efforts at >>>> virtualizing kernel-level objects makes VPS the future of FreeBSD's >>>> virtualization, leaving jails as a great way to isolate >>>> applications... >>> >>> it the approaches are compatible I see no reason to not combine >>> but I'll know more when I have read it. >> >> the approaches are different but not necessarily incompatible >> In particular I'd like to hear what Jamie (James Gritton) >> thinks about it, as he has most of the jail direction in his head :-) >> >> I'd certainly like to be able do do many of the things they hope to >> do (or have a start at). > > we had Klaus at the DevSummit in Karlsruhe and he gave a talk at > EuroBSDCon as well. See the wiki and the 2010.eurobsdcon.org web > page (thought the material should be on his weg page as well). > > So, yes, we are talking to him, even though I got busy the last weeks. > As he's piggybacking on VNET/VIMAGE and there'll me more things he and > we'll do, there's certainly code going to be shared (I would hope). \ this sort of dovetails into something I've been thinking about for a while, which is NUMA support. Now, you may well ask "what has this to do with NUMA?" so I will explain. It seems to me that as the speed limits for single CPUS seem to be getting reached, we are seeing the transfer from speed to breadth. i.e more and more CPUs. We will shortly see the intruduction of systems that support not 2 or 4 threads of operation but thousands of threads of operation. We are already seeing some early implementations of this. A single image OS will have great trouble handling this sort of system, but the people who are hoping to make it all work are teh people like VMWARE. They will take systems with thousands of threads and break them up in a more-or less dynamic manner, keeping the NUMA layout of the system in mind. I have been wondering if we might not be able to do something similar. where we one one BSD system which overall may not be that efficient, but break it up along the NUMA fault lines using something like VPS to present a whole bunch of independent systems that can run very efficiently (on local NUMA RAM) an yet be 'tightly coupled' by the fact that they are running in one overall system with full cross connectivity (even though it be slightly less efficient than the NUMA couplings). We could, scale our systems up to tens of thousands of CPUS if we were willing to break them up to a large number of 'jails'/VPSs, each with only 32 or so threads (or whatever the hardware effieciently supports). Then the overall system could be responsible mostly for inter-system communications and housekeeping. Duplicated OS images and seaprated data ranges might all come together to form this "cluster of systems in a single OS". Anyhow, that the way I see things progressing and the OS that can harness all these CPUs in a useful way will have a lot if demand. whether it will be ESX+OS-de-jour or Linux++ or ClusterBSD will remain to be seen. Julian > > /bz >