From owner-freebsd-questions@FreeBSD.ORG Fri Oct 14 17:24:39 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D0C4106566B for ; Fri, 14 Oct 2011 17:24:39 +0000 (UTC) (envelope-from tundra@tundraware.com) Received: from ozzie.tundraware.com (ozzie.tundraware.com [75.145.138.73]) by mx1.freebsd.org (Postfix) with ESMTP id 4A3E08FC16 for ; Fri, 14 Oct 2011 17:24:38 +0000 (UTC) Received: from [192.168.0.2] (viper.tundraware.com [192.168.0.2]) (authenticated bits=0) by ozzie.tundraware.com (8.14.5/8.14.5) with ESMTP id p9EHOWoi035025 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Fri, 14 Oct 2011 12:24:32 -0500 (CDT) (envelope-from tundra@tundraware.com) Message-ID: <4E98707F.3070202@tundraware.com> Date: Fri, 14 Oct 2011 12:25:19 -0500 From: Tim Daneliuk User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 CC: freebsd-questions@freebsd.org References: <4E9866CF.6010209@gmx.com> In-Reply-To: <4E9866CF.6010209@gmx.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (ozzie.tundraware.com [75.145.138.73]); Fri, 14 Oct 2011 12:24:32 -0500 (CDT) X-TundraWare-MailScanner-Information: Please contact the ISP for more information X-TundraWare-MailScanner-ID: p9EHOWoi035025 X-TundraWare-MailScanner: Found to be clean X-TundraWare-MailScanner-From: tundra@tundraware.com X-Spam-Status: No Subject: Re: Very large swap X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2011 17:24:39 -0000 On 10/14/2011 11:43 AM, Nikos Vassiliadis wrote: > On 10/14/2011 8:08 AM, Dennis Glatting wrote: >> >> This is kind of stupid question but at a minimum I thought it would be >> interesting to know. >> >> What is the limitations in terms of swap devices under RELENG_8 (or 9)? >> >> A single swap dev appears to be limited to 32GB (there are truncation >> messages on boot). I am looking at a possible need of 2-20TB (probably >> more) with as much main memory that is affordable. > > The limit is raised to 256GB in HEAD and RELENG_8 > http://svnweb.freebsd.org/base?view=revision&revision=225076 > >> I am working with large data sets and there are various ways of solving >> the problem sets but simply letting the processors swap as they work >> through a given problem is a possible technique. > > I would advise against this technique. Possibly, it's easier to design > your program to user smaller amounts of memory and avoid swapping. > > After all, designing your program to use big amounts of swapped out > memory *and* perform in a timely manner, can be very challenging. > > Nikos Well ... I dunno how much large dataset processing you've done, but it's not that simple. Ordinarily, with modern machines and architectures, you're right. In fact, you NEVER want to swap, instead, throw memory at the problem. But when you get into really big datasets, it's a different story. You probably will not find a mobo with 20TB memory capacity :) So ... you have to do something with disk. You generally get two choices: Memory mapped files or swap. It's been some years since I considered either seriously, but they do have some tradeoffs. MM files give the programmer very fine grained control of just what might get pushed out to disk at the cost of user space context switching. Swap gets managed by the kernel which is about as efficient as disk I/O is going to get, but that means what and how things get moved on- and off disk is invisible to the application. What a lot of big data shops are moving to is SSD for such operations. SSD is VERY fast and can be RAIDed to overcome the tendency of at least the early SSD products' tendency to, um ... blow up. As always, scale is hard, and giant data problems are Really Hard (tm). That's why people like IBM, Sun/Oracle, and Teradata make lots of money building giant iron farms. 'Just my 2^1 cents worth ... -- ---------------------------------------------------------------------------- Tim Daneliuk tundra@tundraware.com PGP Key: http://www.tundraware.com/PGP/