From owner-freebsd-questions@FreeBSD.ORG Mon Jun 30 23:37:53 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E3FE1065672 for ; Mon, 30 Jun 2008 23:37:53 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 968478FC17; Mon, 30 Jun 2008 23:37:52 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <48696E4E.2040602@FreeBSD.org> Date: Tue, 01 Jul 2008 01:37:50 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: snott References: <18204938.post@talk.nabble.com> <486960E8.5010108@cwis.biz> <18205925.post@talk.nabble.com> In-Reply-To: <18205925.post@talk.nabble.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-questions@freebsd.org Subject: Re: Maximum swap size? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 23:37:53 -0000 snott wrote: > Trust me, I really do want 250GB (or more) of swap. I'm using swap as a > backing store for an HTTP reverse proxy for very large cache sets. Its more > efficient to just use the vm layer for LRU object management than to create > a huge mmap'd file with file buf caching. I am not certain but I think on 64-bit systems the 32GB limitation is per swap device (on i386 there is a 16GB total limitation because of a 32-bit counter of 512 byte blocks), so you can add multiple swap devices. However if you slice up a disk into many partitions you might lose performance because it will try to round robin between them, assuming they are independent (but they're not; you'll lose I/O throughput from seek delays). It may not be hard to change this behaviour. In general your strategy is a good one but there are other problems; managing that amount of swap will require a lot of auxiliary kernel memory. It is hard to estimate exactly how much for various reasons (it's not entirely deterministic), but in my environment even 20GB of swap requires increasing kern.maxswzone=209715200 i.e. about 200MB of memory just to keep track of the allocated swap. If you don't tune this then you'll run out of "swap zone" when you allocate beyond a certain point, and the kernel will deadlock. I think the default value allows about 8GB of swap use. Other kernel limits will prevent this from being raised above about 1500MB (although a forthcoming change in 8.0 will bring it up to 4GB). Basically, even though there are valid reasons to want to do what you're doing (and I do it myself on the build cluster that builds the FreeBSD packages), you're operating in a zone that would have been considered complete insanity until recently, and sufficiently few people have wanted to try that no-one has thought about optimizing in this regime. I think it would be quite an interesting project to try, though. Kris