From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 19 02:16:17 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2D44616A4CE for ; Fri, 19 Dec 2003 02:16:17 -0800 (PST) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9441E43D1F for ; Fri, 19 Dec 2003 02:16:15 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfmqn.dialup.mindspring.com ([165.247.219.87] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 1AXHg2-0006p7-00; Fri, 19 Dec 2003 02:16:11 -0800 Message-ID: <3FE2CFE3.EAA9ADBC@mindspring.com> Date: Fri, 19 Dec 2003 02:16:03 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: andykinney@advantagecom.net References: <3FE1936A.10308.75BF6ECA@localhost> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a467bd87b653f727e8c79783687b1830f4666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: hackers@freebsd.org cc: Julian Elischer Subject: Re: Machines with >= 4GB of RAM X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2003 10:16:17 -0000 Andrew Kinney wrote: > On 17 Dec 2003 at 15:44, Julian Elischer wrote: > > > options KVA_PAGES=512 > > > > may be a start, but is it still required, and do I have to change > > anything else to match it? (where does the Makefile work out where to > > link the kernel for?) > > Is a value of 512 enough for a machine with 16GB of RAM? > > > > Any hints, (even a better google search string) appreciated. > > We have a 4GB machine running 4.8-RELEASE, so we aren't using PAE, > but we had to make changes similar to what you're asking about for a > different reason. > > Your requirements will vary depending on the version of FreeBSD > you're running, but in general, increasing KVA_PAGES will help > considerably with stability on large memory machines. It should be > noted that releases prior to 4.8 required more changes than just > KVA_PAGES, but the documentation is a bit muddied on that subject. In general, the kmem_map size and other kernel memory usage, including page tables necessary to reference the full memory, end up taking more than 1G, so the 3G user:1G kernel ratio that's the default for older FreeBSD won't work at all. I usually recommend that people make it 1G user:3G kernel; you can get away with 2G:2G if you aren't going to be allocating lots of mbufs or supporting lots of open sockets, etc., but in general most people throw 4G+ into a box because they plan on building a network server and then throwing some serious load on it. > I don't know if it is required, but we rebuilt the world after > changing KVA_PAGES just to make sure that any hidden dependencies on > that value were handled in things other than the kernel. Depends. The normal case where this will be required is for prebuilt kernel modules. The only user space code I'm aware of which cares is the Linux threads package (and anything that links against it), since the threads mailboxes are in a fixed location apriori known to both the kernel module and the threads library, and the location has to be changed when the KVA space changes, since it assumes a 3:1 or whatever was in effect when it was compiled. > As far as 512 being a large enough setting for a 16GB machine, that > depends entirely on what you plan to do with the machine and its > usage pattern of various system resources. In my personal experience, th kernel and data structures consume over 1G in a 4G box, du to the auto-tuning cruft trying to b smarter than it actually is, and making bad decisions. In the 4.7/4.8 time frame, this was catastrophic with more than 4G, since it didn't stop scaling at 4G (the kernel can only address 4G, without PAE, no matter what, since pointers are 32 bits). Scaling above that point tris to allocate more memory for the kernel than the kernel is capable of addressing. > For instance, on our 4GB machine, it does a lot of heavy web serving, > databases, and email. We needed the 2GB KVA on that machine because > of large numbers of files, large network buffers, and some weirdness > relating to Apache and pv entries. If your usage patterns were > similar and you wanted to make full use of the 16GB without getting > trap 12 panics, then 2GB KVA may be inadequate. Older boxes won't even boot with 3:1 if you jam 4G in them, priod. -- Terry