From owner-svn-src-all@FreeBSD.ORG Sat Nov 10 19:18:19 2012 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ED252C26 for ; Sat, 10 Nov 2012 19:18:18 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 519398FC16 for ; Sat, 10 Nov 2012 19:18:17 +0000 (UTC) Received: (qmail 79430 invoked from network); 10 Nov 2012 20:52:52 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 10 Nov 2012 20:52:52 -0000 Message-ID: <509EA869.6030407@freebsd.org> Date: Sat, 10 Nov 2012 20:18:01 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Peter Wemm Subject: Re: svn commit: r242847 - in head/sys: i386/include kern References: <201211100208.qAA28e0v004842@svn.freebsd.org> <509DC25E.5030306@mu.org> <509E3162.5020702@FreeBSD.org> <509E7E7C.9000104@mu.org> <509E830D.5080006@mu.org> <509E847E.30509@mu.org> <509E8930.50800@mu.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: src-committers@freebsd.org, Eitan Adler , Alfred Perlstein , svn-src-all@freebsd.org, Alfred Perlstein , svn-src-head@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2012 19:18:19 -0000 On 10.11.2012 19:04, Peter Wemm wrote: > On Sat, Nov 10, 2012 at 9:48 AM, Eitan Adler wrote: >> On 10 November 2012 12:45, Peter Wemm wrote: >>> On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler wrote: >>>> On 10 November 2012 12:04, Alfred Perlstein wrote: >>>>> Sure, if you'd like you can help me craft that comment now? >>>> >>>> I think this is short and clear: >>>> === >>>> Limit the amount of kernel address space used to a fixed cap. >>>> 384 is an arbitrarily chosen value that leaves 270 MB of KVA available >>>> of the 2 MB total. On systems with large amount of memory reduce the >>>> the slope of the function in order to avoiding exhausting KVA. >>>> === >>> >>> That's actually completely 100% incorrect... >> >> okay. I'm going by the log messages posted so far. I have no idea how >> this works. Can you explain it better? > > That's exactly my point.. > > You get 1 maxuser per 2MB of physical ram. > If you get more than 384 maxusers (ie: 192GB of ram) we scale it > differently for the part past 192GB. I have no idea how the hell to Rather past 768MB of RAM. > calculate that. > You get an unlimited number of regular mbufs. > You get 64 clusters per maxuser (128k) > Unless I fubared the numbers, this currently works out to be 6%, or 1/16. > > Each MD backend gets to provide a cap for maxusers, which is in units > of 2MB. For an i386 PAE machine you have a finite amount of KVA space > (1GB, but this is adjustable.. you can easily configure it for 3GB kva > with one compile option for the kernel). The backends where the > nmbclusters comes out of KVA should calculate the number of 2MB units > to avoid running out of KVA. > > amd64 does a mixture of direct map and kva allocations. eg: mbufs and > clusters come from direct map, the jumbo clusters come from kva. > > So side effects of nmbclusters for amd64 are more complicated. > > 1/2 of the nmbclusters (which are in physcal ram) are allocated as > jumbo frames (kva) > 1/4 of nmbclusters (physical) are 9k jumbo frames (kva) > 1/8 of nmbclusters (physical) are used to set the 16k kva backed jumbo > frame pool. The mbufs and clusters of different types are not allocated at startup time, but rather their total allocation at runtime is *limited* to that maximal value in UMA. > amd64 kva is "large enough" now, but my recollection is that sparc64 > has a small kva plus a large direct map. Tuning for amd64 isn't > relevant for sparc64. mips has direct map, but doesn't have a "large" > direct map, nor a "large" kva. > > This is complicated but we need a simple user visible view of it. It > really needs to be something like "nmbclusters defaults to 6% of > physical ram, with machine dependent limits". The MD limits are bad > enough, and using bogo-units like "maxusers" just makes it worse. Yes, that would be optimal. -- Andre