From owner-freebsd-arch@FreeBSD.ORG Mon Jul 26 18:19:28 2010 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D6F61065676 for ; Mon, 26 Jul 2010 18:19:28 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E1DB98FC1A for ; Mon, 26 Jul 2010 18:19:27 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id VAA28708; Mon, 26 Jul 2010 21:19:25 +0300 (EEST) (envelope-from avg@freebsd.org) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1OdSGr-000H3b-3K; Mon, 26 Jul 2010 21:19:25 +0300 Message-ID: <4C4DD1AA.3050906@freebsd.org> Date: Mon, 26 Jul 2010 21:19:22 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.24 (X11/20100603) MIME-Version: 1.0 To: Matthew Fleming , freebsd-arch@freebsd.org References: <4C4DB2B8.9080404@freebsd.org> In-Reply-To: X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: amd64: change VM_KMEM_SIZE_SCALE to 1? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2010 18:19:28 -0000 on 26/07/2010 20:04 Matthew Fleming said the following: > On Mon, Jul 26, 2010 at 9:07 AM, Andriy Gapon wrote: >> Anyone knows any reason why VM_KMEM_SIZE_SCALE on amd64 should not be set to 1? >> I mean things potentially breaking, or some unpleasant surprise for an >> administrator/user... > > As I understand it, it's merely a resource usage issue. amd64 needs > page table entries for the expected virtual address space, so allowing > more than e.g. 1/3 of physical memory means needing more PTEs. But > the memory overhead isn't all that large IIRC: each 4k physical memory > devoted to PTEs maps 512 4k virtual addresses, or 2MB, so e.g. it > takes about 4MB reserved as PTE pages to map 2GB of kernel virtual > address space. My understanding is that paging entries are only allocated when actual (physical) memory allocation is done. But I am not sure. > Having cut my OS teeth on AIX/PowerPC where virutal address space is > free and has no relation to the size of the hardware page table, the > FreeBSD architecture limiting the size of the kernel virtual space > seemed weird to me. However, since FreeBSD also does not page kernel > data to disk, there's a good reason to limit the size of the kernel's > virtual space, since that also limits the kernel's physical space. > > In other words, setting it to 1 could lead to the system being out of > memory but not trying to fail kernel malloc requests. I'm not > entirely sure this is a new problem since one could also chew through > physical memory with sub-page uma allocations as well on amd64. Well, personally I would prefer kernel eating a lot of memory over getting "kmem_map too small" panic. Unexpectedly large memory usage by kernel can be detected and diagnosed, and then proper limits and (auto-)tuning could be put in place. Panic at some random allocation is not that helpful. Besides, presently there are more and more workloads that require a lot of kernel memory - e.g. ZFS is gaining popularity. Hence, the question/suggestion. Of course, the things can be tuned by hand, but I think that VM_KMEM_SIZE_SCALE=1 would be a more reasonable default than current value. -- Andriy Gapon