From owner-freebsd-performance@FreeBSD.ORG Wed Aug 22 17:09:37 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EAFCC1065672; Wed, 22 Aug 2012 17:09:36 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id A67358FC12; Wed, 22 Aug 2012 17:09:36 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so1823108pbb.13 for ; Wed, 22 Aug 2012 10:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=ufXi3+0ZwFFZwVuYK44MCK11pJBFX6i3KI87TPydGys=; b=V9rGT02wh7yQQ/FWffbv2bTMPUzYeH2vTphw/4W9ULhbacS5lbZx9r0+c7DyRWDJBo KO5NPQN0XsjUTjrGnwE2Fn7Q8PybHIvXAgrrm8W56OFqlRF1Di4XCdJDtO942be0gFx7 AeQnOAcLvcafchhUKBQ3Xtvcwe36GZXNfed3mCMwCtWLqVocMfuYpZMzk3K6TaPy490L bDczg6cxIXV2+cl5GjwudOlJhigYnMz4hUliumdzZdBcEMxYJTdxzOMEkDc51ltWQ4rh 9QVucuvoQOW9slCfhIzB6q2TjPL1wQ4YsQN85mMpeqO2ihUjJO9O6RWsaZ9HAWuLE0iu MDvA== Received: by 10.68.237.38 with SMTP id uz6mr49092368pbc.23.1345655376062; Wed, 22 Aug 2012 10:09:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.117.145 with HTTP; Wed, 22 Aug 2012 10:09:15 -0700 (PDT) In-Reply-To: <503418C0.5000901@rice.edu> References: <502DEAD9.6050304@zonov.org> <502EB081.3030801@rice.edu> <502FE98E.40807@rice.edu> <50325634.7090904@rice.edu> <503418C0.5000901@rice.edu> From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Date: Wed, 22 Aug 2012 10:09:15 -0700 Message-ID: To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov , kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 17:09:37 -0000 On Tue, Aug 21, 2012 at 4:24 PM, Alan Cox wrote: > On 8/20/2012 8:26 PM, Gezeala M. Bacu=F1o II wrote: >> >> On Mon, Aug 20, 2012 at 9:07 AM, Gezeala M. Bacu=F1o II >> wrote: >>> >>> On Mon, Aug 20, 2012 at 8:22 AM, Alan Cox wrote: >>>> >>>> On 08/18/2012 19:57, Gezeala M. Bacu=F1o II wrote: >>>>> >>>>> On Sat, Aug 18, 2012 at 12:14 PM, Alan Cox wrote: >>>>>> >>>>>> On 08/17/2012 17:08, Gezeala M. Bacu=F1o II wrote: >>>>>>> >>>>>>> On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox wrote: >>>>>>>> >>>>>>>> vm.kmem_size controls the maximum size of the kernel's heap, i.e., >>>>>>>> the >>>>>>>> region where the kernel's slab and malloc()-like memory allocators >>>>>>>> obtain >>>>>>>> their memory. While this heap may occupy the largest portion of t= he >>>>>>>> kernel's virtual address space, it cannot occupy the entirety of t= he >>>>>>>> address >>>>>>>> space. There are other things that must be given space within the >>>>>>>> kernel's >>>>>>>> address space, for example, the file system buffer map. >>>>>>>> >>>>>>>> ZFS does not, however, use the regular file system buffer cache. T= he >>>>>>>> ARC >>>>>>>> takes its place, and the ARC abuses the kernel's heap like nothing >>>>>>>> else. >>>>>>>> So, if you are running a machine that only makes trivial use of a >>>>>>>> non-ZFS >>>>>>>> file system, like you boot from UFS, but store all of your data in >>>>>>>> ZFS, >>>>>>>> then >>>>>>>> you can dramatically reduce the size of the buffer map via boot >>>>>>>> loader >>>>>>>> tuneables and proportionately increase vm.kmem_size. >>>>>>>> >>>>>>>> Any further increases in the kernel virtual address space size wil= l, >>>>>>>> however, require code changes. Small changes, but changes >>>>>>>> nonetheless. >>>>>>>> >>>>>>>> Alan >>>>>>>> >>> <> >>>>>> >>>>>> Your objective should be to reduce the value of "sysctl >>>>>> vfs.maxbufspace". >>>>>> You can do this by setting the loader.conf tuneable "kern.maxbcache" >>>>>> to >>>>>> the >>>>>> desired value. >>>>>> >>>>>> What does your machine currently report for "sysctl vfs.maxbufspace"= ? >>>>>> >>>>> Here you go: >>>>> vfs.maxbufspace: 54967025664 >>>>> kern.maxbcache: 0 >>>> >>>> >>>> Try setting kern.maxbcache to two billion and adding 50 billion to the >>>> setting of vm.kmem_size{,_max}. >>>> >> 2 : 50 =3D=3D>> is this the ratio for further tuning >> kern.maxbcache:vm.kmem_size? Is kern.maxbcache also in bytes? >> > > No, this is not a ratio. Yes, kern.maxbcache is in bytes. Basically, for > every byte that you subtract from vfs.maxbufspace, through setting > kern.maxbcache, you can add a byte to vm.kmem_size{,_max}. > > Alan > Great! Thanks. Are there other sysctls aside from vfs.bufspace that I should monitor for vfs.maxbufspace usage? I just want to make sure that vfs.maxbufspace is sufficient for our needs.