From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 06:55:25 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5B60D106566B for ; Fri, 17 Aug 2012 06:55:25 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id C266B8FC0A for ; Fri, 17 Aug 2012 06:55:24 +0000 (UTC) Received: by lbbgk8 with SMTP id gk8so2325906lbb.13 for ; Thu, 16 Aug 2012 23:55:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=gjbDKCC8RbEX8cCUxm3t+S9f5YmlqorlDrsohlXwqUk=; b=WPcuCEhnsoWAggK4WJxjYI8TTtmQoVta59gLHOnfIjfP6UVOQMq4zJOkd7chYRC4sO C2XRUmvmpSew9XHPmD2aKgadcburgoYy36D0ytRVgfqFmUfzMnBVhjASAfvMJ87fEYKf omc+NHX4hlq0NWvmLtALYI7WYh0+pUW0qxCH2kMQwGVIqPA6cBmAb0oMMTO822KAavDs Z89+ItIJ985R/R8lAHs9B3vmIOQQMBcMWNoEKyarmFV3dkxFqF/AXS6pItR/LF8UEhef gOjEM3Sw3g5iRE22pC8j1qOSlzxsCXyrbQlrUC3G8MEE26YVIbAp6ktIZd+CpBTKM9XC TwVw== Received: by 10.112.103.68 with SMTP id fu4mr1805956lbb.56.1345186523367; Thu, 16 Aug 2012 23:55:23 -0700 (PDT) Received: from zont-osx.local (ppp95-165-138-47.pppoe.spdop.ru. [95.165.138.47]) by mx.google.com with ESMTPS id h8sm1510097lbi.13.2012.08.16.23.55.22 (version=SSLv3 cipher=OTHER); Thu, 16 Aug 2012 23:55:22 -0700 (PDT) Message-ID: <502DEAD9.6050304@zonov.org> Date: Fri, 17 Aug 2012 10:55:21 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Marie Bacuno II References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Gm-Message-State: ALoCoQk86KJKZT3emDy3DjAqabnVSeADSO3e77HViiX25VqPEO14RbKKQwWDh+2MyuaNEwzbbilJ Cc: Garrett Cooper , "freebsd-performance@freebsd.org" Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 06:55:25 -0000 On 8/17/12 7:15 AM, Marie Bacuno II wrote: > > On Aug 16, 2012, at 18:47, Garrett Cooper wrote: > >> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper wrote: >>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacuño II wrote: >>>> Hello fellow listers, >>>> >>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>>> being auto-tuned to use >329853485875 (~307GB). >>>> >>>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>> >>>> # from sysctl >>>> vm.kmem_size_max: 329853485875 >>>> vm.kmem_size: 329853485875 >>>> >>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max >>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>>> manually set it in /boot/loader.conf. >>>> >>>> But on the machine with 512GB of RAM it just resets. For the machine >>>> to boot, we need to go to the loader prompt and issue: >>>> >>>> OK set vm.kmem_size_max="300G" >>>> OK boot >>>> >>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>>> vm.kmem_size_max is always set to 329853485875. >>>> >>>> How can I increase vm.kmem_size_max to use at least 500GB? And how is >>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max >>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>>> 490GB. >>>> >>>> I'm browsing thru the source code at >>>> http://fxr.watson.org/fxr/search?v=FREEBSD9&string=vm.kmem_size_max >>>> and I'm still trying to make sense of how vm.kmem_size_max is >>>> computed. >>>> >>>> I have posted the same topic on forums.freebsd.org but I'm not getting >>>> any recommendations. >>>> >>>> Please see the link for additional details: >>>> http://forums.freebsd.org/showthread.php?t=33977 >>> >>> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>> >>> Its architecture specific BTW... see >>> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >> >> Also, it's a tunable, not a sysctl... so you need to set the value in >> /boot/loader.conf . >> -Garrett > > Thanks for the quick reply. > > Yes, had it set on /boot/loader.conf and by trial and error on the loader prompt. > > We were able to bump it to 400G successfully. Tried 500G and 450G and the machine just spews out garbage in the screen. > > The latest output from "zfs-stats -a" with vm.kmem_size_max="400G" is in the forum: http://forums.freebsd.org/showthread.php?t=33977 > > About the code, I am looking into amd64 arch. Still checking the values of the variables.. Can't just retrieve them using getconf. If you can point me to a doxygen like documentation appreciate it a lot. > > Where does the constant value 329853485875 came from? > It comes from this macro: #define VM_KMEM_SIZE_MAX ((VM_MAX_KERNEL_ADDRESS - \ VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5) ((1<<39) * 3 / 5) = 329853488332 AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb. May be it's time to increase it again. I would asked kib@ or alc@ about that. -- Andrey Zonov