From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 00:47:19 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2FD81065672 for ; Fri, 17 Aug 2012 00:47:19 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8A5478FC12 for ; Fri, 17 Aug 2012 00:47:19 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2598789pbb.13 for ; Thu, 16 Aug 2012 17:47:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=FVgNNgNp8DS++1hSlVA+Sk+iOpvqFM6b7r0cxcm2TCU=; b=AQmZ4v2McMkBFcg5bR14ETEbZu3nSbyZJAt/V1M7U1eK9WgFUtuYrODP4xk1Lr9q+W axUzWC+UvOjSvLmOz48waxpm4jZNavXy5qDh8pKY2BV/Q7Oq2smbyD01vMFiqTwLVXtl YydX9Qe7osINTAgvVZGf+CZ4pLkciioOY7Y6iXGfkeCvykXyOq/CvScmWTwTxt4dbpGd LKVsudxrCZKioAQg20ZOWrg4Q+A7IikhX0RI5S9Edjo1dSaIFNAOiXKkZ21w56lSm36q 3MbF5RQZFHOYc4gnpn4BaXA7NZIxAv9VFgwOc9M5EPMc1Nfkn2YzHeF42tEgMiFInCvB 2h7g== Received: by 10.68.230.232 with SMTP id tb8mr7290117pbc.19.1345164438590; Thu, 16 Aug 2012 17:47:18 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.190.71 with HTTP; Thu, 16 Aug 2012 17:46:58 -0700 (PDT) From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Date: Thu, 16 Aug 2012 17:46:58 -0700 Message-ID: To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 00:47:19 -0000 Hello fellow listers, On a server with 512GB RAM it appears that vm.kmem_size_max is not being auto-tuned to use >329853485875 (~307GB). On this machine vm.kmem_size is equal to vm.kmem_size_max # from sysctl vm.kmem_size_max: 329853485875 vm.kmem_size: 329853485875 On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max to 330GB and vm.kmem_size automatically adjusts to 1GB even if I manually set it in /boot/loader.conf. But on the machine with 512GB of RAM it just resets. For the machine to boot, we need to go to the loader prompt and issue: OK set vm.kmem_size_max="300G" OK boot On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, vm.kmem_size_max is always set to 329853485875. How can I increase vm.kmem_size_max to use at least 500GB? And how is 329853485875 determined (formula)? I need to increase vm.kmem_size_max and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say 490GB. I'm browsing thru the source code at http://fxr.watson.org/fxr/search?v=FREEBSD9&string=vm.kmem_size_max and I'm still trying to make sense of how vm.kmem_size_max is computed. I have posted the same topic on forums.freebsd.org but I'm not getting any recommendations. Please see the link for additional details: http://forums.freebsd.org/showthread.php?t=33977 TIA! -- regards From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 01:44:47 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F7B71065673 for ; Fri, 17 Aug 2012 01:44:47 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 333758FC0C for ; Fri, 17 Aug 2012 01:44:46 +0000 (UTC) Received: by obbun3 with SMTP id un3so6052323obb.13 for ; Thu, 16 Aug 2012 18:44:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=gpz++Wqp5x/vCohN5B80l8MAvwUMTkMrDLSoLRVyhKI=; b=c4JVmL07RoiUzFKxTjhwCsG1VjaUgmjpNIHoj1qf6YkecuYcCnoG23CFl1lzBJ/ysp uoBAFRvjUy2cqqgoMci34ICSXpCo5sWt1bi2WSjARlasKhBSvWR6ZkQlXHfra7PMsZ5O m3mj4lFNfNndRQiAdAbw/3FkddVAa0gWYiU/H6xHqv3SuRSUCe2216YaTx91t1AhKuOb 1Y40JNMHLaQlR5U6t055J6jKKVLOduL0jUVJJzzSzy3FpZ8KD5+kw4jgBLhpdyPuavBP Us39jjggKBQbqB7VPrbwjv43IB69/DKKrRisnptCR4bwMtXU8BgKUTcQOMsNv3e60GX/ krrA== MIME-Version: 1.0 Received: by 10.182.111.39 with SMTP id if7mr2551360obb.56.1345167886251; Thu, 16 Aug 2012 18:44:46 -0700 (PDT) Received: by 10.76.142.201 with HTTP; Thu, 16 Aug 2012 18:44:46 -0700 (PDT) In-Reply-To: References: Date: Thu, 16 Aug 2012 18:44:46 -0700 Message-ID: From: Garrett Cooper To: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 01:44:47 -0000 On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=F1o II = wrote: > Hello fellow listers, > > On a server with 512GB RAM it appears that vm.kmem_size_max is not > being auto-tuned to use >329853485875 (~307GB). > > On this machine vm.kmem_size is equal to vm.kmem_size_max > > # from sysctl > vm.kmem_size_max: 329853485875 > vm.kmem_size: 329853485875 > > On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max > to 330GB and vm.kmem_size automatically adjusts to 1GB even if I > manually set it in /boot/loader.conf. > > But on the machine with 512GB of RAM it just resets. For the machine > to boot, we need to go to the loader prompt and issue: > > OK set vm.kmem_size_max=3D"300G" > OK boot > > On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, > vm.kmem_size_max is always set to 329853485875. > > How can I increase vm.kmem_size_max to use at least 500GB? And how is > 329853485875 determined (formula)? I need to increase vm.kmem_size_max > and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say > 490GB. > > I'm browsing thru the source code at > http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_max > and I'm still trying to make sense of how vm.kmem_size_max is > computed. > > I have posted the same topic on forums.freebsd.org but I'm not getting > any recommendations. > > Please see the link for additional details: > http://forums.freebsd.org/showthread.php?t=3D33977 Have you tried defining VM_KMEM_SIZE_MAX to your target value? Its architecture specific BTW... see sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. Cheers, -Garrett From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 01:47:30 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4C331065672 for ; Fri, 17 Aug 2012 01:47:30 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id A7CDB8FC12 for ; Fri, 17 Aug 2012 01:47:30 +0000 (UTC) Received: by obbun3 with SMTP id un3so6056855obb.13 for ; Thu, 16 Aug 2012 18:47:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=9K3ymMfRTl0BRyJWhbp6zUjm2I3BADpZQP69gu4yN00=; b=TfPxxzLW/FHIZ5mMHziIAR7zLoAfoOA2x5QxJmdpeUTTcKFYKKJqsUAYwXIJfzsa6M lxcCQhrVct9oL0QfVDoYboDQhuD96ziZVm/fY9oDQW+E5vesNz3x2ubhvTJCuptr+Uwx C708ItuHQXr0U9cnfBfT0WVs+yP7TWZF6sEf+CSB0TzbMN4qn4YjL5y62XDBXW9kPzUB qqvwbXygeJyGnhoBsT2fARAdF/sU4W5eEBfoHekjSh0gMmJyYuo3hmOxDpyabnorrGWQ cH6Kh5MtSRDhPNqlWhxMQHhh0kI8YpGRvLou2kVfjunnyh+dnXMxx+pFwo337O0Sp8m2 F+MA== MIME-Version: 1.0 Received: by 10.60.28.101 with SMTP id a5mr2599047oeh.69.1345168050024; Thu, 16 Aug 2012 18:47:30 -0700 (PDT) Received: by 10.76.142.201 with HTTP; Thu, 16 Aug 2012 18:47:29 -0700 (PDT) In-Reply-To: References: Date: Thu, 16 Aug 2012 18:47:29 -0700 Message-ID: From: Garrett Cooper To: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 01:47:31 -0000 On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper wrote: > On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=F1o II wrote: >> Hello fellow listers, >> >> On a server with 512GB RAM it appears that vm.kmem_size_max is not >> being auto-tuned to use >329853485875 (~307GB). >> >> On this machine vm.kmem_size is equal to vm.kmem_size_max >> >> # from sysctl >> vm.kmem_size_max: 329853485875 >> vm.kmem_size: 329853485875 >> >> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max >> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >> manually set it in /boot/loader.conf. >> >> But on the machine with 512GB of RAM it just resets. For the machine >> to boot, we need to go to the loader prompt and issue: >> >> OK set vm.kmem_size_max=3D"300G" >> OK boot >> >> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >> vm.kmem_size_max is always set to 329853485875. >> >> How can I increase vm.kmem_size_max to use at least 500GB? And how is >> 329853485875 determined (formula)? I need to increase vm.kmem_size_max >> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >> 490GB. >> >> I'm browsing thru the source code at >> http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_max >> and I'm still trying to make sense of how vm.kmem_size_max is >> computed. >> >> I have posted the same topic on forums.freebsd.org but I'm not getting >> any recommendations. >> >> Please see the link for additional details: >> http://forums.freebsd.org/showthread.php?t=3D33977 > > Have you tried defining VM_KMEM_SIZE_MAX to your target value? > > Its architecture specific BTW... see > sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. Also, it's a tunable, not a sysctl... so you need to set the value in /boot/loader.conf . -Garrett From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 03:15:47 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F4C2106566B for ; Fri, 17 Aug 2012 03:15:47 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4BB698FC08 for ; Fri, 17 Aug 2012 03:15:47 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2781967pbb.13 for ; Thu, 16 Aug 2012 20:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:in-reply-to:mime-version:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=V82TycksMa6Xc59BS/W0KEcMGZgG4L9jVc5gsuB0cHM=; b=Hb+30HV1n9y7mCvwORHVI3BrJFfALER+8JXQLHRgmub2dfMLSDAQQ5xPl+/iSzxQBX ddDcoAb3RJCqKG95R9Njeh0aPYee1bpahdFeraoWeXvHfYwQDvYqMnogFCCPUFrbk/Hl 9M/+WM+MfdLnW6vF8p+B3d4MM9FfXRXiosQPREwonyzXoj0XPJqTIuYRh/MjtcmtLrkB GFf9VYJDtFA5AqACa53KJcnyj05jm8ZItYx+L1DWVDKd3MJeaSclmIej2gmeU1w7mHOp qOuJ5DAcVbV3x3Kp+6qmIzqoiWBPGvjvz8A1L3uGlZCyJRhNtUmJwVUrMufoH+j7E7O9 ovNw== Received: by 10.66.74.100 with SMTP id s4mr6436401pav.27.1345173346628; Thu, 16 Aug 2012 20:15:46 -0700 (PDT) Received: from [192.168.1.69] (pool-96-251-115-86.lsanca.dsl-w.verizon.net. [96.251.115.86]) by mx.google.com with ESMTPS id or1sm3943274pbb.10.2012.08.16.20.15.44 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 16 Aug 2012 20:15:45 -0700 (PDT) References: In-Reply-To: Mime-Version: 1.0 (iPod Mail 8C148) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPod Mail (8C148) From: Marie Bacuno II Date: Thu, 16 Aug 2012 20:15:48 -0700 To: Garrett Cooper Cc: "freebsd-performance@freebsd.org" Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 03:15:47 -0000 On Aug 16, 2012, at 18:47, Garrett Cooper wrote: > On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper wrote= : >> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=C3=B1o II wrote: >>> Hello fellow listers, >>>=20 >>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>> being auto-tuned to use >329853485875 (~307GB). >>>=20 >>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>=20 >>> # from sysctl >>> vm.kmem_size_max: 329853485875 >>> vm.kmem_size: 329853485875 >>>=20 >>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max >>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>> manually set it in /boot/loader.conf. >>>=20 >>> But on the machine with 512GB of RAM it just resets. For the machine >>> to boot, we need to go to the loader prompt and issue: >>>=20 >>> OK set vm.kmem_size_max=3D"300G" >>> OK boot >>>=20 >>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>> vm.kmem_size_max is always set to 329853485875. >>>=20 >>> How can I increase vm.kmem_size_max to use at least 500GB? And how is >>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max >>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>> 490GB. >>>=20 >>> I'm browsing thru the source code at >>> http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_max >>> and I'm still trying to make sense of how vm.kmem_size_max is >>> computed. >>>=20 >>> I have posted the same topic on forums.freebsd.org but I'm not getting >>> any recommendations. >>>=20 >>> Please see the link for additional details: >>> http://forums.freebsd.org/showthread.php?t=3D33977 >>=20 >> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>=20 >> Its architecture specific BTW... see >> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >=20 > Also, it's a tunable, not a sysctl... so you need to set the value in > /boot/loader.conf . > -Garrett Thanks for the quick reply. Yes, had it set on /boot/loader.conf and by trial and error on the loader pr= ompt. We were able to bump it to 400G successfully. Tried 500G and 450G and the ma= chine just spews out garbage in the screen. The latest output from "zfs-stats -a" with vm.kmem_size_max=3D"400G" is in t= he forum: http://forums.freebsd.org/showthread.php?t=3D33977 About the code, I am looking into amd64 arch. Still checking the values of t= he variables.. Can't just retrieve them using getconf. If you can point me t= o a doxygen like documentation appreciate it a lot. Where does the constant value 329853485875 came from? Thanks! From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 06:55:25 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5B60D106566B for ; Fri, 17 Aug 2012 06:55:25 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id C266B8FC0A for ; Fri, 17 Aug 2012 06:55:24 +0000 (UTC) Received: by lbbgk8 with SMTP id gk8so2325906lbb.13 for ; Thu, 16 Aug 2012 23:55:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=gjbDKCC8RbEX8cCUxm3t+S9f5YmlqorlDrsohlXwqUk=; b=WPcuCEhnsoWAggK4WJxjYI8TTtmQoVta59gLHOnfIjfP6UVOQMq4zJOkd7chYRC4sO C2XRUmvmpSew9XHPmD2aKgadcburgoYy36D0ytRVgfqFmUfzMnBVhjASAfvMJ87fEYKf omc+NHX4hlq0NWvmLtALYI7WYh0+pUW0qxCH2kMQwGVIqPA6cBmAb0oMMTO822KAavDs Z89+ItIJ985R/R8lAHs9B3vmIOQQMBcMWNoEKyarmFV3dkxFqF/AXS6pItR/LF8UEhef gOjEM3Sw3g5iRE22pC8j1qOSlzxsCXyrbQlrUC3G8MEE26YVIbAp6ktIZd+CpBTKM9XC TwVw== Received: by 10.112.103.68 with SMTP id fu4mr1805956lbb.56.1345186523367; Thu, 16 Aug 2012 23:55:23 -0700 (PDT) Received: from zont-osx.local (ppp95-165-138-47.pppoe.spdop.ru. [95.165.138.47]) by mx.google.com with ESMTPS id h8sm1510097lbi.13.2012.08.16.23.55.22 (version=SSLv3 cipher=OTHER); Thu, 16 Aug 2012 23:55:22 -0700 (PDT) Message-ID: <502DEAD9.6050304@zonov.org> Date: Fri, 17 Aug 2012 10:55:21 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Marie Bacuno II References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Gm-Message-State: ALoCoQk86KJKZT3emDy3DjAqabnVSeADSO3e77HViiX25VqPEO14RbKKQwWDh+2MyuaNEwzbbilJ Cc: Garrett Cooper , "freebsd-performance@freebsd.org" Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 06:55:25 -0000 On 8/17/12 7:15 AM, Marie Bacuno II wrote: > > On Aug 16, 2012, at 18:47, Garrett Cooper wrote: > >> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper wrote: >>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacuño II wrote: >>>> Hello fellow listers, >>>> >>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>>> being auto-tuned to use >329853485875 (~307GB). >>>> >>>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>> >>>> # from sysctl >>>> vm.kmem_size_max: 329853485875 >>>> vm.kmem_size: 329853485875 >>>> >>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max >>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>>> manually set it in /boot/loader.conf. >>>> >>>> But on the machine with 512GB of RAM it just resets. For the machine >>>> to boot, we need to go to the loader prompt and issue: >>>> >>>> OK set vm.kmem_size_max="300G" >>>> OK boot >>>> >>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>>> vm.kmem_size_max is always set to 329853485875. >>>> >>>> How can I increase vm.kmem_size_max to use at least 500GB? And how is >>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max >>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>>> 490GB. >>>> >>>> I'm browsing thru the source code at >>>> http://fxr.watson.org/fxr/search?v=FREEBSD9&string=vm.kmem_size_max >>>> and I'm still trying to make sense of how vm.kmem_size_max is >>>> computed. >>>> >>>> I have posted the same topic on forums.freebsd.org but I'm not getting >>>> any recommendations. >>>> >>>> Please see the link for additional details: >>>> http://forums.freebsd.org/showthread.php?t=33977 >>> >>> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>> >>> Its architecture specific BTW... see >>> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >> >> Also, it's a tunable, not a sysctl... so you need to set the value in >> /boot/loader.conf . >> -Garrett > > Thanks for the quick reply. > > Yes, had it set on /boot/loader.conf and by trial and error on the loader prompt. > > We were able to bump it to 400G successfully. Tried 500G and 450G and the machine just spews out garbage in the screen. > > The latest output from "zfs-stats -a" with vm.kmem_size_max="400G" is in the forum: http://forums.freebsd.org/showthread.php?t=33977 > > About the code, I am looking into amd64 arch. Still checking the values of the variables.. Can't just retrieve them using getconf. If you can point me to a doxygen like documentation appreciate it a lot. > > Where does the constant value 329853485875 came from? > It comes from this macro: #define VM_KMEM_SIZE_MAX ((VM_MAX_KERNEL_ADDRESS - \ VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5) ((1<<39) * 3 / 5) = 329853488332 AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb. May be it's time to increase it again. I would asked kib@ or alc@ about that. -- Andrey Zonov From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 20:16:35 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B6B7106566C; Fri, 17 Aug 2012 20:16:35 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0C5878FC16; Fri, 17 Aug 2012 20:16:34 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so4142874pbb.13 for ; Fri, 17 Aug 2012 13:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=IvCbIb3NSLZMJAkOkvetpxOzf2GxHLxqle6b7IRngeg=; b=LT0AL7qo2XtaksZWjlsDHAjwxlbYtrOyb4xdLi9O95F7i0ENbwS3FSB2rBAIxPAA3p AU7GYBeFtVr2W6YEcBo79nu2ca+FtJ7q2TnhxVTkCzTzaZkeqA7lde1kq8vvBhuVObx3 YcvHQ1Kgurz8qf22ZktJSdufpfC3Aw6jCb5K112Jpx+qJkPuGZTyo2f6JS6E9DnOjboo 8AVXtdYIu30qA2fVoVCgaYUd1+SEiRZALx1QEgy3EytW+aC8z0QFdNSSGrAhEntHTji0 9Y9MqQ6Fawn3xgvUeSaIfei2tGyeFQdZ/Obby0SYGoAc+1zLASiBel9beaZZqCoXt5/d TlIA== Received: by 10.68.230.232 with SMTP id tb8mr14352500pbc.19.1345234594421; Fri, 17 Aug 2012 13:16:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.190.71 with HTTP; Fri, 17 Aug 2012 13:16:13 -0700 (PDT) In-Reply-To: References: <502DEAD9.6050304@zonov.org> From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Date: Fri, 17 Aug 2012 13:16:13 -0700 Message-ID: To: Andrey Zonov , alc@freebsd.org, kib@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 20:16:35 -0000 On Fri, Aug 17, 2012 at 7:38 AM, Gezeala M. Bacu=F1o II = wrote: > On Thu, Aug 16, 2012 at 11:55 PM, Andrey Zonov wrote: >> On 8/17/12 7:15 AM, Marie Bacuno II wrote: >>> >>> >>> On Aug 16, 2012, at 18:47, Garrett Cooper wrote: >>> >>>> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper >>>> wrote: >>>>> >>>>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=F1o II >>>>> wrote: >>>>>> >>>>>> Hello fellow listers, >>>>>> >>>>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>>>>> being auto-tuned to use >329853485875 (~307GB). >>>>>> >>>>>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>>>> >>>>>> # from sysctl >>>>>> vm.kmem_size_max: 329853485875 >>>>>> vm.kmem_size: 329853485875 >>>>>> >>>>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_m= ax >>>>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>>>>> manually set it in /boot/loader.conf. >>>>>> >>>>>> But on the machine with 512GB of RAM it just resets. For the machine >>>>>> to boot, we need to go to the loader prompt and issue: >>>>>> >>>>>> OK set vm.kmem_size_max=3D"300G" >>>>>> OK boot >>>>>> >>>>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>>>>> vm.kmem_size_max is always set to 329853485875. >>>>>> >>>>>> How can I increase vm.kmem_size_max to use at least 500GB? And how i= s >>>>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_m= ax >>>>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>>>>> 490GB. >>>>>> >>>>>> I'm browsing thru the source code at >>>>>> http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_= max >>>>>> and I'm still trying to make sense of how vm.kmem_size_max is >>>>>> computed. >>>>>> >>>>>> I have posted the same topic on forums.freebsd.org but I'm not getti= ng >>>>>> any recommendations. >>>>>> >>>>>> Please see the link for additional details: >>>>>> http://forums.freebsd.org/showthread.php?t=3D33977 >>>>> >>>>> >>>>> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>>>> >>>>> Its architecture specific BTW... see >>>>> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >>>> >>>> >>>> Also, it's a tunable, not a sysctl... so you need to set the value in >>>> /boot/loader.conf . >>>> -Garrett >>> >>> >>> Thanks for the quick reply. >>> >>> Yes, had it set on /boot/loader.conf and by trial and error on the load= er >>> prompt. >>> >>> We were able to bump it to 400G successfully. Tried 500G and 450G and t= he >>> machine just spews out garbage in the screen. >>> >>> The latest output from "zfs-stats -a" with vm.kmem_size_max=3D"400G" is= in >>> the forum: http://forums.freebsd.org/showthread.php?t=3D33977 >>> >>> About the code, I am looking into amd64 arch. Still checking the values= of >>> the variables.. Can't just retrieve them using getconf. If you can poin= t me >>> to a doxygen like documentation appreciate it a lot. >>> >>> Where does the constant value 329853485875 came from? >>> >> >> It comes from this macro: >> >> #define VM_KMEM_SIZE_MAX ((VM_MAX_KERNEL_ADDRESS - \ >> VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5) >> >> ((1<<39) * 3 / 5) =3D 329853488332 >> >> AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb. May be it's time to >> increase it again. I would asked kib@ or alc@ about that. >> >> -- >> Andrey Zonov > Thanks! That's great (great for deriving 512GB) but looks like bad news for us, we've really hit some limits there (FreeBSD auto-tuning wise). As I've stated above, we have tried setting vm.kmem_size_max to 500GB/450GB unsuccessfully so there may be some part of the code that's breaking. Is there any thread or discussion where you can point me as to why they used only 60%(hard coded) of 512GB? Some relevant codes I've gathered on this machine: /usr/src/sys/amd64/include/vmparam.h /* * Virtual addresses of things. Derived from the page directory and * page table indexes from pmap.h for precision. * * 0x0000000000000000 - 0x00007fffffffffff user map * 0x0000800000000000 - 0xffff7fffffffffff does not exist (hole) * 0xffff800000000000 - 0xffff804020100fff recursive page table (512GB sl= ot) * 0xffff804020101000 - 0xfffffdffffffffff unused * 0xfffffe0000000000 - 0xfffffeffffffffff 1TB direct map * 0xffffff0000000000 - 0xffffff7fffffffff unused * 0xffffff8000000000 - 0xffffffffffffffff 512GB kernel map * * Within the kernel map: * * 0xffffffff80000000 KERNBASE */ #define VM_MAX_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-= 1) #define VM_MIN_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-512, 0, 0) /usr/src/sys/amd64/include/pmap.h /* * Pte related macros. This is complicated by having to deal with * the sign extension of the 48th bit. */ #define KVADDR(l4, l3, l2, l1) ( \ ((unsigned long)-1 << 47) | \ ((unsigned long)(l4) << PML4SHIFT) | \ ((unsigned long)(l3) << PDPSHIFT) | \ ((unsigned long)(l2) << PDRSHIFT) | \ ((unsigned long)(l1) << PAGE_SHIFT)) /usr/src/sys/amd64/include/param.h #define PAGE_SHIFT 12 /* LOG2(PAGE_SIZE) */ #define PDRSHIFT 21 /* LOG2(NBPDR) */ #define PDPSHIFT 30 /* LOG2(NBPDP) */ #define PML4SHIFT 39 /* LOG2(NBPML4) */ Yet to derive: KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-1. Checking pmap.c, vm_machdep.c etc. Additional Info: 1] Installed using PCBSD-9 Release amd64. 2] uname -a FreeBSD fmt-iscsi-stg1.musicreports.com 9.0-RELEASE FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.= 0/sys/GENERIC amd64 3] first few lines from /var/run/dmesg.boot: FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-sourc= e/9.0/sys/GENERIC amd64 CPU: Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (2666.82-MHz K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x206f2 Family =3D 6 Model =3D 2f St= epping =3D 2 Features=3D0xbfebfbff Features2=3D0x29ee3ff AMD Features=3D0x2c100800 AMD Features2=3D0x1 TSC: P-state invariant, performance statistics real memory =3D 549755813888 (524288 MB) avail memory =3D 530339893248 (505771 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 8 package(s) x 8 core(s) 4] relevant sysctl's with manual tuning: kern.maxusers: 384 kern.maxvnodes: 8222162 vfs.numvnodes: 675740 vfs.freevnodes: 417524 kern.ipc.somaxconn: 128 kern.openfiles: 5238 vfs.zfs.arc_max: 428422987776 vfs.zfs.arc_min: 53552873472 vfs.zfs.arc_meta_used: 3167391088 vfs.zfs.arc_meta_limit: 107105746944 vm.kmem_size_max: 429496729600 =3D=3D>> manually tuned vm.kmem_size: 429496729600 =3D=3D>> manually tuned vm.kmem_map_free: 107374727168 vm.kmem_map_size: 144625156096 vfs.wantfreevnodes: 2055540 kern.minvnodes: 2055540 kern.maxfiles: 197248 =3D=3D>> manually tuned vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Processes: (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 150) Virtual Memory: (Total: 1086325716K Active: 12377876K) Real Memory: (Total: 144143408K Active: 803432K) Shared Virtual Memory: (Total: 81384K Active: 37560K) Shared Real Memory: (Total: 32224K Active: 27548K) Free Memory Pages: 365565564K hw.availpages: 134170294 hw.physmem: 549561524224 hw.usermem: 391395241984 hw.realmem: 551836188672 vm.kmem_size_scale: 1 kern.ipc.nmbclusters: 2560000 =3D=3D>> manually tuned kern.ipc.maxsockbuf: 2097152 net.inet.tcp.sendbuf_max: 2097152 net.inet.tcp.recvbuf_max: 2097152 kern.maxfilesperproc: 18000 net.inet.ip.intr_queue_maxlen: 256 kern.maxswzone: 33554432 kern.ipc.shmmax: 10737418240 =3D=3D>> manually tuned kern.ipc.shmall: 2621440 =3D=3D>> manually tuned vfs.zfs.write_limit_override: 0 vfs.zfs.prefetch_disable: 0 hw.pagesize: 4096 hw.availpages: 134170294 kern.ipc.maxpipekva: 8586895360 kern.ipc.shm_use_phys: 1 =3D=3D>> manually tuned vfs.vmiodirenable: 1 debug.numcache: 632148 vfs.ncsizefactor: 2 vm.kvm_size: 549755809792 vm.kvm_free: 54456741888 kern.ipc.semmni: 256 kern.ipc.semmns: 512 kern.ipc.semmnu: 256 Thanks! From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 20:58:58 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5A6491065670; Fri, 17 Aug 2012 20:58:58 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id 22FF28FC12; Fri, 17 Aug 2012 20:58:57 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 9D3C74C01F1; Fri, 17 Aug 2012 15:58:56 -0500 (CDT) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 9B9244C01BC; Fri, 17 Aug 2012 15:58:56 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id 16B_LeHZMAGj; Fri, 17 Aug 2012 15:58:56 -0500 (CDT) Received: from [10.74.20.46] (staff-74-dun20-046.rice.edu [10.74.20.46]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id 487674C01E4; Fri, 17 Aug 2012 15:58:56 -0500 (CDT) Message-ID: <502EB081.3030801@rice.edu> Date: Fri, 17 Aug 2012 15:58:41 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: =?ISO-8859-1?Q?=22Gezeala_M=2E_Bacu=F1o_II=22?= References: <502DEAD9.6050304@zonov.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 17 Aug 2012 21:00:41 +0000 Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov , kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 20:58:58 -0000 vm.kmem_size controls the maximum size of the kernel's heap, i.e., the region where the kernel's slab and malloc()-like memory allocators obtain their memory. While this heap may occupy the largest portion of the kernel's virtual address space, it cannot occupy the entirety of the address space. There are other things that must be given space within the kernel's address space, for example, the file system buffer map. ZFS does not, however, use the regular file system buffer cache. The ARC takes its place, and the ARC abuses the kernel's heap like nothing else. So, if you are running a machine that only makes trivial use of a non-ZFS file system, like you boot from UFS, but store all of your data in ZFS, then you can dramatically reduce the size of the buffer map via boot loader tuneables and proportionately increase vm.kmem_size. Any further increases in the kernel virtual address space size will, however, require code changes. Small changes, but changes nonetheless. Alan On 8/17/2012 3:16 PM, Gezeala M. Bacuño II wrote: > On Fri, Aug 17, 2012 at 7:38 AM, Gezeala M. Bacuño II wrote: >> On Thu, Aug 16, 2012 at 11:55 PM, Andrey Zonov wrote: >>> On 8/17/12 7:15 AM, Marie Bacuno II wrote: >>>> >>>> On Aug 16, 2012, at 18:47, Garrett Cooper wrote: >>>> >>>>> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper >>>>> wrote: >>>>>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacuño II >>>>>> wrote: >>>>>>> Hello fellow listers, >>>>>>> >>>>>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not >>>>>>> being auto-tuned to use >329853485875 (~307GB). >>>>>>> >>>>>>> On this machine vm.kmem_size is equal to vm.kmem_size_max >>>>>>> >>>>>>> # from sysctl >>>>>>> vm.kmem_size_max: 329853485875 >>>>>>> vm.kmem_size: 329853485875 >>>>>>> >>>>>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max >>>>>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I >>>>>>> manually set it in /boot/loader.conf. >>>>>>> >>>>>>> But on the machine with 512GB of RAM it just resets. For the machine >>>>>>> to boot, we need to go to the loader prompt and issue: >>>>>>> >>>>>>> OK set vm.kmem_size_max="300G" >>>>>>> OK boot >>>>>>> >>>>>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have, >>>>>>> vm.kmem_size_max is always set to 329853485875. >>>>>>> >>>>>>> How can I increase vm.kmem_size_max to use at least 500GB? And how is >>>>>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max >>>>>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say >>>>>>> 490GB. >>>>>>> >>>>>>> I'm browsing thru the source code at >>>>>>> http://fxr.watson.org/fxr/search?v=FREEBSD9&string=vm.kmem_size_max >>>>>>> and I'm still trying to make sense of how vm.kmem_size_max is >>>>>>> computed. >>>>>>> >>>>>>> I have posted the same topic on forums.freebsd.org but I'm not getting >>>>>>> any recommendations. >>>>>>> >>>>>>> Please see the link for additional details: >>>>>>> http://forums.freebsd.org/showthread.php?t=33977 >>>>>> >>>>>> Have you tried defining VM_KMEM_SIZE_MAX to your target value? >>>>>> >>>>>> Its architecture specific BTW... see >>>>>> sys//include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`. >>>>> >>>>> Also, it's a tunable, not a sysctl... so you need to set the value in >>>>> /boot/loader.conf . >>>>> -Garrett >>>> >>>> Thanks for the quick reply. >>>> >>>> Yes, had it set on /boot/loader.conf and by trial and error on the loader >>>> prompt. >>>> >>>> We were able to bump it to 400G successfully. Tried 500G and 450G and the >>>> machine just spews out garbage in the screen. >>>> >>>> The latest output from "zfs-stats -a" with vm.kmem_size_max="400G" is in >>>> the forum: http://forums.freebsd.org/showthread.php?t=33977 >>>> >>>> About the code, I am looking into amd64 arch. Still checking the values of >>>> the variables.. Can't just retrieve them using getconf. If you can point me >>>> to a doxygen like documentation appreciate it a lot. >>>> >>>> Where does the constant value 329853485875 came from? >>>> >>> It comes from this macro: >>> >>> #define VM_KMEM_SIZE_MAX ((VM_MAX_KERNEL_ADDRESS - \ >>> VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5) >>> >>> ((1<<39) * 3 / 5) = 329853488332 >>> >>> AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb. May be it's time to >>> increase it again. I would asked kib@ or alc@ about that. >>> >>> -- >>> Andrey Zonov > Thanks! That's great (great for deriving 512GB) but looks like bad > news for us, we've really hit some limits there (FreeBSD auto-tuning > wise). As I've stated above, we have tried setting vm.kmem_size_max to > 500GB/450GB unsuccessfully so there may be some part of the code > that's breaking. Is there any thread or discussion where you can point > me as to why they used only 60%(hard coded) of 512GB? > > Some relevant codes I've gathered on this machine: > > /usr/src/sys/amd64/include/vmparam.h > /* > * Virtual addresses of things. Derived from the page directory and > * page table indexes from pmap.h for precision. > * > * 0x0000000000000000 - 0x00007fffffffffff user map > * 0x0000800000000000 - 0xffff7fffffffffff does not exist (hole) > * 0xffff800000000000 - 0xffff804020100fff recursive page table (512GB slot) > * 0xffff804020101000 - 0xfffffdffffffffff unused > * 0xfffffe0000000000 - 0xfffffeffffffffff 1TB direct map > * 0xffffff0000000000 - 0xffffff7fffffffff unused > * 0xffffff8000000000 - 0xffffffffffffffff 512GB kernel map > * > * Within the kernel map: > * > * 0xffffffff80000000 KERNBASE > */ > > #define VM_MAX_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-1) > #define VM_MIN_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-512, 0, 0) > > /usr/src/sys/amd64/include/pmap.h > /* > * Pte related macros. This is complicated by having to deal with > * the sign extension of the 48th bit. > */ > #define KVADDR(l4, l3, l2, l1) ( \ > ((unsigned long)-1 << 47) | \ > ((unsigned long)(l4) << PML4SHIFT) | \ > ((unsigned long)(l3) << PDPSHIFT) | \ > ((unsigned long)(l2) << PDRSHIFT) | \ > ((unsigned long)(l1) << PAGE_SHIFT)) > > /usr/src/sys/amd64/include/param.h > #define PAGE_SHIFT 12 /* LOG2(PAGE_SIZE) */ > #define PDRSHIFT 21 /* LOG2(NBPDR) */ > #define PDPSHIFT 30 /* LOG2(NBPDP) */ > #define PML4SHIFT 39 /* LOG2(NBPML4) */ > > Yet to derive: KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-1. Checking pmap.c, > vm_machdep.c etc. > > Additional Info: > 1] Installed using PCBSD-9 Release amd64. > > 2] uname -a > FreeBSD fmt-iscsi-stg1.musicreports.com 9.0-RELEASE FreeBSD > 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 > root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.0/sys/GENERIC > amd64 > > 3] first few lines from /var/run/dmesg.boot: > FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 > root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.0/sys/GENERIC > amd64 > CPU: Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (2666.82-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206f2 Family = 6 Model = 2f Stepping = 2 > Features=0xbfebfbff > Features2=0x29ee3ff > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 549755813888 (524288 MB) > avail memory = 530339893248 (505771 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs > FreeBSD/SMP: 8 package(s) x 8 core(s) > > 4] relevant sysctl's with manual tuning: > kern.maxusers: 384 > kern.maxvnodes: 8222162 > vfs.numvnodes: 675740 > vfs.freevnodes: 417524 > kern.ipc.somaxconn: 128 > kern.openfiles: 5238 > vfs.zfs.arc_max: 428422987776 > vfs.zfs.arc_min: 53552873472 > vfs.zfs.arc_meta_used: 3167391088 > vfs.zfs.arc_meta_limit: 107105746944 > vm.kmem_size_max: 429496729600 ==>> manually tuned > vm.kmem_size: 429496729600 ==>> manually tuned > vm.kmem_map_free: 107374727168 > vm.kmem_map_size: 144625156096 > vfs.wantfreevnodes: 2055540 > kern.minvnodes: 2055540 > kern.maxfiles: 197248 ==>> manually tuned > vm.vmtotal: > System wide totals computed every five seconds: (values in kilobytes) > =============================================== > Processes: (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 150) > Virtual Memory: (Total: 1086325716K Active: 12377876K) > Real Memory: (Total: 144143408K Active: 803432K) > Shared Virtual Memory: (Total: 81384K Active: 37560K) > Shared Real Memory: (Total: 32224K Active: 27548K) > Free Memory Pages: 365565564K > > hw.availpages: 134170294 > hw.physmem: 549561524224 > hw.usermem: 391395241984 > hw.realmem: 551836188672 > vm.kmem_size_scale: 1 > kern.ipc.nmbclusters: 2560000 ==>> manually tuned > kern.ipc.maxsockbuf: 2097152 > net.inet.tcp.sendbuf_max: 2097152 > net.inet.tcp.recvbuf_max: 2097152 > kern.maxfilesperproc: 18000 > net.inet.ip.intr_queue_maxlen: 256 > kern.maxswzone: 33554432 > kern.ipc.shmmax: 10737418240 ==>> manually tuned > kern.ipc.shmall: 2621440 ==>> manually tuned > vfs.zfs.write_limit_override: 0 > vfs.zfs.prefetch_disable: 0 > hw.pagesize: 4096 > hw.availpages: 134170294 > kern.ipc.maxpipekva: 8586895360 > kern.ipc.shm_use_phys: 1 ==>> manually tuned > vfs.vmiodirenable: 1 > debug.numcache: 632148 > vfs.ncsizefactor: 2 > vm.kvm_size: 549755809792 > vm.kvm_free: 54456741888 > kern.ipc.semmni: 256 > kern.ipc.semmns: 512 > kern.ipc.semmnu: 256 > > > Thanks! > From owner-freebsd-performance@FreeBSD.ORG Fri Aug 17 22:08:24 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 955EB1065672; Fri, 17 Aug 2012 22:08:24 +0000 (UTC) (envelope-from gezeala@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5673D8FC12; Fri, 17 Aug 2012 22:08:23 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so4276547pbb.13 for ; Fri, 17 Aug 2012 15:08:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=14mBDzPlSaCSQgADgxKuoeTiOMeJZ2qZefMES/wifh0=; b=Np/a5Ae67XA9cWS2F2YaGUXgN5et2DtcXavQ8yivPLS+gdA7RHZfHBRnvGLH9XWVBr AOY8FJQ/qon/BSguuAlD2pamEtJCq7pzsaX1YlU/qLo/WxJGJXDFndh2nI03AZn40b9e +k44o1/Sxrv6qCpYFdC9e8j2EcQAQ8xbrb6gwU/Ub6ughI5q+JBK/8+e4vPybPchKfaE jaw3BKGHds39afwuJ5zkQdtU+rlTX5K9nyfLNLu6ctKb9ovPYBaLUOpCpH7loZQIfoCI 3DeEQaId/Ddczx5J9vR9gHiTbN1Ir+PNx5l5DX7ccKVKBgEmjRhKBRXGA8eT0PggQDqJ Tebg== Received: by 10.66.77.7 with SMTP id o7mr12567374paw.37.1345241303323; Fri, 17 Aug 2012 15:08:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.190.71 with HTTP; Fri, 17 Aug 2012 15:08:03 -0700 (PDT) In-Reply-To: <502EB081.3030801@rice.edu> References: <502DEAD9.6050304@zonov.org> <502EB081.3030801@rice.edu> From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= Date: Fri, 17 Aug 2012 15:08:03 -0700 Message-ID: To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov , kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 22:08:24 -0000 On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox wrote: > vm.kmem_size controls the maximum size of the kernel's heap, i.e., the > region where the kernel's slab and malloc()-like memory allocators obtain > their memory. While this heap may occupy the largest portion of the > kernel's virtual address space, it cannot occupy the entirety of the address > space. There are other things that must be given space within the kernel's > address space, for example, the file system buffer map. > > ZFS does not, however, use the regular file system buffer cache. The ARC > takes its place, and the ARC abuses the kernel's heap like nothing else. > So, if you are running a machine that only makes trivial use of a non-ZFS > file system, like you boot from UFS, but store all of your data in ZFS, then > you can dramatically reduce the size of the buffer map via boot loader > tuneables and proportionately increase vm.kmem_size. > > Any further increases in the kernel virtual address space size will, > however, require code changes. Small changes, but changes nonetheless. > > Alan > > <> >> >> Additional Info: >> 1] Installed using PCBSD-9 Release amd64. >> >> 2] uname -a >> FreeBSD fmt-iscsi-stg1.musicreports.com 9.0-RELEASE FreeBSD >> 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 >> >> root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.0/sys/GENERIC >> amd64 >> >> 3] first few lines from /var/run/dmesg.boot: >> FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 >> >> root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.0/sys/GENERIC >> amd64 >> CPU: Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (2666.82-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x206f2 Family = 6 Model = 2f Stepping >> = 2 >> >> Features=0xbfebfbff >> >> Features2=0x29ee3ff >> AMD Features=0x2c100800 >> AMD Features2=0x1 >> TSC: P-state invariant, performance statistics >> real memory = 549755813888 (524288 MB) >> avail memory = 530339893248 (505771 MB) >> Event timer "LAPIC" quality 600 >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs >> FreeBSD/SMP: 8 package(s) x 8 core(s) >> >> 4] relevant sysctl's with manual tuning: >> kern.maxusers: 384 >> kern.maxvnodes: 8222162 >> vfs.numvnodes: 675740 >> vfs.freevnodes: 417524 >> kern.ipc.somaxconn: 128 >> kern.openfiles: 5238 >> vfs.zfs.arc_max: 428422987776 >> vfs.zfs.arc_min: 53552873472 >> vfs.zfs.arc_meta_used: 3167391088 >> vfs.zfs.arc_meta_limit: 107105746944 >> vm.kmem_size_max: 429496729600 ==>> manually tuned >> vm.kmem_size: 429496729600 ==>> manually tuned >> vm.kmem_map_free: 107374727168 >> vm.kmem_map_size: 144625156096 >> vfs.wantfreevnodes: 2055540 >> kern.minvnodes: 2055540 >> kern.maxfiles: 197248 ==>> manually tuned >> vm.vmtotal: >> System wide totals computed every five seconds: (values in kilobytes) >> =============================================== >> Processes: (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 150) >> Virtual Memory: (Total: 1086325716K Active: 12377876K) >> Real Memory: (Total: 144143408K Active: 803432K) >> Shared Virtual Memory: (Total: 81384K Active: 37560K) >> Shared Real Memory: (Total: 32224K Active: 27548K) >> Free Memory Pages: 365565564K >> >> hw.availpages: 134170294 >> hw.physmem: 549561524224 >> hw.usermem: 391395241984 >> hw.realmem: 551836188672 >> vm.kmem_size_scale: 1 >> kern.ipc.nmbclusters: 2560000 ==>> manually tuned >> kern.ipc.maxsockbuf: 2097152 >> net.inet.tcp.sendbuf_max: 2097152 >> net.inet.tcp.recvbuf_max: 2097152 >> kern.maxfilesperproc: 18000 >> net.inet.ip.intr_queue_maxlen: 256 >> kern.maxswzone: 33554432 >> kern.ipc.shmmax: 10737418240 ==>> manually tuned >> kern.ipc.shmall: 2621440 ==>> manually tuned >> vfs.zfs.write_limit_override: 0 >> vfs.zfs.prefetch_disable: 0 >> hw.pagesize: 4096 >> hw.availpages: 134170294 >> kern.ipc.maxpipekva: 8586895360 >> kern.ipc.shm_use_phys: 1 ==>> manually tuned >> vfs.vmiodirenable: 1 >> debug.numcache: 632148 >> vfs.ncsizefactor: 2 >> vm.kvm_size: 549755809792 >> vm.kvm_free: 54456741888 >> kern.ipc.semmni: 256 >> kern.ipc.semmns: 512 >> kern.ipc.semmnu: 256 >> Thanks. It will be mainly used for postgreSQL and java. We have a huge db (3TB and growing) and we need to have as much of it as we can on zfs' ARC. All data resides on zpools while root is on ufs. On 8.2 and 9 machines vm.kmem_size is always auto-tuned to almost the same size as our installed RAM. What I've tuned on those machines is lower vfs.zfs.arc_max to 50% or 75% of vm.kmem_size and that have worked well for us and the machines does not swap out. Now on this machine, I do think that I need to adjust my formula for tuning vfs.zfs.arc_max, 25% for other stuff is probably overkill. We were able to successfully bump vm.kmem_size_max and vm.kmem_size to 400GB: vm.kmem_size_max: 429496729600 ==>> manually tuned vm.kmem_size: 429496729600 ==>> manually tuned vfs.zfs.arc_max: 428422987776 ==>> auto-tuned (vm.kmem_size - 1G) vfs.zfs.arc_min: 53552873472 ==>> auto-tuned Which other tuneables do I need to set on /boot/loader.conf so we can boot the machine with vm.kmem_size > 400G. As I don't know which part of the boot-up process is failing with vm.kmem_size/_max set to 450G or 500G, I have no idea which to tune next. Thanks in advance. From owner-freebsd-performance@FreeBSD.ORG Sat Aug 18 19:14:25 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 92A99106564A; Sat, 18 Aug 2012 19:14:25 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh10.mail.rice.edu (mh10.mail.rice.edu [128.42.201.30]) by mx1.freebsd.org (Postfix) with ESMTP id 5AE848FC14; Sat, 18 Aug 2012 19:14:25 +0000 (UTC) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id BF5FC604FA; Sat, 18 Aug 2012 14:14:24 -0500 (CDT) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id BDA9860392; Sat, 18 Aug 2012 14:14:24 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh10.mail.rice.edu, auth channel Received: from mh10.mail.rice.edu ([127.0.0.1]) by mh10.mail.rice.edu (mh10.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id 4EaDTDgHDvNR; Sat, 18 Aug 2012 14:14:24 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh10.mail.rice.edu (Postfix) with ESMTPSA id 2688B603D3; Sat, 18 Aug 2012 14:14:24 -0500 (CDT) Message-ID: <502FE98E.40807@rice.edu> Date: Sat, 18 Aug 2012 14:14:22 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: =?ISO-8859-1?Q?=22Gezeala_M=2E_Bacu=F1o_II=22?= References: <502DEAD9.6050304@zonov.org> <502EB081.3030801@rice.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Sat, 18 Aug 2012 23:21:06 +0000 Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov , kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Aug 2012 19:14:25 -0000 On 08/17/2012 17:08, Gezeala M. Bacuño II wrote: > On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox wrote: >> vm.kmem_size controls the maximum size of the kernel's heap, i.e., the >> region where the kernel's slab and malloc()-like memory allocators obtain >> their memory. While this heap may occupy the largest portion of the >> kernel's virtual address space, it cannot occupy the entirety of the address >> space. There are other things that must be given space within the kernel's >> address space, for example, the file system buffer map. >> >> ZFS does not, however, use the regular file system buffer cache. The ARC >> takes its place, and the ARC abuses the kernel's heap like nothing else. >> So, if you are running a machine that only makes trivial use of a non-ZFS >> file system, like you boot from UFS, but store all of your data in ZFS, then >> you can dramatically reduce the size of the buffer map via boot loader >> tuneables and proportionately increase vm.kmem_size. >> >> Any further increases in the kernel virtual address space size will, >> however, require code changes. Small changes, but changes nonetheless. >> >> Alan >> >> > <> > >>> Additional Info: >>> 1] Installed using PCBSD-9 Release amd64. >>> >>> 2] uname -a >>> FreeBSD fmt-iscsi-stg1.musicreports.com 9.0-RELEASE FreeBSD >>> 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 >>> >>> root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.0/sys/GENERIC >>> amd64 >>> >>> 3] first few lines from /var/run/dmesg.boot: >>> FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 >>> >>> root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.0/sys/GENERIC >>> amd64 >>> CPU: Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (2666.82-MHz K8-class CPU) >>> Origin = "GenuineIntel" Id = 0x206f2 Family = 6 Model = 2f Stepping >>> = 2 >>> >>> Features=0xbfebfbff >>> >>> Features2=0x29ee3ff >>> AMD Features=0x2c100800 >>> AMD Features2=0x1 >>> TSC: P-state invariant, performance statistics >>> real memory = 549755813888 (524288 MB) >>> avail memory = 530339893248 (505771 MB) >>> Event timer "LAPIC" quality 600 >>> ACPI APIC Table: >>> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs >>> FreeBSD/SMP: 8 package(s) x 8 core(s) >>> >>> 4] relevant sysctl's with manual tuning: >>> kern.maxusers: 384 >>> kern.maxvnodes: 8222162 >>> vfs.numvnodes: 675740 >>> vfs.freevnodes: 417524 >>> kern.ipc.somaxconn: 128 >>> kern.openfiles: 5238 >>> vfs.zfs.arc_max: 428422987776 >>> vfs.zfs.arc_min: 53552873472 >>> vfs.zfs.arc_meta_used: 3167391088 >>> vfs.zfs.arc_meta_limit: 107105746944 >>> vm.kmem_size_max: 429496729600 ==>> manually tuned >>> vm.kmem_size: 429496729600 ==>> manually tuned >>> vm.kmem_map_free: 107374727168 >>> vm.kmem_map_size: 144625156096 >>> vfs.wantfreevnodes: 2055540 >>> kern.minvnodes: 2055540 >>> kern.maxfiles: 197248 ==>> manually tuned >>> vm.vmtotal: >>> System wide totals computed every five seconds: (values in kilobytes) >>> =============================================== >>> Processes: (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 150) >>> Virtual Memory: (Total: 1086325716K Active: 12377876K) >>> Real Memory: (Total: 144143408K Active: 803432K) >>> Shared Virtual Memory: (Total: 81384K Active: 37560K) >>> Shared Real Memory: (Total: 32224K Active: 27548K) >>> Free Memory Pages: 365565564K >>> >>> hw.availpages: 134170294 >>> hw.physmem: 549561524224 >>> hw.usermem: 391395241984 >>> hw.realmem: 551836188672 >>> vm.kmem_size_scale: 1 >>> kern.ipc.nmbclusters: 2560000 ==>> manually tuned >>> kern.ipc.maxsockbuf: 2097152 >>> net.inet.tcp.sendbuf_max: 2097152 >>> net.inet.tcp.recvbuf_max: 2097152 >>> kern.maxfilesperproc: 18000 >>> net.inet.ip.intr_queue_maxlen: 256 >>> kern.maxswzone: 33554432 >>> kern.ipc.shmmax: 10737418240 ==>> manually tuned >>> kern.ipc.shmall: 2621440 ==>> manually tuned >>> vfs.zfs.write_limit_override: 0 >>> vfs.zfs.prefetch_disable: 0 >>> hw.pagesize: 4096 >>> hw.availpages: 134170294 >>> kern.ipc.maxpipekva: 8586895360 >>> kern.ipc.shm_use_phys: 1 ==>> manually tuned >>> vfs.vmiodirenable: 1 >>> debug.numcache: 632148 >>> vfs.ncsizefactor: 2 >>> vm.kvm_size: 549755809792 >>> vm.kvm_free: 54456741888 >>> kern.ipc.semmni: 256 >>> kern.ipc.semmns: 512 >>> kern.ipc.semmnu: 256 >>> > Thanks. It will be mainly used for postgreSQL and java. We have a huge > db (3TB and growing) and we need to have as much of it as we can on > zfs' ARC. All data resides on zpools while root is on ufs. On 8.2 and > 9 machines vm.kmem_size is always auto-tuned to almost the same size > as our installed RAM. What I've tuned on those machines is lower > vfs.zfs.arc_max to 50% or 75% of vm.kmem_size and that have worked > well for us and the machines does not swap out. Now on this machine, I > do think that I need to adjust my formula for tuning vfs.zfs.arc_max, > 25% for other stuff is probably overkill. > > We were able to successfully bump vm.kmem_size_max and vm.kmem_size to 400GB: > vm.kmem_size_max: 429496729600 ==>> manually tuned > vm.kmem_size: 429496729600 ==>> manually tuned > vfs.zfs.arc_max: 428422987776 ==>> auto-tuned (vm.kmem_size - 1G) > vfs.zfs.arc_min: 53552873472 ==>> auto-tuned > > Which other tuneables do I need to set on /boot/loader.conf so we can > boot the machine with vm.kmem_size> 400G. As I don't know which part > of the boot-up process is failing with vm.kmem_size/_max set to 450G > or 500G, I have no idea which to tune next. Your objective should be to reduce the value of "sysctl vfs.maxbufspace". You can do this by setting the loader.conf tuneable "kern.maxbcache" to the desired value. What does your machine currently report for "sysctl vfs.maxbufspace"?