Date: Thu, 23 Aug 2012 14:02:53 -0500 From: Alan Cox <alc@rice.edu> To: =?ISO-8859-1?Q?=22Gezeala_M=2E_Bacu=F1o_II=22?= <gezeala@gmail.com> Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov <andrey@zonov.org>, kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) Message-ID: <50367E5D.1020702@rice.edu> In-Reply-To: <CAJKO3mUkjEbY=t6K5MGphMQ_myxUHnScP8gy8v3J%2BARFMf15=g@mail.gmail.com> References: <CAJKO3mU8bfn=jmWNSpvAXOR1AWyAAM0Sio1D1PnOYg8P59V9cg@mail.gmail.com> <CAGH67wS=jue7%2B92jSCyaydOLHC=hPwtndV64FVtC7nhDsPvFng@mail.gmail.com> <CAGH67wTNfW45pgJ_%2BVn_sX%2BP9M5B5wzPT9270dRmWjYF6KerrA@mail.gmail.com> <B74BE4AB-AB67-45BD-BFC3-9AE33A85751C@gmail.com> <502DEAD9.6050304@zonov.org> <CAJKO3mVWOFa9Cby_EWsf_OFHux7YBGSV7aGYSP2YANeJkqZtoQ@mail.gmail.com> <CAJKO3mU1NdkQwNSEDk3wWyLN700=dQ0_jSXt_sx-ABpywNjfsg@mail.gmail.com> <502EB081.3030801@rice.edu> <CAJKO3mWEXUvLtdSvmjgNhhyVqw4j0DuTYm9MqLd9=i9==WLAaA@mail.gmail.com> <502FE98E.40807@rice.edu> <CAJKO3mVUMRfkUpSuk0fDdnEMc3hr087iH5u8b5N60CnPs-gP1g@mail.gmail.com> <50325634.7090904@rice.edu> <CAJKO3mXPZVhLo=si%2BEoFPGD5R_m297xedRFY-0N__WOsZBaiCA@mail.gmail.com> <CAJKO3mXQ2_XrdxWgE6JRVOpMu_cEBa_=nJCxFDJ%2BJ=f5_OUsPQ@mail.gmail.com> <503418C0.5000901@rice.edu> <CAJKO3mUkjEbY=t6K5MGphMQ_myxUHnScP8gy8v3J%2BARFMf15=g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 08/22/2012 12:09, Gezeala M. Bacuņo II wrote:
> On Tue, Aug 21, 2012 at 4:24 PM, Alan Cox<alc@rice.edu> wrote:
>> On 8/20/2012 8:26 PM, Gezeala M. Bacuņo II wrote:
>>> On Mon, Aug 20, 2012 at 9:07 AM, Gezeala M. Bacuņo II<gezeala@gmail.com>
>>> wrote:
>>>> On Mon, Aug 20, 2012 at 8:22 AM, Alan Cox<alc@rice.edu> wrote:
>>>>> On 08/18/2012 19:57, Gezeala M. Bacuņo II wrote:
>>>>>> On Sat, Aug 18, 2012 at 12:14 PM, Alan Cox<alc@rice.edu> wrote:
>>>>>>> On 08/17/2012 17:08, Gezeala M. Bacuņo II wrote:
>>>>>>>> On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox<alc@rice.edu> wrote:
>>>>>>>>> vm.kmem_size controls the maximum size of the kernel's heap, i.e.,
>>>>>>>>> the
>>>>>>>>> region where the kernel's slab and malloc()-like memory allocators
>>>>>>>>> obtain
>>>>>>>>> their memory. While this heap may occupy the largest portion of the
>>>>>>>>> kernel's virtual address space, it cannot occupy the entirety of the
>>>>>>>>> address
>>>>>>>>> space. There are other things that must be given space within the
>>>>>>>>> kernel's
>>>>>>>>> address space, for example, the file system buffer map.
>>>>>>>>>
>>>>>>>>> ZFS does not, however, use the regular file system buffer cache. The
>>>>>>>>> ARC
>>>>>>>>> takes its place, and the ARC abuses the kernel's heap like nothing
>>>>>>>>> else.
>>>>>>>>> So, if you are running a machine that only makes trivial use of a
>>>>>>>>> non-ZFS
>>>>>>>>> file system, like you boot from UFS, but store all of your data in
>>>>>>>>> ZFS,
>>>>>>>>> then
>>>>>>>>> you can dramatically reduce the size of the buffer map via boot
>>>>>>>>> loader
>>>>>>>>> tuneables and proportionately increase vm.kmem_size.
>>>>>>>>>
>>>>>>>>> Any further increases in the kernel virtual address space size will,
>>>>>>>>> however, require code changes. Small changes, but changes
>>>>>>>>> nonetheless.
>>>>>>>>>
>>>>>>>>> Alan
>>>>>>>>>
>>>> <<snip>>
>>>>>>> Your objective should be to reduce the value of "sysctl
>>>>>>> vfs.maxbufspace".
>>>>>>> You can do this by setting the loader.conf tuneable "kern.maxbcache"
>>>>>>> to
>>>>>>> the
>>>>>>> desired value.
>>>>>>>
>>>>>>> What does your machine currently report for "sysctl vfs.maxbufspace"?
>>>>>>>
>>>>>> Here you go:
>>>>>> vfs.maxbufspace: 54967025664
>>>>>> kern.maxbcache: 0
>>>>>
>>>>> Try setting kern.maxbcache to two billion and adding 50 billion to the
>>>>> setting of vm.kmem_size{,_max}.
>>>>>
>>> 2 : 50 ==>> is this the ratio for further tuning
>>> kern.maxbcache:vm.kmem_size? Is kern.maxbcache also in bytes?
>>>
>> No, this is not a ratio. Yes, kern.maxbcache is in bytes. Basically, for
>> every byte that you subtract from vfs.maxbufspace, through setting
>> kern.maxbcache, you can add a byte to vm.kmem_size{,_max}.
>>
>> Alan
>>
> Great! Thanks. Are there other sysctls aside from vfs.bufspace that I
> should monitor for vfs.maxbufspace usage? I just want to make sure
> that vfs.maxbufspace is sufficient for our needs.
You might keep an eye on "sysctl vfs.bufdefragcnt". If it starts
rapidly increasing, you may want to increase vfs.maxbufspace.
Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50367E5D.1020702>