Date: Mon, 20 Aug 2012 18:26:58 -0700 From: =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= <gezeala@gmail.com> To: Alan Cox <alc@rice.edu> Cc: alc@freebsd.org, freebsd-performance@freebsd.org, Andrey Zonov <andrey@zonov.org>, kib@freebsd.org Subject: Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB) Message-ID: <CAJKO3mXQ2_XrdxWgE6JRVOpMu_cEBa_=nJCxFDJ%2BJ=f5_OUsPQ@mail.gmail.com> In-Reply-To: <CAJKO3mXPZVhLo=si%2BEoFPGD5R_m297xedRFY-0N__WOsZBaiCA@mail.gmail.com> References: <CAJKO3mU8bfn=jmWNSpvAXOR1AWyAAM0Sio1D1PnOYg8P59V9cg@mail.gmail.com> <CAGH67wS=jue7%2B92jSCyaydOLHC=hPwtndV64FVtC7nhDsPvFng@mail.gmail.com> <CAGH67wTNfW45pgJ_%2BVn_sX%2BP9M5B5wzPT9270dRmWjYF6KerrA@mail.gmail.com> <B74BE4AB-AB67-45BD-BFC3-9AE33A85751C@gmail.com> <502DEAD9.6050304@zonov.org> <CAJKO3mVWOFa9Cby_EWsf_OFHux7YBGSV7aGYSP2YANeJkqZtoQ@mail.gmail.com> <CAJKO3mU1NdkQwNSEDk3wWyLN700=dQ0_jSXt_sx-ABpywNjfsg@mail.gmail.com> <502EB081.3030801@rice.edu> <CAJKO3mWEXUvLtdSvmjgNhhyVqw4j0DuTYm9MqLd9=i9==WLAaA@mail.gmail.com> <502FE98E.40807@rice.edu> <CAJKO3mVUMRfkUpSuk0fDdnEMc3hr087iH5u8b5N60CnPs-gP1g@mail.gmail.com> <50325634.7090904@rice.edu> <CAJKO3mXPZVhLo=si%2BEoFPGD5R_m297xedRFY-0N__WOsZBaiCA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Aug 20, 2012 at 9:07 AM, Gezeala M. Bacu=F1o II <gezeala@gmail.com>=
wrote:
> On Mon, Aug 20, 2012 at 8:22 AM, Alan Cox <alc@rice.edu> wrote:
>> On 08/18/2012 19:57, Gezeala M. Bacu=F1o II wrote:
>>>
>>> On Sat, Aug 18, 2012 at 12:14 PM, Alan Cox<alc@rice.edu> wrote:
>>>>
>>>> On 08/17/2012 17:08, Gezeala M. Bacu=F1o II wrote:
>>>>>
>>>>> On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox<alc@rice.edu> wrote:
>>>>>>
>>>>>> vm.kmem_size controls the maximum size of the kernel's heap, i.e., t=
he
>>>>>> region where the kernel's slab and malloc()-like memory allocators
>>>>>> obtain
>>>>>> their memory. While this heap may occupy the largest portion of the
>>>>>> kernel's virtual address space, it cannot occupy the entirety of the
>>>>>> address
>>>>>> space. There are other things that must be given space within the
>>>>>> kernel's
>>>>>> address space, for example, the file system buffer map.
>>>>>>
>>>>>> ZFS does not, however, use the regular file system buffer cache. The
>>>>>> ARC
>>>>>> takes its place, and the ARC abuses the kernel's heap like nothing
>>>>>> else.
>>>>>> So, if you are running a machine that only makes trivial use of a
>>>>>> non-ZFS
>>>>>> file system, like you boot from UFS, but store all of your data in Z=
FS,
>>>>>> then
>>>>>> you can dramatically reduce the size of the buffer map via boot load=
er
>>>>>> tuneables and proportionately increase vm.kmem_size.
>>>>>>
>>>>>> Any further increases in the kernel virtual address space size will,
>>>>>> however, require code changes. Small changes, but changes nonethele=
ss.
>>>>>>
>>>>>> Alan
>>>>>>
> <<snip>>
>>>> Your objective should be to reduce the value of "sysctl vfs.maxbufspac=
e".
>>>> You can do this by setting the loader.conf tuneable "kern.maxbcache" t=
o
>>>> the
>>>> desired value.
>>>>
>>>> What does your machine currently report for "sysctl vfs.maxbufspace"?
>>>>
>>> Here you go:
>>> vfs.maxbufspace: 54967025664
>>> kern.maxbcache: 0
>>
>>
>> Try setting kern.maxbcache to two billion and adding 50 billion to the
>> setting of vm.kmem_size{,_max}.
>>
2 : 50 =3D=3D>> is this the ratio for further tuning
kern.maxbcache:vm.kmem_size? Is kern.maxbcache also in bytes?
>
> Thank you. We'll try this and post back results.
>
>>> Other (probably) relevant values:
>>> vfs.hirunningspace: 16777216
>>> vfs.lorunningspace: 11206656
>>> vfs.bufdefragcnt: 0
>>> vfs.buffreekvacnt: 2
>>> vfs.bufreusecnt: 320149
>>> vfs.hibufspace: 54966370304
>>> vfs.lobufspace: 54966304768
>>> vfs.maxmallocbufspace: 2748318515
>>> vfs.bufmallocspace: 0
>>> vfs.bufspace: 10490478592
>>> vfs.runningbufspace: 0
>>>
>>> Let me know if you need other tuneables or sysctl values. Thanks a lot
>>> for looking into this.
>>>
>>
Setting the following as instructed, machine started successfully with
446GB for vm.kmem_size/_max.
kern.maxbcache: 2000000000
vm.kmem_size_max: 479496729600
vm.kmem_size: 479496729600
## auto-tuned
vfs.maxbufspace: 1999994880
...
vfs.hirunningspace: 16777216
vfs.lorunningspace: 11206656
vfs.bufdefragcnt: 0
vfs.buffreekvacnt: 2
vfs.bufreusecnt: 11511
vfs.hibufspace: 1999339520
vfs.lobufspace: 1999273984
vfs.maxmallocbufspace: 99966976
vfs.bufmallocspace: 0
vfs.bufspace: 377028608
vfs.runningbufspace: 0
## additional manual tuning
vfs.zfs.arc_max: 455521893120
vfs.zfs.arc_min: 227760946560
kern.ipc.semmni: 256
kern.ipc.semmns: 512
kern.ipc.semmnu: 256
kern.ipc.shm_use_phys: 1
kern.ipc.shmmax: 24000000000
kern.ipc.shmall: 5859375
kern.ipc.nmbclusters: 2560000
kern.maxfiles: 197248
We'll do some further tests and report back if there are any issues.
Thanks a lot!!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJKO3mXQ2_XrdxWgE6JRVOpMu_cEBa_=nJCxFDJ%2BJ=f5_OUsPQ>