Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 17 Aug 2012 13:16:13 -0700
From:      =?ISO-8859-1?Q?Gezeala_M=2E_Bacu=F1o_II?= <gezeala@gmail.com>
To:        Andrey Zonov <andrey@zonov.org>, alc@freebsd.org, kib@freebsd.org
Cc:        freebsd-performance@freebsd.org
Subject:   Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB)
Message-ID:  <CAJKO3mU1NdkQwNSEDk3wWyLN700=dQ0_jSXt_sx-ABpywNjfsg@mail.gmail.com>
In-Reply-To: <CAJKO3mVWOFa9Cby_EWsf_OFHux7YBGSV7aGYSP2YANeJkqZtoQ@mail.gmail.com>
References:  <CAJKO3mU8bfn=jmWNSpvAXOR1AWyAAM0Sio1D1PnOYg8P59V9cg@mail.gmail.com> <CAGH67wS=jue7%2B92jSCyaydOLHC=hPwtndV64FVtC7nhDsPvFng@mail.gmail.com> <CAGH67wTNfW45pgJ_%2BVn_sX%2BP9M5B5wzPT9270dRmWjYF6KerrA@mail.gmail.com> <B74BE4AB-AB67-45BD-BFC3-9AE33A85751C@gmail.com> <502DEAD9.6050304@zonov.org> <CAJKO3mVWOFa9Cby_EWsf_OFHux7YBGSV7aGYSP2YANeJkqZtoQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Aug 17, 2012 at 7:38 AM, Gezeala M. Bacu=F1o II <gezeala@gmail.com>=
 wrote:
> On Thu, Aug 16, 2012 at 11:55 PM, Andrey Zonov <andrey@zonov.org> wrote:
>> On 8/17/12 7:15 AM, Marie Bacuno II wrote:
>>>
>>>
>>> On Aug 16, 2012, at 18:47, Garrett Cooper <yanegomi@gmail.com> wrote:
>>>
>>>> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper <yanegomi@gmail.com>
>>>> wrote:
>>>>>
>>>>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacu=F1o II
>>>>> <gezeala@gmail.com> wrote:
>>>>>>
>>>>>> Hello fellow listers,
>>>>>>
>>>>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not
>>>>>> being auto-tuned to use >329853485875 (~307GB).
>>>>>>
>>>>>> On this machine vm.kmem_size is equal to vm.kmem_size_max
>>>>>>
>>>>>> # from sysctl
>>>>>> vm.kmem_size_max: 329853485875
>>>>>> vm.kmem_size: 329853485875
>>>>>>
>>>>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_m=
ax
>>>>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I
>>>>>> manually set it in /boot/loader.conf.
>>>>>>
>>>>>> But on the machine with 512GB of RAM it just resets. For the machine
>>>>>> to boot, we need to go to the loader prompt and issue:
>>>>>>
>>>>>> OK set vm.kmem_size_max=3D"300G"
>>>>>> OK boot
>>>>>>
>>>>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have,
>>>>>> vm.kmem_size_max is always set to 329853485875.
>>>>>>
>>>>>> How can I increase vm.kmem_size_max to use at least 500GB? And how i=
s
>>>>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_m=
ax
>>>>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say
>>>>>> 490GB.
>>>>>>
>>>>>> I'm browsing thru the source code at
>>>>>> http://fxr.watson.org/fxr/search?v=3DFREEBSD9&string=3Dvm.kmem_size_=
max
>>>>>> and I'm still trying to make sense of how vm.kmem_size_max is
>>>>>> computed.
>>>>>>
>>>>>> I have posted the same topic on forums.freebsd.org but I'm not getti=
ng
>>>>>> any recommendations.
>>>>>>
>>>>>> Please see the link for additional details:
>>>>>> http://forums.freebsd.org/showthread.php?t=3D33977
>>>>>
>>>>>
>>>>> Have you tried defining VM_KMEM_SIZE_MAX to your target value?
>>>>>
>>>>> Its architecture specific BTW... see
>>>>> sys/<architecture>/include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`.
>>>>
>>>>
>>>> Also, it's a tunable, not a sysctl... so you need to set the value in
>>>> /boot/loader.conf .
>>>> -Garrett
>>>
>>>
>>> Thanks for the quick reply.
>>>
>>> Yes, had it set on /boot/loader.conf and by trial and error on the load=
er
>>> prompt.
>>>
>>> We were able to bump it to 400G successfully. Tried 500G and 450G and t=
he
>>> machine just spews out garbage in the screen.
>>>
>>> The latest output from "zfs-stats -a" with vm.kmem_size_max=3D"400G" is=
 in
>>> the forum: http://forums.freebsd.org/showthread.php?t=3D33977
>>>
>>> About the code, I am looking into amd64 arch. Still checking the values=
 of
>>> the variables.. Can't just retrieve them using getconf. If you can poin=
t me
>>> to a doxygen like documentation appreciate it a lot.
>>>
>>> Where does the constant value 329853485875 came from?
>>>
>>
>> It comes from this macro:
>>
>> #define VM_KMEM_SIZE_MAX        ((VM_MAX_KERNEL_ADDRESS - \
>>     VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5)
>>
>> ((1<<39) * 3 / 5) =3D 329853488332
>>
>> AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb.  May be it's time to
>> increase it again.  I would asked kib@ or alc@ about that.
>>
>> --
>> Andrey Zonov
>

Thanks! That's great (great for deriving 512GB) but looks like bad
news for us, we've really hit some limits there (FreeBSD auto-tuning
wise). As I've stated above, we have tried setting vm.kmem_size_max to
500GB/450GB unsuccessfully so there may be some part of the code
that's breaking. Is there any thread or discussion where you can point
me as to why they used only 60%(hard coded) of 512GB?

Some relevant codes I've gathered on this machine:

/usr/src/sys/amd64/include/vmparam.h
/*
 * Virtual addresses of things.  Derived from the page directory and
 * page table indexes from pmap.h for precision.
 *
 * 0x0000000000000000 - 0x00007fffffffffff   user map
 * 0x0000800000000000 - 0xffff7fffffffffff   does not exist (hole)
 * 0xffff800000000000 - 0xffff804020100fff   recursive page table (512GB sl=
ot)
 * 0xffff804020101000 - 0xfffffdffffffffff   unused
 * 0xfffffe0000000000 - 0xfffffeffffffffff   1TB direct map
 * 0xffffff0000000000 - 0xffffff7fffffffff   unused
 * 0xffffff8000000000 - 0xffffffffffffffff   512GB kernel map
 *
 * Within the kernel map:
 *
 * 0xffffffff80000000                        KERNBASE
 */

#define VM_MAX_KERNEL_ADDRESS   KVADDR(KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-=
1)
#define VM_MIN_KERNEL_ADDRESS   KVADDR(KPML4I, NPDPEPG-512, 0, 0)

/usr/src/sys/amd64/include/pmap.h
/*
 * Pte related macros.  This is complicated by having to deal with
 * the sign extension of the 48th bit.
 */
#define KVADDR(l4, l3, l2, l1) ( \
        ((unsigned long)-1 << 47) | \
        ((unsigned long)(l4) << PML4SHIFT) | \
        ((unsigned long)(l3) << PDPSHIFT) | \
        ((unsigned long)(l2) << PDRSHIFT) | \
        ((unsigned long)(l1) << PAGE_SHIFT))

/usr/src/sys/amd64/include/param.h
#define PAGE_SHIFT      12              /* LOG2(PAGE_SIZE) */
#define PDRSHIFT        21              /* LOG2(NBPDR) */
#define PDPSHIFT        30              /* LOG2(NBPDP) */
#define PML4SHIFT       39              /* LOG2(NBPML4) */

Yet to derive: KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-1. Checking pmap.c,
vm_machdep.c etc.

Additional Info:
1] Installed using PCBSD-9 Release amd64.

2] uname -a
FreeBSD fmt-iscsi-stg1.musicreports.com 9.0-RELEASE FreeBSD
9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011
root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-source/9.=
0/sys/GENERIC
 amd64

3] first few lines from /var/run/dmesg.boot:
FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011
    root@build9x64.pcbsd.org:/usr/obj/builds/amd64/pcbsd-build90/fbsd-sourc=
e/9.0/sys/GENERIC
amd64
CPU: Intel(R) Xeon(R) CPU E7- 8837  @ 2.67GHz (2666.82-MHz K8-class CPU)
  Origin =3D "GenuineIntel"  Id =3D 0x206f2  Family =3D 6  Model =3D 2f  St=
epping =3D 2
  Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG=
E,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=3D0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SS=
SE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
  AMD Features=3D0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=3D0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  =3D 549755813888 (524288 MB)
avail memory =3D 530339893248 (505771 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs
FreeBSD/SMP: 8 package(s) x 8 core(s)

4] relevant sysctl's with manual tuning:
kern.maxusers: 384
kern.maxvnodes: 8222162
vfs.numvnodes: 675740
vfs.freevnodes: 417524
kern.ipc.somaxconn: 128
kern.openfiles: 5238
vfs.zfs.arc_max: 428422987776
vfs.zfs.arc_min: 53552873472
vfs.zfs.arc_meta_used: 3167391088
vfs.zfs.arc_meta_limit: 107105746944
vm.kmem_size_max: 429496729600    =3D=3D>> manually tuned
vm.kmem_size: 429496729600    =3D=3D>> manually tuned
vm.kmem_map_free: 107374727168
vm.kmem_map_size: 144625156096
vfs.wantfreevnodes: 2055540
kern.minvnodes: 2055540
kern.maxfiles: 197248    =3D=3D>> manually tuned
vm.vmtotal:
System wide totals computed every five seconds: (values in kilobytes)
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Processes:              (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 150)
Virtual Memory:         (Total: 1086325716K Active: 12377876K)
Real Memory:            (Total: 144143408K Active: 803432K)
Shared Virtual Memory:  (Total: 81384K Active: 37560K)
Shared Real Memory:     (Total: 32224K Active: 27548K)
Free Memory Pages:      365565564K

hw.availpages: 134170294
hw.physmem: 549561524224
hw.usermem: 391395241984
hw.realmem: 551836188672
vm.kmem_size_scale: 1
kern.ipc.nmbclusters: 2560000    =3D=3D>> manually tuned
kern.ipc.maxsockbuf: 2097152
net.inet.tcp.sendbuf_max: 2097152
net.inet.tcp.recvbuf_max: 2097152
kern.maxfilesperproc: 18000
net.inet.ip.intr_queue_maxlen: 256
kern.maxswzone: 33554432
kern.ipc.shmmax: 10737418240    =3D=3D>> manually tuned
kern.ipc.shmall: 2621440    =3D=3D>> manually tuned
vfs.zfs.write_limit_override: 0
vfs.zfs.prefetch_disable: 0
hw.pagesize: 4096
hw.availpages: 134170294
kern.ipc.maxpipekva: 8586895360
kern.ipc.shm_use_phys: 1    =3D=3D>> manually tuned
vfs.vmiodirenable: 1
debug.numcache: 632148
vfs.ncsizefactor: 2
vm.kvm_size: 549755809792
vm.kvm_free: 54456741888
kern.ipc.semmni: 256
kern.ipc.semmns: 512
kern.ipc.semmnu: 256


Thanks!



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJKO3mU1NdkQwNSEDk3wWyLN700=dQ0_jSXt_sx-ABpywNjfsg>