Date: Wed, 23 Jan 2013 09:10:38 +0100 From: Andre Oppermann <andre@freebsd.org> To: Artem Belevich <art@freebsd.org> Cc: Matthew Fleming <mdf@freebsd.org>, FreeBSD Current <freebsd-current@freebsd.org>, freebsd-hackers <freebsd-hackers@freebsd.org> Subject: Re: kmem_map auto-sizing and size dependencies Message-ID: <50FF9AFE.9000406@freebsd.org> In-Reply-To: <CAFqOu6jO1iMM=gS5-x%2B_91JfCF3zdLW-Dfnvja2QozDWUV%2BkTA@mail.gmail.com> References: <50F96A67.9080203@freebsd.org> <CAMBSHm-xPo7bH556h4QwtHhCUHX%2B4g_pv3=t29C1SLD4VnBWsQ@mail.gmail.com> <20130121210645.GC1341@garage.freebsd.pl> <CAFqOu6jO1iMM=gS5-x%2B_91JfCF3zdLW-Dfnvja2QozDWUV%2BkTA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23.01.2013 00:22, Artem Belevich wrote: > On Mon, Jan 21, 2013 at 1:06 PM, Pawel Jakub Dawidek <pjd@freebsd.org> wrote: >> On Fri, Jan 18, 2013 at 08:26:04AM -0800, mdf@FreeBSD.org wrote: >>>> Should it be set to a larger initial value based on min(physical,KVM) space >>>> available? >>> >>> It needs to be smaller than the physical space, [...] >> >> Or larger, as the address space can get fragmented and you might not be >> able to allocate memory even if you have physical pages available. > > +1 for relaxing upper limit. > > I routinely patch all my systems that use ZFS to allow kmem_map size > to be larger than physical memory. Otherwise on a system where most of > RAM goes towards ZFS ARC I used to eventually run into dreaded > kmem_map too small panic. During startup and VM initialization the following kernel VM maps are created: kernel_map (parent) specifying the entire kernel virtual address space. It is 512GB on amd64 currently. Out of the kernel_map a number of sub-maps are created: clean_map which isn't referenced anywhere else buffer_map used in vfs_bio.c for i/o buffers pager_map used in vm_page.c for paging exec_map used in kern/kern_exec.c and other places for program startup pipe_map used in kern/sys_pipe.c for pipe buffering kmem_map used in kern/kern_malloc. and vm/uma_core.c among other places and provides all kernel malloc and UMA zone memory allocations. Having the kernel occupy all of physical RAM eventually isn't pretty. So the problem you're describing is that even though enough kernel_map space is still available it is too fragmented to find a sufficiently large chunk. If the kmem_map is larger than the available physical memory another mechanism has to track and limit its physical memory consumption. This may become a SMP bottleneck due to synchronization issues. I haven't looked how the maps are managed internally. Maybe there is a natural hook to attach such a mechanism and to allow the sub-maps to be larger in kVM space than physical memory. Maybe ZFS then can have its own sub-map for ARC too. -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50FF9AFE.9000406>