From owner-svn-src-head@FreeBSD.ORG Fri Jan 11 17:46:15 2013 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C5F1E3BF; Fri, 11 Jan 2013 17:46:15 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id 9A9E58C3; Fri, 11 Jan 2013 17:46:15 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 2DE754C0248; Fri, 11 Jan 2013 11:46:15 -0600 (CST) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 2BFD54C01C5; Fri, 11 Jan 2013 11:46:15 -0600 (CST) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id 7bGPqfKR7cmc; Fri, 11 Jan 2013 11:46:15 -0600 (CST) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id 7F13D4C0246; Fri, 11 Jan 2013 11:46:14 -0600 (CST) Message-ID: <50F04FE5.7010406@rice.edu> Date: Fri, 11 Jan 2013 11:46:13 -0600 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: "Jayachandran C." Subject: Re: svn commit: r243631 - in head/sys: kern sys References: <201211272119.qARLJxXV061083@svn.freebsd.org> <50C1BC90.90106@freebsd.org> <50C25A27.4060007@bluezbox.com> <50C26331.6030504@freebsd.org> <50C26AE9.4020600@bluezbox.com> <50C3A3D3.9000804@freebsd.org> <50C3AF72.4010902@rice.edu> <330405A1-312A-45A5-BB86-4969478D8BBD@bluezbox.com> <50D03E83.8060908@rice.edu> <50DD081E.8000409@bluezbox.com> <50EB1841.5030006@bluezbox.com> <50EB22D2.6090103@rice.edu> <50EB415F.8020405@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Andre Oppermann , Oleksandr Tymoshenko X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 17:46:15 -0000 On 01/11/2013 05:38, Jayachandran C. wrote: > On Tue, Jan 8, 2013 at 3:12 AM, Andre Oppermann wrote: >> On 07.01.2013 20:32, Alan Cox wrote: >>> On 01/07/2013 12:47, Oleksandr Tymoshenko wrote: >>>> On 12/27/2012 6:46 PM, Oleksandr Tymoshenko wrote: >>>>> On 12/18/2012 1:59 AM, Alan Cox wrote: >>>>>> On 12/17/2012 23:40, Oleksandr Tymoshenko wrote: >>>>>>> On 2012-12-08, at 1:21 PM, Alan Cox wrote: >>>>>>> >>>>>>>> On 12/08/2012 14:32, Andre Oppermann wrote: >>>>>>> .. skipped .. >>>>>>> >>>>>>>>> The trouble seems to come from NSFBUFS which is (512 + maxusers * >>>>>>>>> 16) >>>>>>>>> resulting in a kernel map of (512 + 400 * 16) * PAGE_SIZE = >>>>>>>>> 27MB. This >>>>>>>>> seem to be pushing it with the smaller ARM kmap layout. >>>>>>>>> >>>>>>>>> Does it boot and run when you set the tunable kern.ipc.nsfbufs=3500? >>>>>>>>> >>>>>>>>> ARM does have a direct map mode as well which doesn't require the >>>>>>>>> allocation >>>>>>>>> of sfbufs. I'm not sure which other problems that approach has. >>>>>>>>> >>>>>>>> Only a few (3?) platforms use it. It reduces the size of the user >>>>>>>> address space, and translation between physical addresses and >>>>>>>> direct map >>>>>>>> addresses is not computationally trivial as it is on other >>>>>>>> architectures, e.g., amd64, ia64. However, it does try to use large >>>>>>>> page mappings. >>>>>>>> >>>>>>>> >>>>>>>>> Hopefully alc@ (added to cc) can answer that and also why the >>>>>>>>> kmap of >>>>>>>>> 27MB >>>>>>>>> manages to wrench the ARM kernel. >>>>>>>>> >>>>>>>> Arm does not define caps on either the buffer map size (param.h) >>>>>>>> or the >>>>>>>> kmem map size (vmparam.h). It would probably make sense to copy >>>>>>>> these >>>>>>>> definitions from i386. >>>>>>> Adding caps didn't help. I did some digging and found out that >>>>>>> although address range >>>>>>> 0xc0000000 .. 0xffffffff is indeed valid for ARM in general actual >>>>>>> KVA space varies for >>>>>>> each specific hardware platform. This "real" KVA is defined by >>>>>>> >>>>>>> pair and ifI use them instead of >>>>>> VM_MAX_KERNEL_ADDRESS> >>>>>>> in init_param2 function my pandaboard successfully boots. Since >>>>>>> former pair is used for defining >>>>>>> kernel_map boundaries I believe it should be used for auto tuning >>>>>>> as well. >>>>>> >>>>>> That makes sense. However, "virtual_avail" isn't the start of the >>>>>> kernel address space. The kernel map always starts at >>>>>> VM_MIN_KERNEL_ADDRESS. (See kmem_init().) "virtual_avail" represents >>>>>> the next unallocated virtual address in the kernel address space at an >>>>>> early point in initialization. "virtual_avail" and "virtual_end" >>>>>> aren't >>>>>> used after that, or outside the VM system. Please use >>>>>> vm_map_min(kernel_map) and vm_map_max(kernel_map) instead. >>>>> >>>>> I checked: kernel_map is not available (NULL) at this point. So we >>>>> can't use it to >>>>> determine real KVA size. Closest thing we can get is >>>>> virtual_avail/virtual_end pair. >>>>> >>>>> Andre, could you approve attached patch for commit or suggest better >>>>> solution? >>>> >>>> Any update on this one? Can I proceed with commit? >>>> >>> Sorry, I've been away from my e-mail since the 30th, and I'm now in the >>> process of getting caught up. Give me a day or so to look at this. > I see an issue with commit on MIPS XLP platform as well. > > With 16 GB physical memory, the ncallout is calculated to be 538881 > (since it is based on maxfiles - which is now based on the physical > memory). Due to this, the callwheel allocation per cpu is 16MB > (callwheelsize is 1MB). And on a 32 CPU machine, the total allocation > for callouts comes to 32*16MB = 512MB. > > I have worked around this issue for now by increasing VM_KMEM_SIZE_MAX > (which is 200MB now) - but I think a better fix is needed for this. > MIPS should use a definition for VM_KMEM_SIZE_MAX that scales with the kernel address space size, like amd64, i386, and sparc64, and not a fixed number. I think that the following should work for both 32- and 64-bit processors: Index: mips/include/vmparam.h =================================================================== --- mips/include/vmparam.h (revision 245229) +++ mips/include/vmparam.h (working copy) @@ -130,10 +130,11 @@ #endif /* - * Ceiling on amount of kmem_map kva space. + * Ceiling on the amount of kmem_map KVA space: 40% of the entire KVA space. */ #ifndef VM_KMEM_SIZE_MAX -#define VM_KMEM_SIZE_MAX (200 * 1024 * 1024) +#define VM_KMEM_SIZE_MAX ((VM_MAX_KERNEL_ADDRESS - \ + VM_MIN_KERNEL_ADDRESS + 1) * 2 / 5) #endif /* initial pagein size of beginning of executable file */