From owner-svn-src-all@FreeBSD.ORG Fri Jan 11 11:38:31 2013 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E2A48733; Fri, 11 Jan 2013 11:38:31 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-ob0-f175.google.com (mail-ob0-f175.google.com [209.85.214.175]) by mx1.freebsd.org (Postfix) with ESMTP id 7D3696B5; Fri, 11 Jan 2013 11:38:31 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id vb8so1599687obc.6 for ; Fri, 11 Jan 2013 03:38:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=p8JGpA2Wc8209lV+oQ/pMxbtiKwp7zgNtPP29C0HGd8=; b=sp47VUmgw0E4VJp2um89ysaLkq0N3rRYPaSFQqZG7cwj4NcZ+s7dXn41qhpIV6XJzG atgEvExd59DwQcVQe/yYdN9yB1FizAuUTbrScodvs+DTMFZW/qwligUG/reoG/Aa3Mjz XafCv89VpRPCmPouzwSGCc9AOTOyQ2iel+ozs8dcGnkF1BtYGd5TeOPBh+1KdYLLBV2i 84JfOFVtV79ccgaUrCsY8OISXsPxRh/JlYYggXKxATv7MPdh3TQIbf8bPEgFEmn1N6DL PNWoYw3ug2mf+58z6cghSoLGYIaJhU8geq7eLiQacM+Mfe+L5VBY2xI16I+4AwTgJ5la 2WVg== MIME-Version: 1.0 Received: by 10.182.38.69 with SMTP id e5mr54739976obk.79.1357904304943; Fri, 11 Jan 2013 03:38:24 -0800 (PST) Sender: c.jayachandran@gmail.com Received: by 10.182.111.197 with HTTP; Fri, 11 Jan 2013 03:38:24 -0800 (PST) In-Reply-To: <50EB415F.8020405@freebsd.org> References: <201211272119.qARLJxXV061083@svn.freebsd.org> <50C1BC90.90106@freebsd.org> <50C25A27.4060007@bluezbox.com> <50C26331.6030504@freebsd.org> <50C26AE9.4020600@bluezbox.com> <50C3A3D3.9000804@freebsd.org> <50C3AF72.4010902@rice.edu> <330405A1-312A-45A5-BB86-4969478D8BBD@bluezbox.com> <50D03E83.8060908@rice.edu> <50DD081E.8000409@bluezbox.com> <50EB1841.5030006@bluezbox.com> <50EB22D2.6090103@rice.edu> <50EB415F.8020405@freebsd.org> Date: Fri, 11 Jan 2013 17:08:24 +0530 X-Google-Sender-Auth: 4sQ3DDJKmx1hTtTZNjfhKtZ1XiA Message-ID: Subject: Re: svn commit: r243631 - in head/sys: kern sys From: "Jayachandran C." To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Oleksandr Tymoshenko , Alan Cox X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 11:38:32 -0000 On Tue, Jan 8, 2013 at 3:12 AM, Andre Oppermann wrote: > On 07.01.2013 20:32, Alan Cox wrote: >> >> On 01/07/2013 12:47, Oleksandr Tymoshenko wrote: >>> >>> On 12/27/2012 6:46 PM, Oleksandr Tymoshenko wrote: >>>> >>>> On 12/18/2012 1:59 AM, Alan Cox wrote: >>>>> >>>>> On 12/17/2012 23:40, Oleksandr Tymoshenko wrote: >>>>>> >>>>>> On 2012-12-08, at 1:21 PM, Alan Cox wrote: >>>>>> >>>>>>> On 12/08/2012 14:32, Andre Oppermann wrote: >>>>>> >>>>>> .. skipped .. >>>>>> >>>>>>>> The trouble seems to come from NSFBUFS which is (512 + maxusers * >>>>>>>> 16) >>>>>>>> resulting in a kernel map of (512 + 400 * 16) * PAGE_SIZE = >>>>>>>> 27MB. This >>>>>>>> seem to be pushing it with the smaller ARM kmap layout. >>>>>>>> >>>>>>>> Does it boot and run when you set the tunable kern.ipc.nsfbufs=3500? >>>>>>>> >>>>>>>> ARM does have a direct map mode as well which doesn't require the >>>>>>>> allocation >>>>>>>> of sfbufs. I'm not sure which other problems that approach has. >>>>>>>> >>>>>>> Only a few (3?) platforms use it. It reduces the size of the user >>>>>>> address space, and translation between physical addresses and >>>>>>> direct map >>>>>>> addresses is not computationally trivial as it is on other >>>>>>> architectures, e.g., amd64, ia64. However, it does try to use large >>>>>>> page mappings. >>>>>>> >>>>>>> >>>>>>>> Hopefully alc@ (added to cc) can answer that and also why the >>>>>>>> kmap of >>>>>>>> 27MB >>>>>>>> manages to wrench the ARM kernel. >>>>>>>> >>>>>>> Arm does not define caps on either the buffer map size (param.h) >>>>>>> or the >>>>>>> kmem map size (vmparam.h). It would probably make sense to copy >>>>>>> these >>>>>>> definitions from i386. >>>>>> >>>>>> Adding caps didn't help. I did some digging and found out that >>>>>> although address range >>>>>> 0xc0000000 .. 0xffffffff is indeed valid for ARM in general actual >>>>>> KVA space varies for >>>>>> each specific hardware platform. This "real" KVA is defined by >>>>>> >>>>>> pair and ifI use them instead of >>>>> VM_MAX_KERNEL_ADDRESS> >>>>>> in init_param2 function my pandaboard successfully boots. Since >>>>>> former pair is used for defining >>>>>> kernel_map boundaries I believe it should be used for auto tuning >>>>>> as well. >>>>> >>>>> >>>>> That makes sense. However, "virtual_avail" isn't the start of the >>>>> kernel address space. The kernel map always starts at >>>>> VM_MIN_KERNEL_ADDRESS. (See kmem_init().) "virtual_avail" represents >>>>> the next unallocated virtual address in the kernel address space at an >>>>> early point in initialization. "virtual_avail" and "virtual_end" >>>>> aren't >>>>> used after that, or outside the VM system. Please use >>>>> vm_map_min(kernel_map) and vm_map_max(kernel_map) instead. >>>> >>>> >>>> I checked: kernel_map is not available (NULL) at this point. So we >>>> can't use it to >>>> determine real KVA size. Closest thing we can get is >>>> virtual_avail/virtual_end pair. >>>> >>>> Andre, could you approve attached patch for commit or suggest better >>>> solution? >>> >>> >>> Any update on this one? Can I proceed with commit? >>> >> >> Sorry, I've been away from my e-mail since the 30th, and I'm now in the >> process of getting caught up. Give me a day or so to look at this. I see an issue with commit on MIPS XLP platform as well. With 16 GB physical memory, the ncallout is calculated to be 538881 (since it is based on maxfiles - which is now based on the physical memory). Due to this, the callwheel allocation per cpu is 16MB (callwheelsize is 1MB). And on a 32 CPU machine, the total allocation for callouts comes to 32*16MB = 512MB. I have worked around this issue for now by increasing VM_KMEM_SIZE_MAX (which is 200MB now) - but I think a better fix is needed for this. JC.