From owner-svn-src-all@FreeBSD.ORG Fri Dec 28 02:47:02 2012 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 41166856; Fri, 28 Dec 2012 02:47:02 +0000 (UTC) (envelope-from gonzo@id.bluezbox.com) Received: from id.bluezbox.com (id.bluezbox.com [88.198.91.248]) by mx1.freebsd.org (Postfix) with ESMTP id 997CE8FC12; Fri, 28 Dec 2012 02:47:01 +0000 (UTC) Received: from [88.198.91.248] (helo=[IPv6:::1]) by id.bluezbox.com with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1ToPyM-0004fa-F0; Thu, 27 Dec 2012 18:47:00 -0800 Message-ID: <50DD081E.8000409@bluezbox.com> Date: Thu, 27 Dec 2012 18:46:54 -0800 From: Oleksandr Tymoshenko User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Alan Cox Subject: Re: svn commit: r243631 - in head/sys: kern sys References: <201211272119.qARLJxXV061083@svn.freebsd.org> <50C1BC90.90106@freebsd.org> <50C25A27.4060007@bluezbox.com> <50C26331.6030504@freebsd.org> <50C26AE9.4020600@bluezbox.com> <50C3A3D3.9000804@freebsd.org> <50C3AF72.4010902@rice.edu> <330405A1-312A-45A5-BB86-4969478D8BBD@bluezbox.com> <50D03E83.8060908@rice.edu> In-Reply-To: <50D03E83.8060908@rice.edu> Content-Type: multipart/mixed; boundary="------------000507020005010409010409" Sender: gonzo@id.bluezbox.com X-Spam-Level: -- X-Spam-Report: Spam detection software, running on the system "id.bluezbox.com", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see The administrator of that system for details. Content preview: On 12/18/2012 1:59 AM, Alan Cox wrote: > On 12/17/2012 23:40, Oleksandr Tymoshenko wrote: >> On 2012-12-08, at 1:21 PM, Alan Cox wrote: >> >>> On 12/08/2012 14:32, Andre Oppermann wrote: >> .. skipped .. >> >>>> The trouble seems to come from NSFBUFS which is (512 + maxusers * 16) >>>> resulting in a kernel map of (512 + 400 * 16) * PAGE_SIZE = 27MB. This >>>> seem to be pushing it with the smaller ARM kmap layout. >>>> >>>> Does it boot and run when you set the tunable kern.ipc.nsfbufs=3500? >>>> >>>> ARM does have a direct map mode as well which doesn't require the >>>> allocation >>>> of sfbufs. I'm not sure which other problems that approach has. >>>> >>> Only a few (3?) platforms use it. It reduces the size of the user >>> address space, and translation between physical addresses and direct map >>> addresses is not computationally trivial as it is on other >>> architectures, e.g., amd64, ia64. However, it does try to use large >>> page mappings. >>> >>> >>>> Hopefully alc@ (added to cc) can answer that and also why the kmap of >>>> 27MB >>>> manages to wrench the ARM kernel. >>>> >>> Arm does not define caps on either the buffer map size (param.h) or the >>> kmem map size (vmparam.h). It would probably make sense to copy these >>> definitions from i386. >> Adding caps didn't help. I did some digging and found out that although address range >> 0xc0000000 .. 0xffffffff is indeed valid for ARM in general actual KVA space varies for >> each specific hardware platform. This "real" KVA is defined by >> pair and ifI use them instead of >> in init_param2 function my pandaboard successfully boots. Since former pair is used for defining >> kernel_map boundaries I believe it should be used for auto tuning as well. > > That makes sense. However, "virtual_avail" isn't the start of the > kernel address space. The kernel map always starts at > VM_MIN_KERNEL_ADDRESS. (See kmem_init().) "virtual_avail" represents > the next [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Andre Oppermann X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2012 02:47:02 -0000 This is a multi-part message in MIME format. --------------000507020005010409010409 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 12/18/2012 1:59 AM, Alan Cox wrote: > On 12/17/2012 23:40, Oleksandr Tymoshenko wrote: >> On 2012-12-08, at 1:21 PM, Alan Cox wrote: >> >>> On 12/08/2012 14:32, Andre Oppermann wrote: >> .. skipped .. >> >>>> The trouble seems to come from NSFBUFS which is (512 + maxusers * 16) >>>> resulting in a kernel map of (512 + 400 * 16) * PAGE_SIZE = 27MB. This >>>> seem to be pushing it with the smaller ARM kmap layout. >>>> >>>> Does it boot and run when you set the tunable kern.ipc.nsfbufs=3500? >>>> >>>> ARM does have a direct map mode as well which doesn't require the >>>> allocation >>>> of sfbufs. I'm not sure which other problems that approach has. >>>> >>> Only a few (3?) platforms use it. It reduces the size of the user >>> address space, and translation between physical addresses and direct map >>> addresses is not computationally trivial as it is on other >>> architectures, e.g., amd64, ia64. However, it does try to use large >>> page mappings. >>> >>> >>>> Hopefully alc@ (added to cc) can answer that and also why the kmap of >>>> 27MB >>>> manages to wrench the ARM kernel. >>>> >>> Arm does not define caps on either the buffer map size (param.h) or the >>> kmem map size (vmparam.h). It would probably make sense to copy these >>> definitions from i386. >> Adding caps didn't help. I did some digging and found out that although address range >> 0xc0000000 .. 0xffffffff is indeed valid for ARM in general actual KVA space varies for >> each specific hardware platform. This "real" KVA is defined by >> pair and ifI use them instead of >> in init_param2 function my pandaboard successfully boots. Since former pair is used for defining >> kernel_map boundaries I believe it should be used for auto tuning as well. > > That makes sense. However, "virtual_avail" isn't the start of the > kernel address space. The kernel map always starts at > VM_MIN_KERNEL_ADDRESS. (See kmem_init().) "virtual_avail" represents > the next unallocated virtual address in the kernel address space at an > early point in initialization. "virtual_avail" and "virtual_end" aren't > used after that, or outside the VM system. Please use > vm_map_min(kernel_map) and vm_map_max(kernel_map) instead. I checked: kernel_map is not available (NULL) at this point. So we can't use it to determine real KVA size. Closest thing we can get is virtual_avail/virtual_end pair. Andre, could you approve attached patch for commit or suggest better solution? > > That said, I would still add caps on the buffer map and kmem map size. > As memory sizes on arm systems grow, you'll eventually need them, just > like we did on i386. What's the names for #defines for these? --------------000507020005010409010409 Content-Type: text/plain; charset=windows-1252; name="autotune-kva-fix.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="autotune-kva-fix.diff" Index: kern/subr_param.c =================================================================== --- kern/subr_param.c (revision 244665) +++ kern/subr_param.c (working copy) @@ -333,7 +333,7 @@ * At most it can be 3/4 of available kernel memory. */ realmem = qmin((quad_t)physpages * PAGE_SIZE, - VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS); + virtual_end - virtual_avail); maxmbufmem = realmem / 2; TUNABLE_QUAD_FETCH("kern.maxmbufmem", &maxmbufmem); if (maxmbufmem > (realmem / 4) * 3) @@ -347,8 +347,8 @@ TUNABLE_LONG_FETCH("kern.ipc.maxpipekva", &maxpipekva); if (maxpipekva < 512 * 1024) maxpipekva = 512 * 1024; - if (maxpipekva > (VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS) / 64) - maxpipekva = (VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS) / + if (maxpipekva > (virtual_end - virtual_avail) / 64) + maxpipekva = (virtual_end - virtual_avail) / 64; } --------------000507020005010409010409--