From owner-freebsd-current@FreeBSD.ORG Mon May 18 03:17:02 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 25DEC9C3; Mon, 18 May 2015 03:17:02 +0000 (UTC) Received: from mail-yk0-x22e.google.com (mail-yk0-x22e.google.com [IPv6:2607:f8b0:4002:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D448B1307; Mon, 18 May 2015 03:17:01 +0000 (UTC) Received: by ykec202 with SMTP id c202so49729847yke.2; Sun, 17 May 2015 20:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=4YjfmzHYtiNQot5Mxy7rZBeO3KVHtTWKMCPiU4MYCMY=; b=SVXu75YWc04NxU2yhB6Z9l0MSPnrdrgH9s8aT9fZM7/l8yVwaL5N4NkyifBupWMDvi cy2PrcTJwT/0m5Yy8TlH3mllaZ0AVfyyZkUPdDoExdDzsgQ024ZGmS/Ch1Nh7abx+6ps IC3A7wnBlrnAyi92cqm5ZhAFClt9+NMHh2RMnk/PTsg7xEx96eFI8LRwMcOQ1Z6FwkdY TYqbOsIW8tLHIwI2lwV3jS3wlqztjCLy7JIQmnN7BJX8BdGK0/7ZLswgZEQ98CWZxUSo d+qR+7wOifg/sLIfgSti5R5Mta6/Rlz1qkD7sK6pNVNTQ8u/Hh7xdRF+MLWQCd5q+7S+ 3iUA== MIME-Version: 1.0 X-Received: by 10.170.46.203 with SMTP id 194mr22624178yko.51.1431919020923; Sun, 17 May 2015 20:17:00 -0700 (PDT) Sender: pkelsey@gmail.com Received: by 10.13.201.71 with HTTP; Sun, 17 May 2015 20:17:00 -0700 (PDT) In-Reply-To: References: <20150319095306.C7719@admin16.site.uottawa.ca> <528B3B45-378E-44CD-A286-0269E84C4F70@me.com> <54997.66.186.85.232.1427195954.squirrel@courriel.site.uottawa.ca> <1A6D6059-21A2-467C-979D-E413623753FD@me.com> <20150324155426.M7719@admin16.site.uottawa.ca> Date: Sun, 17 May 2015 23:17:00 -0400 X-Google-Sender-Auth: g9dKZ9_W6Bb6rBmRNoF_0Cv8T7U Message-ID: Subject: Re: panic: UMA: Increase vm.boot_pages on Dell R920 r279210 From: Patrick Kelsey To: Adrian Chadd Cc: Keith White , Rui Paulo , freebsd-current Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 03:17:02 -0000 On Sat, May 2, 2015 at 10:25 PM, Adrian Chadd wrote: > hi, > > Hm, should we be upping this limit automatically? Can we get cpu > counts or memory amount early enough in boot to have a hope of > auto-tuning? > > 64 seems low, 1024 seems high as a default. :) > What is it that's exhausting the boot_pages? I'm semi-guessing it's the number of vm radix tree nodes needed for the TiB of memory. The only thing I'm aware of (allow for ignorance here) that consumes boot_pages and scales with the cpu count is the uma zone used for uma cache objects, but on amd64 this zone only needs 640 + cpus * 128 bytes, or about 4 pages for 120 cpus. vm radix nodes are 144 bytes each on amd64, and by my back-of-the-envelope calculations (using traces of non-vm-radix boot_page use from another amd64 system), 64 boot_pages would be exhausted after about 1000 vm radix nodes were allocated. It would be interesting to know how many boot_pages were actually required for this particular system. In any event, since startup_alloc() is designed to exhaust all the boot_pages before switching to the normal allocators, it doesn't seem necessarily harmful to err on the high side either in bumping up the static default or introducing an auto-tuned value (provided the excess is not so perversely large that startup_alloc() isn't able to make use of an embarrassment of pages due to zone creation timing and usage patterns). We know the number of cpus at the time boot_pages is put to use, but I don't think we know how much memory there is (and even less sure that even if we did, we'd really want to try to estimate things the vm radix tree size in a generic way). Something like a default of boot_pages = max(64, 32 + k * cpus) might be sufficient for k = 4 or 8 (gathering some data points would give a clue here), and palatable since it is at a minimum the current value that's been in use, and at the other end approaches a modest commitment of 16 or 32 KiB per cpu in the worst case (unused and unreclaimed boot_pages with high cpu count). -Patrick > > > On 24 March 2015 at 13:00, Keith White wrote: > > On Tue, 24 Mar 2015, Rui Paulo wrote: > > > >> On Mar 24, 2015, at 04:19, kwhite@site.uottawa.ca wrote: > >>> > >>> > >>> I'm using /boot/loader.conf. Is there another place I should be doing > >>> this? > >> > >> > >> No, that's correct, but apparently there's a problem: the RDTUN sysctl > is > >> not picked up early enough. Can you try this patch? I haven't really > >> tested it. :-) > >> > >> diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c > >> index 79665ba..a764788 100644 > >> --- a/sys/vm/vm_page.c > >> +++ b/sys/vm/vm_page.c > >> @@ -134,8 +134,9 @@ long first_page; > >> int vm_page_zero_count; > >> > >> static int boot_pages = UMA_BOOT_PAGES; > >> -SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN, &boot_pages, 0, > >> - "number of pages allocated for bootstrapping the VM system"); > >> +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, > >> + &boot_pages, 0, > >> + "number of pages allocated for bootstrapping the VM system"); > >> > >> static int pa_tryrelock_restart; > >> SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD, > >> @@ -349,6 +350,7 @@ vm_page_startup(vm_offset_t vaddr) > >> * Allocate memory for use when boot strapping the kernel memory > >> * allocator. > >> */ > >> + TUNABLE_INT_FETCH("vm.boot_pages", &boot_pages); > >> new_end = end - (boot_pages * UMA_SLAB_SIZE); > >> new_end = trunc_page(new_end); > >> mapped = pmap_map(&vaddr, new_end, end, > >> @@ -443,7 +445,7 @@ vm_page_startup(vm_offset_t vaddr) > >> > >> > >> -- > >> Rui Paulo > > > > > > Patch tried. Success! > > > > I now get this after setting vm.boot_pages=1024 in /boot/loader.conf: > > > > Booting... > > GDB: no debug ports present > > KDB: debugger backends: ddb > > KDB: current backend: ddb > > Copyright (c) 1992-2015 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 11.0-CURRENT #1: Tue Mar 24 13:44:48 UTC 2015 > > root@:/usr/obj/usr/src/sys/GENERIC amd64 > > FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115 > > WARNING: WITNESS option enabled, expect reduced performance. > > UMA startup boot_pages: 1024 > > ... > > > > And can start all 120 processors. > > > > Thanks! > > > > ...keith > > -- > > Keith White, genie.uottawa.ca engineering.uottawa.ca > > kwhite@uottawa.ca [+1 613 562 5800 x6681] > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >