Date: Tue, 13 Aug 2013 18:03:59 -0700 From: Jim Harris <jim.harris@gmail.com> To: Jeff Roberson <jroberson@jroberson.net> Cc: Alan Cox <alc@freebsd.org>, Attilio Rao <attilio@freebsd.org>, FreeBSD current <freebsd-current@freebsd.org>, Konstantin Belousov <kib@freebsd.org>, Colin Percival <cperciva@freebsd.org> Subject: Re: panic: UMA: Increase vm.boot_pages with 32 CPUs Message-ID: <CAJP=Hc9KZfD-uKkOM%2BO=oAHiGj3vNwRa-fMgtKWEacOKL110DQ@mail.gmail.com> In-Reply-To: <alpine.BSF.2.00.1308131203350.4083@desktop> References: <5208A488.2050603@freebsd.org> <alpine.BSF.2.00.1308131203350.4083@desktop>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Aug 13, 2013 at 3:05 PM, Jeff Roberson <jroberson@jroberson.net>wrote: > On Mon, 12 Aug 2013, Colin Percival wrote: > > Hi all, >> >> A HEAD@254238 kernel fails to boot in EC2 with >> >>> panic: UMA: Increase vm.boot_pages >>> >> on 32-CPU instances. Instances with up to 16 CPUs boot fine. >> >> I know there has been some mucking about with VM recently -- anyone want >> to claim this, or should I start doing a binary search? >> > > It's not any one commit really, just creeping demand for more pages before > the VM can get started. I would suggest making boot pages scale with > MAXCPU. Or just raising it as the panic suggests. We could rewrite the > way that the vm gets these early pages but it's a lot of work and typically > people just bump it and forget about it. > > I ran into this problem today when enabling hyperthreading on my dual-socket Xeon E5 system. It looks like r254025 is actually the culprit. Specifically, the new mallocinit()/kmeminit() now invoke the new vmem_init() before uma_startup2(), which allocates 16 zones out of the boot pages if I am reading this correctly. This is all done before uma_startup2() is called, triggering the panic. Anything less than 28 CPUs, and the zone size (uma_zone + uma_cache * (mp_maxid + 1)) is <= PAGE_SIZE and we can successfully boot. So at 32 CPUs, we need two boot pages per zone which consumes more than the default 64 boot pages. The size of these structures do not appear to have materially changed any time recently. Scaling with MAXCPU seems to be an OK solution, but should it be based directly on the size of (uma_zone + uma_cache * MAXCPU)? I am not very familiar with uma startup, but it seems like these zones are the primary consumers of the boot pages, so the UMA_BOOT_PAGES default should be based directly on that size.. Regards, -Jim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJP=Hc9KZfD-uKkOM%2BO=oAHiGj3vNwRa-fMgtKWEacOKL110DQ>