From owner-freebsd-current@FreeBSD.ORG Mon Oct 18 21:01:35 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A489F1065672; Mon, 18 Oct 2010 21:01:35 +0000 (UTC) (envelope-from giovanni.trematerra@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id DB5AF8FC15; Mon, 18 Oct 2010 21:01:34 +0000 (UTC) Received: by wyb38 with SMTP id 38so1745184wyb.13 for ; Mon, 18 Oct 2010 14:01:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=Ed2N+BziV1Jpgr5HR3EZIwEFcoI+wA7N46T2tDQ7J/I=; b=Qcvr5wbl3uDS8qUyNao19IrE6QCHMoSWcz7Gri2is2KiFR+eoqEMdU9bqBjN27/stA UPzw/GfTdJy8WMBYhXDsG6RHLSG8D9uGS85js66E9i0lH2Gok7BWuxobswNu510P7DBX mi4ES7aAx5R3Np58GgGLqhZUzMO8BjoP1i5aQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=LNDco2X/bYbLCpnWP8PhsD+5GUuuKL1IBoVEa6xqZWdhGaFrnk7cFzjud/QE2Nnwbs 6tfFhi7BXx5/j99cytU2IKqvszpoh3aYNVy9b4+WYR6w0mQhiM1rIPLb6mWfe/ugOL/5 F6ImYMb71GH9idGlTEDS81gqlVqZrhY4bGxQU= MIME-Version: 1.0 Received: by 10.227.128.7 with SMTP id i7mr1415905wbs.165.1287435693667; Mon, 18 Oct 2010 14:01:33 -0700 (PDT) Sender: giovanni.trematerra@gmail.com Received: by 10.227.144.203 with HTTP; Mon, 18 Oct 2010 14:01:33 -0700 (PDT) In-Reply-To: <4CBC5719.1020807@freebsd.org> References: <4C9B9B9C.6000807@freebsd.org> <4CBBEBDF.3060905@freebsd.org> <4CBC5719.1020807@freebsd.org> Date: Mon, 18 Oct 2010 23:01:33 +0200 X-Google-Sender-Auth: sF5rgwovGJOPCPc61PIELScdm2Y Message-ID: From: Giovanni Trematerra To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: alc@freebsd.org, freebsd-current@freebsd.org Subject: Re: panic in uma_startup for many-core amd64 system X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Oct 2010 21:01:35 -0000 On Mon, Oct 18, 2010 at 4:18 PM, Andriy Gapon wrote: > on 18/10/2010 16:40 Giovanni Trematerra said the following: >> On Mon, Oct 18, 2010 at 8:40 AM, Andriy Gapon wrote: >>> on 23/09/2010 21:25 Andriy Gapon said the following: >>>> >>>> Jeff, >>>> >>>> just for the kicks I tried to emulate a machine with 64 logical CPUs u= sing >>>> qemu-devel port: >>>> qemu-system-x86_64 -smp sockets=3D4,cores=3D8,threads=3D2 ... >>>> >>>> It seems that FreeBSD agreed to recognize only first 32 CPUs, but it p= aniced anyway. >>>> >>>> Here's a backtrace: >>>> #34 0xffffffff804fe7f5 in zone_alloc_item (zone=3D0xffffffff80be1554, >>>> udata=3D0xffffffff80be1550, flags=3D1924) at /usr/src/sys/vm/uma_core.= c:2506 >>>> #35 0xffffffff804ff35d in hash_alloc (hash=3D0xffffff001ffdb030) at >>>> /usr/src/sys/vm/uma_core.c:483 >>>> #36 0xffffffff804ff642 in keg_ctor (mem=3DVariable "mem" is not availa= ble. >>>> ) at /usr/src/sys/vm/uma_core.c:1396 >>>> #37 0xffffffff804fe91b in zone_alloc_item (zone=3D0xffffffff80a1f300, >>>> udata=3D0xffffffff80be1b60, flags=3D2) at /usr/src/sys/vm/uma_core.c:2= 544 >>>> #38 0xffffffff804ff92e in zone_ctor (mem=3DVariable "mem" is not avail= able. >>>> ) at /usr/src/sys/vm/uma_core.c:1832 >>>> #39 0xffffffff804ffca4 in uma_startup (bootmem=3D0xffffff001ffac000, b= oot_pages=3D48) >>>> at /usr/src/sys/vm/uma_core.c:1741 >>>> #40 0xffffffff80514822 in vm_page_startup (vaddr=3D1844674407157681766= 4) at >>>> /usr/src/sys/vm/vm_page.c:360 >>>> #41 0xffffffff805060c5 in vm_mem_init (dummy=3DVariable "dummy" is not= available. >>>> ) at /usr/src/sys/vm/vm_init.c:118 >>>> #42 0xffffffff803258b9 in mi_startup () at /usr/src/sys/kern/init_main= .c:253 >>>> #43 0xffffffff8017177c in btext () at /usr/src/sys/amd64/amd64/locore.= S:81 >>>> [[[ >>>> Note: >>>> 1. Frame numbers are high because the backtrace is obtained via gdb re= motely >>>> connected to qemu and also there is bunch of extra frames from DDB, et= c. >>>> 2. Line numbers in uma_core. won't match those in FreeBSD tree, becaus= e I've doing >>>> some unrelated hacking in the file. >>>> ]]] >>>> >>>> The problem seems to be with creation of "UMA Zones" zone and keg. >>>> Because of the large number of processors, size argument in the follow= ing snippet >>>> is set to a value of 4480: >>>> >>>> args.name =3D "UMA Zones"; >>>> args.size =3D sizeof(struct uma_zone) + >>>> =A0 =A0 (sizeof(struct uma_cache) * (mp_maxid + 1)); >>>> >>>> Because of this, keg_ctor() calls keg_large_init(): >>>> >>>> else if ((keg->uk_size+UMA_FRITM_SZ) > >>>> =A0 =A0 (UMA_SLAB_SIZE - sizeof(struct uma_slab))) >>>> =A0 =A0 =A0 =A0 keg_large_init(keg); >>>> else >>>> =A0 =A0 =A0 =A0 keg_small_init(keg); >>>> >>>> keg_large_init sets UMA_ZONE_OFFPAGE and UMA_ZONE_HASH flags for this = keg. >>>> This leads to hash_alloc() being invoked from keg_ctor(): >>>> >>>> if (keg->uk_flags & UMA_ZONE_HASH) >>>> =A0 =A0 =A0 =A0 hash_alloc(&keg->uk_hash); >>>> >>>> But the problem is that "UMA Hash" zone is not created yet and thus th= e call leads >>>> to the panic. =A0"UMA Hash" zone is the last of system zones created. >>>> >>>> Not sure what the proper fix here could/should be. >>>> Would it work to simply not set UMA_ZONE_HASH flag when UMA_ZFLAG_INTE= RNAL is set? >>>> >>>> >>>> And some final calculations. >>>> On the test system sizeof(struct uma_cache) is 128 bytes and (mp_maxid= + 1) is 32, >>>> so it's already UMA_SLAB_SIZE =3D PAGE_SIZE =3D 4096. >>>> >>> >>> Here is a simple solution that seems to work: >>> http://people.freebsd.org/~avg/uma-many-cpus.diff >>> Not sure if it's the best we can do. >>> >> >> I don't know if it makes sense I only want to raise a flag. >> Is it safe to call kmem_malloc() before bucket_init() during >> uma_startup() to reserve room for CPU caches? > > Hmm, not sure what exactly you mean. Sorry, nevermind > >> Reading the top uma_int.h comment, it seems that the best way to >> handle this issue >> would be to implement and allow for dynamic slab sizes. > > Again, not sure if I follow you, I don't see relation between per-cpu cac= hes and > dynamic slab size. Your patch seems just a work around about initial slab size where the keg is backed. Having dynamic slab sizes would allow to have the keg backed on a larger sl= ab without going OFFPAGE. -- Giovanni Trematerra