From owner-freebsd-arm@FreeBSD.ORG Sun Jun 23 02:35:18 2013 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 36F19B3A for ; Sun, 23 Jun 2013 02:35:18 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pa0-x230.google.com (mail-pa0-x230.google.com [IPv6:2607:f8b0:400e:c03::230]) by mx1.freebsd.org (Postfix) with ESMTP id 10CEE1A16 for ; Sun, 23 Jun 2013 02:35:18 +0000 (UTC) Received: by mail-pa0-f48.google.com with SMTP id kp12so9437908pab.7 for ; Sat, 22 Jun 2013 19:35:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:x-x-sender:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version:content-type:x-gm-message-state; bh=zN6shDXMHFcyCzhPbEJbSqAnLUW8NGj6LtvOK2uuEJg=; b=YhYK/pal62jjJ7RDXR2PDx04YiKmMN6hX2W0hFNdVz7FyEVsksebtZAUsYVunYZqrt VVVvylrRNMKOCT7z2rwnfl2aQSOrwMYfA0QqAlXR2h6Nu4tg/TvmC3A7wPsmDeZHVpFv 3UFsob6fwwCYOCSyGG2r9nEPzRxgQmwo5Z3UN4nBzlnA8BQppYu9Z5/GIwec0+9qDBLr R2pdtOXZ/+rT26QZSR1I9vgHmhn8vCKmXgB88TrNokRmK8XOC1Ex8lJ1ZVW3hFFUUH5L tEc18DNJ7ffSJA5uuix7A464SV8JmrJahwewNNJiVjs/uiiEcfcCAl7RJSAWxASF9T+K k3GA== X-Received: by 10.66.224.237 with SMTP id rf13mr22254140pac.26.1371954917857; Sat, 22 Jun 2013 19:35:17 -0700 (PDT) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by mx.google.com with ESMTPSA id re16sm12773029pac.16.2013.06.22.19.35.16 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Sat, 22 Jun 2013 19:35:17 -0700 (PDT) Date: Sat, 22 Jun 2013 16:37:24 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Zbyszek Bodek Subject: Re: Kernel build fails on ARM: Cannot fork: Cannot allocate memory In-Reply-To: <51C4A067.7010203@semihalf.com> Message-ID: References: <51C1F53B.2080502@semihalf.com> <51C4A067.7010203@semihalf.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Gm-Message-State: ALoCoQlzQCoJC/u6wHAlWgjMNXt/4f7yp5ldxJbijmHR83e0NY1xuF8OqspE+1LgHAPrRqPQEqQy Cc: freebsd-arm@FreeBSD.org, freebsd-current@freebsd.org, jeff@FreeBSD.org X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 02:35:18 -0000 On Fri, 21 Jun 2013, Zbyszek Bodek wrote: > On 21.06.2013 01:56, Jeff Roberson wrote: >> On Thu, 20 Jun 2013, Jeff Roberson wrote: >> >>> On Wed, 19 Jun 2013, Zbyszek Bodek wrote: >>> >>>> Hello, >>>> >>>> I've been trying to compile the kernel on my ARMv7 platform using the >>>> sources from the current FreeBSD HEAD. >>>> >>>> make buildkernel <.....> -j5 >>>> >>>> 1/2 builds fails in the way described below: >>>> -------------------------------------------------------------------------- >>>> >>>> ing-include-dirs -fdiagnostics-show-option -nostdinc -I. >>>> -I/root/src/freebsd-arm-superpages/sys >>>> -I/root/src/freebsd-arm-superpages/sys/contrib/altq >>>> -I/root/src/freebsd-arm-superpages/sys/contrib/libfdt -D_KERNEL >>>> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common >>>> -finline-limit=8000 --param inline-unit-growth=100 --param >>>> large-function-growth=1000 -mno-thumb-interwork -ffreestanding -Werror >>>> /root/src/freebsd-arm-superpages/sys/ufs/ffs/ffs_snapshot.c >>>> Cannot fork: Cannot allocate memory >>>> *** [ffs_snapshot.o] Error code 2 >>>> 1 error >>>> *** [buildkernel] Error code 2 >>>> 1 error >>>> *** [buildkernel] Error code 2 >>>> 1 error >>>> 5487.888u 481.569s 7:35.65 1310.0% 1443+167k 1741+5388io 221pf+0w >>>> -------------------------------------------------------------------------- >>>> >>>> >>>> The warning from std err is: >>>> -------------------------------------------------------------------------- >>>> >>>> vm_thread_new: kstack allocation failed >>>> vm_thread_new: kstack allocation failed >>>> -------------------------------------------------------------------------- >>>> >>>> >>>> I was trying to find out which commit is causing this (because I was >>>> previously working on some older revision) and using bisect I got to: >>>> >>>> -------------------------------------------------------------------------- >>>> >>>> Author: jeff >>>> Date: Tue Jun 18 04:50:20 2013 +0000 >>>> >>>> Refine UMA bucket allocation to reduce space consumption and improve >>>> performance. >>>> >>>> - Always free to the alloc bucket if there is space. This gives >>>> LIFO >>>> allocation order to improve hot-cache performance. This also >>>> allows >>>> for zones with a single bucket per-cpu rather than a pair if the >>>> entire >>>> working set fits in one bucket. >>>> - Enable per-cpu caches of buckets. To prevent recursive bucket >>>> allocation one bucket zone still has per-cpu caches disabled. >>>> - Pick the initial bucket size based on a table driven maximum size >>>> per-bucket rather than the number of items per-page. This gives >>>> more sane initial sizes. >>>> - Only grow the bucket size when we face contention on the zone >>>> lock, this >>>> causes bucket sizes to grow more slowly. >>>> - Adjust the number of items per-bucket to account for the header >>>> space. >>>> This packs the buckets more efficiently per-page while making them >>>> not quite powers of two. >>>> - Eliminate the per-zone free bucket list. Always return buckets >>>> back >>>> to the bucket zone. This ensures that as zones grow into larger >>>> bucket sizes they eventually discard the smaller sizes. It >>>> persists >>>> fewer buckets in the system. The locking is slightly trickier. >>>> - Only switch buckets in zalloc, not zfree, this eliminates >>>> pathological >>>> cases where we ping-pong between two buckets. >>>> - Ensure that the thread that fills a new bucket gets to allocate >>>> from >>>> it to give a better upper bound on allocation time. >>>> >>>> Sponsored by: EMC / Isilon Storage Division >>>> -------------------------------------------------------------------------- >>>> >>>> >>>> I checked this several times and this commits seems to be causing this. >>> >>> Can you tell me how many cores and how much memory you have? And >>> paste the output of vmstat -z when you see this error. >>> >>> You can try changing bucket_select() at line 339 in uma_core.c to read: >>> >>> static int >>> bucket_select(int size) >>> { >>> return (MAX(PAGE_SIZE / size, 1)); >>> } >>> >>> This will approximate the old bucket sizing behavior. >> >> Just to add some more information; On my machine with 16GB of ram the >> handful of recent UMA commits save about 20MB of kmem on boot. There >> are 30% fewer buckets allocated. And all of the malloc zones have >> similar amounts of cached space. Actually the page size malloc bucket >> is taking up much less space. >> >> I don't know if the problem is unique to arm but I have tested x86 >> limited to 512MB of ram without trouble. I will need the stats I >> mentioned before to understand what has happened. >> > > Hello Jeff, > > Thank you for your interest in my problem. > > My system is a quad-core ARMv7 with 2048 MB of RAM on board. > Please see attachment for the output from vmstat -z when the error occurs. > > Changing bucket_select() to > > static int > bucket_select(int size) > { > return (MAX(PAGE_SIZE / size, 1)); > } > > as you suggested helps for the problem. I've performed numerous attempts > to build the kernel and none of them failed. > I don't really see a lot of wasted memory in the zones. There is certainly some. Can you give me sysctl vm from both a working and non-working kernel after the build is done or fails? Thanks, Jeff > Best regards > Zbyszek Bodek >