Date: Fri, 21 Jun 2013 20:50:15 +0200 From: Zbyszek Bodek <zbb@semihalf.com> To: Jeff Roberson <jroberson@jroberson.net> Cc: freebsd-arm@FreeBSD.org, freebsd-current@freebsd.org, jeff@FreeBSD.org Subject: Re: Kernel build fails on ARM: Cannot fork: Cannot allocate memory Message-ID: <51C4A067.7010203@semihalf.com> In-Reply-To: <alpine.BSF.2.00.1306201350330.2005@desktop> References: <51C1F53B.2080502@semihalf.com> <alpine.BSF.2.00.1306200755210.2005@desktop> <alpine.BSF.2.00.1306201350330.2005@desktop>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------040207010309000602070803
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
On 21.06.2013 01:56, Jeff Roberson wrote:
> On Thu, 20 Jun 2013, Jeff Roberson wrote:
>
>> On Wed, 19 Jun 2013, Zbyszek Bodek wrote:
>>
>>> Hello,
>>>
>>> I've been trying to compile the kernel on my ARMv7 platform using the
>>> sources from the current FreeBSD HEAD.
>>>
>>> make buildkernel <.....> -j5
>>>
>>> 1/2 builds fails in the way described below:
>>> --------------------------------------------------------------------------
>>>
>>> ing-include-dirs -fdiagnostics-show-option -nostdinc -I.
>>> -I/root/src/freebsd-arm-superpages/sys
>>> -I/root/src/freebsd-arm-superpages/sys/contrib/altq
>>> -I/root/src/freebsd-arm-superpages/sys/contrib/libfdt -D_KERNEL
>>> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common
>>> -finline-limit=8000 --param inline-unit-growth=100 --param
>>> large-function-growth=1000 -mno-thumb-interwork -ffreestanding -Werror
>>> /root/src/freebsd-arm-superpages/sys/ufs/ffs/ffs_snapshot.c
>>> Cannot fork: Cannot allocate memory
>>> *** [ffs_snapshot.o] Error code 2
>>> 1 error
>>> *** [buildkernel] Error code 2
>>> 1 error
>>> *** [buildkernel] Error code 2
>>> 1 error
>>> 5487.888u 481.569s 7:35.65 1310.0% 1443+167k 1741+5388io 221pf+0w
>>> --------------------------------------------------------------------------
>>>
>>>
>>> The warning from std err is:
>>> --------------------------------------------------------------------------
>>>
>>> vm_thread_new: kstack allocation failed
>>> vm_thread_new: kstack allocation failed
>>> --------------------------------------------------------------------------
>>>
>>>
>>> I was trying to find out which commit is causing this (because I was
>>> previously working on some older revision) and using bisect I got to:
>>>
>>> --------------------------------------------------------------------------
>>>
>>> Author: jeff <jeff@FreeBSD.org>
>>> Date: Tue Jun 18 04:50:20 2013 +0000
>>>
>>> Refine UMA bucket allocation to reduce space consumption and improve
>>> performance.
>>>
>>> - Always free to the alloc bucket if there is space. This gives
>>> LIFO
>>> allocation order to improve hot-cache performance. This also
>>> allows
>>> for zones with a single bucket per-cpu rather than a pair if the
>>> entire
>>> working set fits in one bucket.
>>> - Enable per-cpu caches of buckets. To prevent recursive bucket
>>> allocation one bucket zone still has per-cpu caches disabled.
>>> - Pick the initial bucket size based on a table driven maximum size
>>> per-bucket rather than the number of items per-page. This gives
>>> more sane initial sizes.
>>> - Only grow the bucket size when we face contention on the zone
>>> lock, this
>>> causes bucket sizes to grow more slowly.
>>> - Adjust the number of items per-bucket to account for the header
>>> space.
>>> This packs the buckets more efficiently per-page while making them
>>> not quite powers of two.
>>> - Eliminate the per-zone free bucket list. Always return buckets
>>> back
>>> to the bucket zone. This ensures that as zones grow into larger
>>> bucket sizes they eventually discard the smaller sizes. It
>>> persists
>>> fewer buckets in the system. The locking is slightly trickier.
>>> - Only switch buckets in zalloc, not zfree, this eliminates
>>> pathological
>>> cases where we ping-pong between two buckets.
>>> - Ensure that the thread that fills a new bucket gets to allocate
>>> from
>>> it to give a better upper bound on allocation time.
>>>
>>> Sponsored by: EMC / Isilon Storage Division
>>> --------------------------------------------------------------------------
>>>
>>>
>>> I checked this several times and this commits seems to be causing this.
>>
>> Can you tell me how many cores and how much memory you have? And
>> paste the output of vmstat -z when you see this error.
>>
>> You can try changing bucket_select() at line 339 in uma_core.c to read:
>>
>> static int
>> bucket_select(int size)
>> {
>> return (MAX(PAGE_SIZE / size, 1));
>> }
>>
>> This will approximate the old bucket sizing behavior.
>
> Just to add some more information; On my machine with 16GB of ram the
> handful of recent UMA commits save about 20MB of kmem on boot. There
> are 30% fewer buckets allocated. And all of the malloc zones have
> similar amounts of cached space. Actually the page size malloc bucket
> is taking up much less space.
>
> I don't know if the problem is unique to arm but I have tested x86
> limited to 512MB of ram without trouble. I will need the stats I
> mentioned before to understand what has happened.
>
Hello Jeff,
Thank you for your interest in my problem.
My system is a quad-core ARMv7 with 2048 MB of RAM on board.
Please see attachment for the output from vmstat -z when the error occurs.
Changing bucket_select() to
static int
bucket_select(int size)
{
return (MAX(PAGE_SIZE / size, 1));
}
as you suggested helps for the problem. I've performed numerous attempts
to build the kernel and none of them failed.
Best regards
Zbyszek Bodek
--------------040207010309000602070803--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51C4A067.7010203>