Date: Sat, 22 Jun 2013 16:37:24 -1000 (HST) From: Jeff Roberson <jroberson@jroberson.net> To: Zbyszek Bodek <zbb@semihalf.com> Cc: freebsd-arm@FreeBSD.org, freebsd-current@freebsd.org, jeff@FreeBSD.org Subject: Re: Kernel build fails on ARM: Cannot fork: Cannot allocate memory Message-ID: <alpine.BSF.2.00.1306221636360.43796@desktop> In-Reply-To: <51C4A067.7010203@semihalf.com> References: <51C1F53B.2080502@semihalf.com> <alpine.BSF.2.00.1306200755210.2005@desktop> <alpine.BSF.2.00.1306201350330.2005@desktop> <51C4A067.7010203@semihalf.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 21 Jun 2013, Zbyszek Bodek wrote:
> On 21.06.2013 01:56, Jeff Roberson wrote:
>> On Thu, 20 Jun 2013, Jeff Roberson wrote:
>>
>>> On Wed, 19 Jun 2013, Zbyszek Bodek wrote:
>>>
>>>> Hello,
>>>>
>>>> I've been trying to compile the kernel on my ARMv7 platform using the
>>>> sources from the current FreeBSD HEAD.
>>>>
>>>> make buildkernel <.....> -j5
>>>>
>>>> 1/2 builds fails in the way described below:
>>>> --------------------------------------------------------------------------
>>>>
>>>> ing-include-dirs -fdiagnostics-show-option -nostdinc -I.
>>>> -I/root/src/freebsd-arm-superpages/sys
>>>> -I/root/src/freebsd-arm-superpages/sys/contrib/altq
>>>> -I/root/src/freebsd-arm-superpages/sys/contrib/libfdt -D_KERNEL
>>>> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common
>>>> -finline-limit=8000 --param inline-unit-growth=100 --param
>>>> large-function-growth=1000 -mno-thumb-interwork -ffreestanding -Werror
>>>> /root/src/freebsd-arm-superpages/sys/ufs/ffs/ffs_snapshot.c
>>>> Cannot fork: Cannot allocate memory
>>>> *** [ffs_snapshot.o] Error code 2
>>>> 1 error
>>>> *** [buildkernel] Error code 2
>>>> 1 error
>>>> *** [buildkernel] Error code 2
>>>> 1 error
>>>> 5487.888u 481.569s 7:35.65 1310.0% 1443+167k 1741+5388io 221pf+0w
>>>> --------------------------------------------------------------------------
>>>>
>>>>
>>>> The warning from std err is:
>>>> --------------------------------------------------------------------------
>>>>
>>>> vm_thread_new: kstack allocation failed
>>>> vm_thread_new: kstack allocation failed
>>>> --------------------------------------------------------------------------
>>>>
>>>>
>>>> I was trying to find out which commit is causing this (because I was
>>>> previously working on some older revision) and using bisect I got to:
>>>>
>>>> --------------------------------------------------------------------------
>>>>
>>>> Author: jeff <jeff@FreeBSD.org>
>>>> Date: Tue Jun 18 04:50:20 2013 +0000
>>>>
>>>> Refine UMA bucket allocation to reduce space consumption and improve
>>>> performance.
>>>>
>>>> - Always free to the alloc bucket if there is space. This gives
>>>> LIFO
>>>> allocation order to improve hot-cache performance. This also
>>>> allows
>>>> for zones with a single bucket per-cpu rather than a pair if the
>>>> entire
>>>> working set fits in one bucket.
>>>> - Enable per-cpu caches of buckets. To prevent recursive bucket
>>>> allocation one bucket zone still has per-cpu caches disabled.
>>>> - Pick the initial bucket size based on a table driven maximum size
>>>> per-bucket rather than the number of items per-page. This gives
>>>> more sane initial sizes.
>>>> - Only grow the bucket size when we face contention on the zone
>>>> lock, this
>>>> causes bucket sizes to grow more slowly.
>>>> - Adjust the number of items per-bucket to account for the header
>>>> space.
>>>> This packs the buckets more efficiently per-page while making them
>>>> not quite powers of two.
>>>> - Eliminate the per-zone free bucket list. Always return buckets
>>>> back
>>>> to the bucket zone. This ensures that as zones grow into larger
>>>> bucket sizes they eventually discard the smaller sizes. It
>>>> persists
>>>> fewer buckets in the system. The locking is slightly trickier.
>>>> - Only switch buckets in zalloc, not zfree, this eliminates
>>>> pathological
>>>> cases where we ping-pong between two buckets.
>>>> - Ensure that the thread that fills a new bucket gets to allocate
>>>> from
>>>> it to give a better upper bound on allocation time.
>>>>
>>>> Sponsored by: EMC / Isilon Storage Division
>>>> --------------------------------------------------------------------------
>>>>
>>>>
>>>> I checked this several times and this commits seems to be causing this.
>>>
>>> Can you tell me how many cores and how much memory you have? And
>>> paste the output of vmstat -z when you see this error.
>>>
>>> You can try changing bucket_select() at line 339 in uma_core.c to read:
>>>
>>> static int
>>> bucket_select(int size)
>>> {
>>> return (MAX(PAGE_SIZE / size, 1));
>>> }
>>>
>>> This will approximate the old bucket sizing behavior.
>>
>> Just to add some more information; On my machine with 16GB of ram the
>> handful of recent UMA commits save about 20MB of kmem on boot. There
>> are 30% fewer buckets allocated. And all of the malloc zones have
>> similar amounts of cached space. Actually the page size malloc bucket
>> is taking up much less space.
>>
>> I don't know if the problem is unique to arm but I have tested x86
>> limited to 512MB of ram without trouble. I will need the stats I
>> mentioned before to understand what has happened.
>>
>
> Hello Jeff,
>
> Thank you for your interest in my problem.
>
> My system is a quad-core ARMv7 with 2048 MB of RAM on board.
> Please see attachment for the output from vmstat -z when the error occurs.
>
> Changing bucket_select() to
>
> static int
> bucket_select(int size)
> {
> return (MAX(PAGE_SIZE / size, 1));
> }
>
> as you suggested helps for the problem. I've performed numerous attempts
> to build the kernel and none of them failed.
>
I don't really see a lot of wasted memory in the zones. There is
certainly some. Can you give me sysctl vm from both a working and
non-working kernel after the build is done or fails?
Thanks,
Jeff
> Best regards
> Zbyszek Bodek
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1306221636360.43796>