From owner-freebsd-performance@FreeBSD.ORG  Thu Jan 31 23:30:12 2008
Return-Path: <owner-freebsd-performance@FreeBSD.ORG>
Delivered-To: freebsd-performance@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 8BF5F16A418;
	Thu, 31 Jan 2008 23:30:12 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 2CDE913C465;
	Thu, 31 Jan 2008 23:30:10 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Message-ID: <47A25A01.5080508@FreeBSD.org>
Date: Fri, 01 Feb 2008 00:30:09 +0100
From: Kris Kennaway <kris@FreeBSD.org>
User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031)
MIME-Version: 1.0
To: Alexander Motin <mav@FreeBSD.org>
References: <47A25412.3010301@FreeBSD.org>
In-Reply-To: <47A25412.3010301@FreeBSD.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org
Subject: Re: Memory allocation performance
X-BeenThere: freebsd-performance@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Performance/tuning <freebsd-performance.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-performance>,
	<mailto:freebsd-performance-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-performance>
List-Post: <mailto:freebsd-performance@freebsd.org>
List-Help: <mailto:freebsd-performance-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-performance>,
	<mailto:freebsd-performance-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 31 Jan 2008 23:30:12 -0000

Alexander Motin wrote:
> Hi.
> 
> While profiling netgraph operation on UP HEAD router I have found that 
> huge amount of time it spent on memory allocation/deallocation:
> 
>         0.14  0.05  132119/545292      ip_forward <cycle 1> [12]
>         0.14  0.05  133127/545292      fxp_add_rfabuf [18]
>         0.27  0.10  266236/545292      ng_package_data [17]
> [9]14.1 0.56  0.21  545292         uma_zalloc_arg [9]
>         0.17  0.00  545292/1733401     critical_exit <cycle 2> [98]
>         0.01  0.00  275941/679675      generic_bzero [68]
>         0.01  0.00  133127/133127      mb_ctor_pack [103]
> 
>         0.15  0.06  133100/545266      mb_free_ext [22]
>         0.15  0.06  133121/545266      m_freem [15]
>         0.29  0.11  266236/545266      ng_free_item [16]
> [8]15.2 0.60  0.23  545266         uma_zfree_arg [8]
>         0.17  0.00  545266/1733401     critical_exit <cycle 2> [98]
>         0.00  0.04  133100/133100      mb_dtor_pack [57]
>         0.00  0.00  134121/134121      mb_dtor_mbuf [111]
> 
> I have already optimized all possible allocation calls and those that 
> left are practically unavoidable. But even after this kgmon tells that 
> 30% of CPU time consumed by memory management.
> 
> So I have some questions:
> 1) Is it real situation or just profiler mistake?
> 2) If it is real then why UMA is so slow? I have tried to replace it in 
> some places with preallocated TAILQ of required memory blocks protected 
> by mutex and according to profiler I have got _much_ better results. 
> Will it be a good practice to replace relatively small UMA zones with 
> preallocated queue to avoid part of UMA calls?
> 3) I have seen that UMA does some kind of CPU cache affinity, but does 
> it cost so much that it costs 30% CPU time on UP router?

Make sure you have INVARIANTS disabled, it has a high performance cost 
in UMA.

Kris