From owner-freebsd-performance@FreeBSD.ORG  Mon Feb  4 02:40:44 2008
Return-Path: <owner-freebsd-performance@FreeBSD.ORG>
Delivered-To: freebsd-performance@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B03EF16A417;
	Mon,  4 Feb 2008 02:40:44 +0000 (UTC)
	(envelope-from brde@optusnet.com.au)
Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au
	[211.29.132.199])
	by mx1.freebsd.org (Postfix) with ESMTP id 20C7A13C455;
	Mon,  4 Feb 2008 02:40:43 +0000 (UTC)
	(envelope-from brde@optusnet.com.au)
Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au
	(c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213])
	by mail18.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id
	m142eVi6014054
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 4 Feb 2008 13:40:33 +1100
Date: Mon, 4 Feb 2008 13:40:31 +1100 (EST)
From: Bruce Evans <brde@optusnet.com.au>
X-X-Sender: bde@delplex.bde.org
To: Alexander Motin <mav@freebsd.org>
In-Reply-To: <47A650DA.8020908@FreeBSD.org>
Message-ID: <20080204133437.T63947@delplex.bde.org>
References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org>
	<47A2C2A2.5040109@FreeBSD.org>
	<20080201185435.X88034@fledge.watson.org>
	<47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org>
	<47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org>
	<47A650DA.8020908@FreeBSD.org>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: freebsd-hackers@freebsd.org, Kris Kennaway <kris@freebsd.org>,
	Robert Watson <rwatson@freebsd.org>, freebsd-performance@freebsd.org,
	Julian Elischer <julian@elischer.org>
Subject: Re: Memory allocation performance
X-BeenThere: freebsd-performance@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Performance/tuning <freebsd-performance.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-performance>,
	<mailto:freebsd-performance-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-performance>
List-Post: <mailto:freebsd-performance@freebsd.org>
List-Help: <mailto:freebsd-performance-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-performance>,
	<mailto:freebsd-performance-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Feb 2008 02:40:44 -0000

On Mon, 4 Feb 2008, Alexander Motin wrote:

> Kris Kennaway wrote:
>> You can look at the raw output from pmcstat, which is a collection of 
>> instruction pointers that you can feed to e.g. addr2line to find out 
>> exactly where in those functions the events are occurring.  This will often 
>> help to track down the precise causes.
>
> Thanks to the hint, it was interesting hunting, but it shown nothing. It hits 
> into very simple lines like:
> bucket = cache->uc_freebucket;
> cache->uc_allocs++;
> if (zone->uz_ctor != NULL) {
> cache->uc_frees++;
> and so on.
> There is no loops, there is no inlines or macroses. Nothing! And the only 
> hint about it is a huge number of "p4-resource-stall"s in those lines. I have 
> no idea what exactly does it means, why does it happens mostly here and how 
> to fight it.

Try profiling it one another type of CPU, to get different performance
counters but hopefully not very different stalls.  If the other CPU doesn't
stall at all, put another black mark against P4 and delete your copies of
it :-).

Bruce