From owner-freebsd-hackers@FreeBSD.ORG  Fri Jan 28 17:20:57 2011
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id E87EE10656A4
	for <freebsd-hackers@freebsd.org>; Fri, 28 Jan 2011 17:20:57 +0000 (UTC)
	(envelope-from ivoras@gmail.com)
Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com
	[209.85.220.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 9FE928FC17
	for <freebsd-hackers@freebsd.org>; Fri, 28 Jan 2011 17:20:57 +0000 (UTC)
Received: by vxa40 with SMTP id 40so148634vxa.13
	for <freebsd-hackers@freebsd.org>; Fri, 28 Jan 2011 09:20:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:sender:in-reply-to:references:from
	:date:x-google-sender-auth:message-id:subject:to:cc:content-type
	:content-transfer-encoding;
	bh=lqVbz7ocdOx/mRvRtWZ6PvuYsi1Pdlu5zo58LQ09lGM=;
	b=wDYeiHBXpxZbQuiqtr+sk7Mzgb2gwhhBhflpPrM9a0zoECXR3gbzG9z5TnfdehJZ83
	kOx7KrWVxu/EttoEAukE3hV2s7sgHRh2PC/xLjf2weTuIZLexjbcMaZZBAEWLUDSVBgz
	2itDSeG5VrO+wwRF441ml2tE5sL7f4FKOxKhA=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:sender:in-reply-to:references:from:date
	:x-google-sender-auth:message-id:subject:to:cc:content-type
	:content-transfer-encoding;
	b=Fc2yx5y7gzugZmKnsF5psDyOUajfV0/q4Whgb6/c5kEA++L8JkXZc/MGXis/arV1Vt
	0I+CAUfuRKBQ/4rqgX/x+1+RnzXeLd6csFRM/Fjk3yuSq6rN4Om84jmuhq6HWSz2WOdM
	rkIiwuUO2zXWgW9flXC/vnabjy1Eu/M++Fw2I=
Received: by 10.229.181.74 with SMTP id bx10mr2868447qcb.215.1296235255812;
	Fri, 28 Jan 2011 09:20:55 -0800 (PST)
MIME-Version: 1.0
Sender: ivoras@gmail.com
Received: by 10.229.67.2 with HTTP; Fri, 28 Jan 2011 09:20:15 -0800 (PST)
In-Reply-To: <20110128152505.GP75125@dan.emsphone.com>
References: <ihuhav$qso$1@dough.gmane.org>
	<20110128152505.GP75125@dan.emsphone.com>
From: Ivan Voras <ivoras@freebsd.org>
Date: Fri, 28 Jan 2011 18:20:15 +0100
X-Google-Sender-Auth: ZFH-TGykQUhuI0BKDIT95Y_FMOM
Message-ID: <AANLkTinCbhY5N+puAs9gXT3WOKmPtHCZrb_BwWu__nV6@mail.gmail.com>
To: Dan Nelson <dnelson@allantgroup.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: freebsd-hackers@freebsd.org
Subject: Re: Namecache lock contention?
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 28 Jan 2011 17:20:58 -0000

On 28 January 2011 16:25, Dan Nelson <dnelson@allantgroup.com> wrote:

> My guess would be:
>
> kern/vfs_cache.c:151 static struct rwlock cache_lock;
> kern/vfs_cache.c:152 RW_SYSINIT(vfscache, &cache_lock, "Name Cache");
>
> The CACHE_*LOCK() macros.c in vfs_cache use cache_lock, so you've got lot=
s
> of possible contention points. =C2=A0procstat -ka and/or dtrace might hel=
p you
> determine exactly where.

I'm new with dtrace so I tried this:

lockstat:::rw-block
{
        @traces[stack()] =3D count();
}

with these results:

http://ivoras.net/stuff/rw-block.txt

It's informative because most of the traces are namecache-related. As
suspected, the most blocking occurs in stat().

As this is a rwlock I'd interpret it as waiting for a write lock to be
lifted so the readers can acquire it, but I need to confirm this as
there's a lot of things that can in theory be stat()ed here.

I'm going to continue investigating with dtrace but I'd appreciate
pointers on how to make the output more useful (like including
filenames from stat()).