From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 28 21:18:51 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 30E6A10656C1; Fri, 28 Jan 2011 21:18:51 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8B9B28FC13; Fri, 28 Jan 2011 21:18:50 +0000 (UTC) Received: by eyf6 with SMTP id 6so1844610eyf.13 for ; Fri, 28 Jan 2011 13:18:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=8vEXFk2jvvz/FZZg8IL1CAW4JrFZxHO0nXsBjWaCRaQ=; b=X12WRfXVS6n5MgvdC4SS6ZCg9NwxI15zxccm7Is39y6nW6YywcBEpSLbrgz2ghlAK8 yCKJYQ3kpLMndTe5nxOaEfEa2yyVuKRSNzao6bhkTY/3JxuZrHs52FlF7uDGVeXEsc05 CN5zkg65bPfI0cMkvh9V0y35HWyexIQkfToeE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; b=rnfApNG2JIM8MOSBNjnWcz8SEIWEDIY4/JGM1FLnuRFvhCf52oDwjA8J1CnnhIyuB6 nbYMhX8VtOaJfYGCe9Bj351hNDMlMuB1l6df1sQH4gu5VUyVRqaD7kotkF9IC+kn18j/ u3dgtJOQJIFEb68p5y9Yz4jWHzfVk/5ovaNAg= Received: by 10.213.104.140 with SMTP id p12mr5515372ebo.76.1296249528504; Fri, 28 Jan 2011 13:18:48 -0800 (PST) Received: from localhost (lan-78-157-92-5.vln.skynet.lt [78.157.92.5]) by mx.google.com with ESMTPS id x54sm14132705eeh.17.2011.01.28.13.18.47 (version=SSLv3 cipher=RC4-MD5); Fri, 28 Jan 2011 13:18:47 -0800 (PST) Date: Fri, 28 Jan 2011 23:18:34 +0200 From: Gleb Kurtsou To: Ivan Voras Message-ID: <20110128211834.GA84881@tops.skynet.lt> References: <20110128152505.GP75125@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org, Dan Nelson Subject: Re: Namecache lock contention? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 21:18:51 -0000 On (28/01/2011 18:20), Ivan Voras wrote: > On 28 January 2011 16:25, Dan Nelson wrote: > > > My guess would be: > > > > kern/vfs_cache.c:151 static struct rwlock cache_lock; > > kern/vfs_cache.c:152 RW_SYSINIT(vfscache, &cache_lock, "Name Cache"); > > > > The CACHE_*LOCK() macros.c in vfs_cache use cache_lock, so you've got lots > > of possible contention points.  procstat -ka and/or dtrace might help you > > determine exactly where. > > I'm new with dtrace so I tried this: > > lockstat:::rw-block > { > @traces[stack()] = count(); > } > > with these results: > > http://ivoras.net/stuff/rw-block.txt > > It's informative because most of the traces are namecache-related. As > suspected, the most blocking occurs in stat(). > > As this is a rwlock I'd interpret it as waiting for a write lock to be > lifted so the readers can acquire it, but I need to confirm this as > there's a lot of things that can in theory be stat()ed here. > > I'm going to continue investigating with dtrace but I'd appreciate > pointers on how to make the output more useful (like including > filenames from stat()). You could try replacing rwlock with plain mutex to check if there are priority propagation issues among readers/writers. SX locks should also work but would likely to be a considerable performance regression. Finding out home much activity is there outside of storage/a/b/file (sessions storage) could also be helpful.