Date: Fri, 28 Jan 2011 23:18:34 +0200 From: Gleb Kurtsou <gleb.kurtsou@gmail.com> To: Ivan Voras <ivoras@freebsd.org> Cc: freebsd-hackers@freebsd.org, Dan Nelson <dnelson@allantgroup.com> Subject: Re: Namecache lock contention? Message-ID: <20110128211834.GA84881@tops.skynet.lt> In-Reply-To: <AANLkTinCbhY5N%2BpuAs9gXT3WOKmPtHCZrb_BwWu__nV6@mail.gmail.com> References: <ihuhav$qso$1@dough.gmane.org> <20110128152505.GP75125@dan.emsphone.com> <AANLkTinCbhY5N%2BpuAs9gXT3WOKmPtHCZrb_BwWu__nV6@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On (28/01/2011 18:20), Ivan Voras wrote:
> On 28 January 2011 16:25, Dan Nelson <dnelson@allantgroup.com> wrote:
>
> > My guess would be:
> >
> > kern/vfs_cache.c:151 static struct rwlock cache_lock;
> > kern/vfs_cache.c:152 RW_SYSINIT(vfscache, &cache_lock, "Name Cache");
> >
> > The CACHE_*LOCK() macros.c in vfs_cache use cache_lock, so you've got lots
> > of possible contention points. procstat -ka and/or dtrace might help you
> > determine exactly where.
>
> I'm new with dtrace so I tried this:
>
> lockstat:::rw-block
> {
> @traces[stack()] = count();
> }
>
> with these results:
>
> http://ivoras.net/stuff/rw-block.txt
>
> It's informative because most of the traces are namecache-related. As
> suspected, the most blocking occurs in stat().
>
> As this is a rwlock I'd interpret it as waiting for a write lock to be
> lifted so the readers can acquire it, but I need to confirm this as
> there's a lot of things that can in theory be stat()ed here.
>
> I'm going to continue investigating with dtrace but I'd appreciate
> pointers on how to make the output more useful (like including
> filenames from stat()).
You could try replacing rwlock with plain mutex to check if there are
priority propagation issues among readers/writers. SX locks should also
work but would likely to be a considerable performance regression.
Finding out home much activity is there outside of storage/a/b/file
(sessions storage) could also be helpful.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110128211834.GA84881>