Date: Fri, 28 Jan 2011 09:25:06 -0600 From: Dan Nelson <dnelson@allantgroup.com> To: Ivan Voras <ivoras@freebsd.org> Cc: freebsd-hackers@freebsd.org Subject: Re: Namecache lock contention? Message-ID: <20110128152505.GP75125@dan.emsphone.com> In-Reply-To: <ihuhav$qso$1@dough.gmane.org> References: <ihuhav$qso$1@dough.gmane.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In the last episode (Jan 28), Ivan Voras said: > I have this situation on a PHP server: > > 36623 www 1 76 0 237M 30600K *Name 6 0:14 47.27% php-cgi > 36638 www 1 76 0 237M 30600K *Name 3 0:14 46.97% php-cgi > 36628 www 1 105 0 237M 30600K *Name 2 0:14 46.88% php-cgi > 36627 www 1 105 0 237M 30600K *Name 0 0:14 46.78% php-cgi > 36639 www 1 105 0 237M 30600K *Name 5 0:14 46.58% php-cgi > 36643 www 1 105 0 237M 30600K *Name 7 0:14 46.39% php-cgi > 36629 www 1 76 0 237M 30600K *Name 1 0:14 46.39% php-cgi > 36642 www 1 105 0 237M 30600K *Name 2 0:14 46.39% php-cgi > 36626 www 1 105 0 237M 30600K *Name 5 0:14 46.19% php-cgi > 36654 www 1 105 0 237M 30600K *Name 7 0:13 46.19% php-cgi > 36645 www 1 105 0 237M 30600K *Name 1 0:14 45.75% php-cgi > 36625 www 1 105 0 237M 30600K *Name 0 0:14 45.56% php-cgi > 36624 www 1 105 0 237M 30600K *Name 6 0:14 45.56% php-cgi > 36630 www 1 76 0 237M 30600K *Name 7 0:14 45.17% php-cgi > 36631 www 1 105 0 237M 30600K RUN 4 0:14 45.17% php-cgi > 36636 www 1 105 0 237M 30600K *Name 3 0:14 44.87% php-cgi > > It looks like periodically most or all of the php-cgi processes are > blocked in "*Name" for long enough that "top" notices, then continue, > probably in a "thundering herd" way. From grepping inside /sys the most > likely suspect seems to be something in the namecache, but I can't find > exactly a symbol named "Name" or string beginning with "Name" that would > be connected to a lock. My guess would be: kern/vfs_cache.c:151 static struct rwlock cache_lock; kern/vfs_cache.c:152 RW_SYSINIT(vfscache, &cache_lock, "Name Cache"); The CACHE_*LOCK() macros.c in vfs_cache use cache_lock, so you've got lots of possible contention points. procstat -ka and/or dtrace might help you determine exactly where. -- Dan Nelson dnelson@allantgroup.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110128152505.GP75125>