From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 28 15:34:33 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 728361065673; Fri, 28 Jan 2011 15:34:33 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 489028FC15; Fri, 28 Jan 2011 15:34:33 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 02A8E46B2E; Fri, 28 Jan 2011 10:34:33 -0500 (EST) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 223878A027; Fri, 28 Jan 2011 10:34:32 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Fri, 28 Jan 2011 10:15:36 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201101281015.36218.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Fri, 28 Jan 2011 10:34:32 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.9 required=4.2 tests=BAYES_00 autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: Ivan Voras Subject: Re: Namecache lock contention? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 15:34:33 -0000 On Friday, January 28, 2011 8:46:07 am Ivan Voras wrote: > I have this situation on a PHP server: > > 36623 www 1 76 0 237M 30600K *Name 6 0:14 47.27% php-cgi > 36638 www 1 76 0 237M 30600K *Name 3 0:14 46.97% php-cgi > 36628 www 1 105 0 237M 30600K *Name 2 0:14 46.88% php-cgi > 36627 www 1 105 0 237M 30600K *Name 0 0:14 46.78% php-cgi > 36639 www 1 105 0 237M 30600K *Name 5 0:14 46.58% php-cgi > 36643 www 1 105 0 237M 30600K *Name 7 0:14 46.39% php-cgi > 36629 www 1 76 0 237M 30600K *Name 1 0:14 46.39% php-cgi > 36642 www 1 105 0 237M 30600K *Name 2 0:14 46.39% php-cgi > 36626 www 1 105 0 237M 30600K *Name 5 0:14 46.19% php-cgi > 36654 www 1 105 0 237M 30600K *Name 7 0:13 46.19% php-cgi > 36645 www 1 105 0 237M 30600K *Name 1 0:14 45.75% php-cgi > 36625 www 1 105 0 237M 30600K *Name 0 0:14 45.56% php-cgi > 36624 www 1 105 0 237M 30600K *Name 6 0:14 45.56% php-cgi > 36630 www 1 76 0 237M 30600K *Name 7 0:14 45.17% php-cgi > 36631 www 1 105 0 237M 30600K RUN 4 0:14 45.17% php-cgi > 36636 www 1 105 0 237M 30600K *Name 3 0:14 44.87% php-cgi > > It looks like periodically most or all of the php-cgi processes are > blocked in "*Name" for long enough that "top" notices, then continue, > probably in a "thundering herd" way. From grepping inside /sys the most > likely suspect seems to be something in the namecache, but I can't find > exactly a symbol named "Name" or string beginning with "Name" that would > be connected to a lock. In vfs_cache.c: static struct rwlock cache_lock; RW_SYSINIT(vfscache, &cache_lock, "Name Cache"); What are the php scripts doing? Do they all try to create and delete files at the same time (or do renames)? -- John Baldwin