From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 28 13:46:25 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22A991065694 for ; Fri, 28 Jan 2011 13:46:25 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id A2F648FC12 for ; Fri, 28 Jan 2011 13:46:24 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Pioec-0000Wz-Ip for freebsd-hackers@freebsd.org; Fri, 28 Jan 2011 14:46:22 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 28 Jan 2011 14:46:22 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 28 Jan 2011 14:46:22 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Fri, 28 Jan 2011 14:46:07 +0100 Lines: 32 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6 X-Enigmail-Version: 1.1.2 Subject: Namecache lock contention? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 13:46:25 -0000 I have this situation on a PHP server: 36623 www 1 76 0 237M 30600K *Name 6 0:14 47.27% php-cgi 36638 www 1 76 0 237M 30600K *Name 3 0:14 46.97% php-cgi 36628 www 1 105 0 237M 30600K *Name 2 0:14 46.88% php-cgi 36627 www 1 105 0 237M 30600K *Name 0 0:14 46.78% php-cgi 36639 www 1 105 0 237M 30600K *Name 5 0:14 46.58% php-cgi 36643 www 1 105 0 237M 30600K *Name 7 0:14 46.39% php-cgi 36629 www 1 76 0 237M 30600K *Name 1 0:14 46.39% php-cgi 36642 www 1 105 0 237M 30600K *Name 2 0:14 46.39% php-cgi 36626 www 1 105 0 237M 30600K *Name 5 0:14 46.19% php-cgi 36654 www 1 105 0 237M 30600K *Name 7 0:13 46.19% php-cgi 36645 www 1 105 0 237M 30600K *Name 1 0:14 45.75% php-cgi 36625 www 1 105 0 237M 30600K *Name 0 0:14 45.56% php-cgi 36624 www 1 105 0 237M 30600K *Name 6 0:14 45.56% php-cgi 36630 www 1 76 0 237M 30600K *Name 7 0:14 45.17% php-cgi 36631 www 1 105 0 237M 30600K RUN 4 0:14 45.17% php-cgi 36636 www 1 105 0 237M 30600K *Name 3 0:14 44.87% php-cgi It looks like periodically most or all of the php-cgi processes are blocked in "*Name" for long enough that "top" notices, then continue, probably in a "thundering herd" way. From grepping inside /sys the most likely suspect seems to be something in the namecache, but I can't find exactly a symbol named "Name" or string beginning with "Name" that would be connected to a lock. Has anyone investigated this before? Any ideas where to look? The PHP script used above should not do any filesystem writing but it stats and reads a lot of small files and libraries. This is 8-stable, UFS.