From owner-freebsd-stable@FreeBSD.ORG Sat Jun 24 19:45:50 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8C82516A49E for ; Sat, 24 Jun 2006 19:45:50 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3037043D5C for ; Sat, 24 Jun 2006 19:45:49 +0000 (GMT) (envelope-from scrappy@hub.org) Received: from localhost (mx1.hub.org [200.46.208.251]) by hub.org (Postfix) with ESMTP id 1B4E0290C29; Sat, 24 Jun 2006 16:45:46 -0300 (ADT) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 50654-03; Sat, 24 Jun 2006 16:45:48 -0300 (ADT) Received: from ganymede.hub.org (blk-7-151-244.eastlink.ca [71.7.151.244]) by hub.org (Postfix) with ESMTP id 4034C290C1E; Sat, 24 Jun 2006 16:45:45 -0300 (ADT) Received: by ganymede.hub.org (Postfix, from userid 1000) id 6169148F4F; Sat, 24 Jun 2006 16:45:49 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id 603134865B; Sat, 24 Jun 2006 16:45:49 -0300 (ADT) Date: Sat, 24 Jun 2006 16:45:49 -0300 (ADT) From: "Marc G. Fournier" To: Kostik Belousov In-Reply-To: <20060624190912.GD79678@deviant.kiev.zoral.com.ua> Message-ID: <20060624164343.V1114@ganymede.hub.org> References: <20060623172557.H1114@ganymede.hub.org> <261AD16B-C3FE-4671-996E-563053508CE8@mac.com> <20060624022227.X1114@ganymede.hub.org> <20060624115505.E14669@woozle.rinet.ru> <20060624090656.GB79678@deviant.kiev.zoral.com.ua> <20060624145432.A1114@ganymede.hub.org> <20060624185203.GC79678@deviant.kiev.zoral.com.ua> <20060624190912.GD79678@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-stable@freebsd.org, Dmitry Morozovsky Subject: Re: vmstat 'b' (disk busy?) field keeps climbing ... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jun 2006 19:45:50 -0000 On Sat, 24 Jun 2006, Kostik Belousov wrote: > On Sat, Jun 24, 2006 at 09:52:03PM +0300, Kostik Belousov wrote: >> On Sat, Jun 24, 2006 at 02:57:27PM -0300, Marc G. Fournier wrote: >>> On Sat, 24 Jun 2006, Kostik Belousov wrote: >>> >>>> On Sat, Jun 24, 2006 at 11:55:26AM +0400, Dmitry Morozovsky wrote: >>>>> On Sat, 24 Jun 2006, Marc G. Fournier wrote: >>>>> >>>>> MGF> > 'b' stands for "blocked", not "busy". Judging by your page fault >>>>> rate >>>>> MGF> > and the high number of frees and pages being scanned, you're >>>>> probably >>>>> MGF> > swapping tasks in and out and are waiting on disk. Take a look at >>>>> MGF> > "vmstat -s", and consider adding more RAM if this is correct... >>>>> MGF> >>>>> MGF> is there a way of finding out what processes are blocked? >>>>> >>>>> Aren't they in 'D' status by ps? >>>> Use ps axlww. In this way, at least actual blocking points are shown. >>> >>> 'k, stupid question then ... what am I searching for? >>> >>> # ps axlww | awk '{print $9}' | sort | uniq -c | sort -nr >>> 654 select >>> 230 lockf >>> 166 wait >>> 85 - >>> 80 piperd >>> 71 nanslp >>> 33 kserel >>> 22 user >>> 10 pause >>> 9 ttyin >>> 5 sbwait >>> 3 psleep >>> 3 accept >>> 2 kqread >>> 2 Giant >>> 1 vlruwt >>> 1 syncer >>> 1 sdflus >>> 1 ppwait >>> 1 ktrace >>> 1 MWCHAN >>> >>> According to vmstat, I'm holding at '4 blocked' for the most part ... >>> sbwwait is socket related, not disk ... and none of the others look right >>> ... >> I would say, using big magic cristall ball, that you problems are >> not kernel-related. I see only too suspicious points: >> >> 1. high number of pipe readers and waiters for file locks. It may be >> normal for your load. >> >> 2. 2 Giant holders/lockers. Is it constant ? Are the processes holding/waiting >> for Giant are the same ? >> >> Anyway, being in your shoes, I would start looking at applications. >> >> Ah, and does dmesg show anything ? > > And another question: what are the processes in the state "user" ? > I never see that state. More, search thru the sources does not show > what this could be. Odd, I'm not finding any, but, I did get a Giant on a grep of the ps listing:: pluto# ps axlww | grep " user " 0 93055 46540 0 96 0 348 212 Giant L+ p4 0:00.00 grep user Not sure where those 'user' came from though ... just ran the above again: # ps axlww | awk '{print $9}' | sort | uniq -c | sort -nr 603 select 231 lockf 71 nanslp 33 - 30 kserel 23 wait 9 ttyin 9 sbwait 7 pause 6 accept 4 piperd 3 psleep 3 kqread 3 Giant 1 syncer 1 sdflus 1 ppwait 1 pgzero 1 ktrace 1 MWCHAN And nothing ... Got a Giant lock on sshd too? pluto# ps axlww | grep Giant 0 693 556 1 96 0 6096 2080 Giant Ls ?? 0:02.18 sshd: root@ttyp0 (sshd) 0 94334 46540 0 96 0 348 208 - R+ p4 0:00.00 grep Giant ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664