From owner-freebsd-performance@FreeBSD.ORG Wed Nov 7 00:11:00 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 858B816A46E for ; Wed, 7 Nov 2007 00:11:00 +0000 (UTC) (envelope-from mgowda82@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.229]) by mx1.freebsd.org (Postfix) with ESMTP id 3715E13C4B2 for ; Wed, 7 Nov 2007 00:10:59 +0000 (UTC) (envelope-from mgowda82@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so1361215nzf for ; Tue, 06 Nov 2007 16:10:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=vuoXq75rmc/KM8JlgF5K9ce6ozXZShNNRp14ONNVoYA=; b=LiWyrhtkamMcYOgpwQo0p+SW/ZJ+INFK1y5iUhM7jlFAL4iADjUzcbZZqLucwaKhSObAFBsw7W+7ZZ1U/9ga9AB67+3cczcPFOa1SOW3++NriE5LmoO4suptnRFsBclzYH7fVi/EstkBb2DDVbmCU/P7WqPzJgpYhKlVog3lbd4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=jbPNMpxuoaswRn1aWStMDiDg7mZF41vHl25DOYG1naS7Jqi3uuzgtR5slLKHtg7zWbnLK3UIJycfZn9PfgCcTKlBXHgWdF53jFd/CLvkOahr9AcNhO8W731UEXGR6ZSsU3BI4hGnh0rCSapjwSmyk0KnHmeIzvLQBAVuJ9HI7wY= Received: by 10.142.148.7 with SMTP id v7mr551090wfd.1194394256198; Tue, 06 Nov 2007 16:10:56 -0800 (PST) Received: by 10.142.13.10 with HTTP; Tue, 6 Nov 2007 16:10:56 -0800 (PST) Message-ID: Date: Tue, 6 Nov 2007 16:10:56 -0800 From: "Manjunath R Gowda" To: "Nico -telmich- Schottelius" In-Reply-To: <20071106193305.GJ15790@schottelius.org> MIME-Version: 1.0 References: <20071106122820.GA28254@schottelius.org> <20071106193305.GJ15790@schottelius.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Performance of disk i/o with 3ware X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2007 00:11:00 -0000 On 11/6/07, Nico -telmich- Schottelius < nico-freebsd-performance@schottelius.org> wrote: > > Manjunath R Gowda [Tue, Nov 06, 2007 at 09:54:09AM -0800]: > > On 11/6/07, Nico -telmich- Schottelius < > > nico-freebsd-performance@schottelius.org> wrote: > > > > > > Hello! > > > > > > I've the problem that sometimes there are many disk waiting processes > > > (sysctl -n vm.vmtotal), but systat -vmstat shows da0 and da1 busy with > > > 0-10%. > > > > I woudn't worry about how many are sleeping. But, How long they sleep > would > > be interesting data. > > And also why they sleep. Having about 700 processes in sleep state plus > 400 processes in diskwait state makes me wonder somehow. At a given time only few can run, remaining once will sleep which is an expected behavior. There are also some processes from cron, which are older and also in > state "I" (as shown by ps). You have to dig deep and see what those cron jobs do. > > I guess that the disk i/o is at about 100%, but wondering why I see > > > those strange values. > > > > Why do you think it should be 100%? Are you running any disk I/O intense > > application? > > Because it's the well known bottleneck in that server: > > - it has 4 cpus, which are about 30-50% idle > - it has 4 GiB ram, of which 2GiB is mostly inactive > - it has 4 10k rpm hds in 2 raid1 disk arrays, one for > the mailboxes + root and one for the qmail-queue > > Normally systat -vmstat shows 80-100% busy state on the disk arrays, > but currently it's at about 10%, which cannot be right. > > I think I'll have a look closer look at the patch we used from CVS to > patch the > 3ware twa driver. Looks like your blaming the twa driver based on your experience. OS is a complex piece of software and involves many components, scheduler, vm, scsi-stack, driver, problem could be any where in this. That's why specific data would be useful. Something like on 6.2 running x takes y and on 6.2+patch it takes z would be useful. > > Anyone an idea, > > > a) why systat -vmstat shows so small busy values? > > > b) how to debug it further? > > > > You can try experimenting with diffrent I/O loads to make sure that > there > > is a problem before start debugging it. > > There is a big problem, that even results in taking about 30 seconds > until the '220' messages comes from qmail when connecting via telnet > from outside to it. Your adding one more variable, network here. Verify that when you did cvsup the network driver was not updated. To summarise, what runs on the server: > > - qmail + patches => has /var/qmail/queue on it's own disk-array > - courier imapd > - vpopmail (pop3 auth) > - mysql (needed by vpopmail) > - clamd > - spamassassin > - Webmail (horde on apache) > > So it's mainly a customer mailserver, with nothing special installed, > which may have some load, but not in the way it's not explainable. > > So I'm still hoping somebody with similar problems reads this mail ;-) > > Sincerly > > Nico > > -- > Think about Free and Open Source Software (FOSS). > http://nico.schottelius.org/documentations/foss/the-term-foss/ > > PGP: BFE4 C736 ABE5 406F 8F42 F7CF B8BE F92A 9885 188C > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFHMMFxuL75KpiFGIwRAoKxAKDjh7aN8n/b7B3YPqcdKuqMPv9CGACgsGuf > lI6wV4jUuP8GnUhd37oW/mQ= > =4Yn4 > -----END PGP SIGNATURE----- > >