From owner-freebsd-stable@FreeBSD.ORG Mon Nov 28 14:21:31 2005 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82F5916A420 for ; Mon, 28 Nov 2005 14:21:31 +0000 (GMT) (envelope-from ltning@anduin.net) Received: from anduin.net (anduin.net [212.12.46.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id B417D43D49 for ; Mon, 28 Nov 2005 14:21:30 +0000 (GMT) (envelope-from ltning@anduin.net) Received: from eirik.unicore.no ([213.225.74.166] helo=[10.0.16.10]) by anduin.net with esmtpa (Exim 4.50 (FreeBSD)) id 1Egjsn-000PC1-B7 for stable@freebsd.org; Mon, 28 Nov 2005 15:21:29 +0100 Resent-Message-Id: <20187843-76FC-4EAB-AFF8-7493FB0C0077@anduin.net> Mime-Version: 1.0 (Apple Message framework v746.2) Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed Resent-Date: Mon, 28 Nov 2005 15:21:28 +0100 Message-Id: <8C9B3023-EEBD-47DF-87AD-E0494E86B17A@anduin.net> Content-Transfer-Encoding: quoted-printable Resent-To: stable@freebsd.org From: =?ISO-8859-1?Q?Eirik_=D8verby?= Resent-From: =?ISO-8859-1?Q?Eirik_=D8verby?= Date: Mon, 28 Nov 2005 15:21:08 +0100 To: Joseph Koshy X-Mailer: Apple Mail (2.746.2) Cc: Subject: Re: Reduced java/tomcat performance 6-beta3 -> 6-stable ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2005 14:21:31 -0000 On Nov 28, 2005, at 14:45 , Joseph Koshy wrote: > On 11/26/05, Eirik =D8verby wrote: > E=D8> [Cross-posting after lack of response on -stable] > > The first step would be do some performance debugging. Yep. > - What do top/vmstat/systat say about what the OS and > apps are doing? Is the CPU pegged at 100%? What's > the load seen by the disks? Is the RAID in good health? vmstat during system idle times are found below. I think they are =20 rather interesting. To your other questions: The CPU usage is =20 comparable on both systems. Not pegged at 100%, but load seems to =20 stabilize around 0.5. Disk load is minimal on the application =20 servers, somewhat more on the database servers, but they are not =20 interesting here (they are not the bottle neck, and they perform =20 equally). The RAIDs are in good health on both systems. The vmstat output is interesting. =46rom the "fast" system (6.0-BETA3, ~idle): [root@app_host01] ~# vmstat -w 5 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs =20 us sy id 1 0 0 2439220 38048 14 0 0 0 14 0 0 0 170 141 437 =20 0 0 100 0 0 0 2439220 38028 2 0 0 0 3 0 2 0 192 94 475 =20 0 0 100 0 0 0 2439220 37916 1 0 0 0 6 0 1 0 291 925 926 =20 5 0 94 0 0 0 2439220 37916 0 0 0 0 0 0 0 0 185 91 458 =20 0 0 100 0 0 0 2439220 37820 1 0 0 0 6 0 3 0 289 1163 1124 =20= 6 0 94 0 0 0 2439220 37820 0 0 0 0 0 0 0 0 183 91 454 =20 0 0 100 =46rom the "slow" system (6.0-BETA3, ~idle): [root@app_host02] ~# vmstat -w 5 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs =20 us sy id 0 0 1 2468180 51660 15 0 0 0 18 4 0 0 1048 3200 5130 =20= 0 0 100 0 0 0 2468180 51660 1 0 0 0 0 0 0 0 1004 3068 5063 =20= 0 0 100 0 0 0 2468180 51660 0 0 0 0 0 0 0 0 1003 3094 5057 =20= 0 0 100 0 0 0 2468180 51660 0 0 0 0 0 0 1 0 1005 3068 5065 =20= 0 0 100 0 0 0 2468180 51656 1 0 0 0 0 0 0 0 1002 3090 5054 =20= 0 1 99 0 0 0 2468180 51656 0 0 0 0 0 0 0 0 1002 3064 5053 =20= 0 0 100 *loads* more context switches than on the BETA-3 system. I have not =20 yet tried this during load; I have to wait for the testing window for =20= that. But perhaps this helps? What do I look for next? > - Any unusual messages in /var/log/messages? Any errors > shown by the network interfaces (I'm assuming the > application is using the network). No errors shown that I can determine. > - A brief description of the workload presented by > the app would help. This is a web application (payment gateway) that receives a HTTP =20 POST, does some processing, asks an external service for a piece of =20 information, then returns the gathered information to the client. The =20= call to the external service can be eliminated, but does not change =20 the performance profile. How the application works internally is impossible for me to say; =20 it's 3rd party. I can say, after asking them, that it is "moderately" =20= threaded. Whatever "moderately" threaded. My interpretation is that =20 the heaviest threading happens in tomcat itself, with up to 150 =20 concurrent connection threads running. Thanks, /Eirik > > -- > FreeBSD Volunteer, http://people.freebsd.org/~jkoshy > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-=20 > unsubscribe@freebsd.org" > >