From owner-freebsd-questions@FreeBSD.ORG Wed Nov 12 00:23:17 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D2361065670 for ; Wed, 12 Nov 2008 00:23:17 +0000 (UTC) (envelope-from pprocacci@datapipe.com) Received: from fmailhost06.isp.att.net (fmailhost06.isp.att.net [207.115.11.56]) by mx1.freebsd.org (Postfix) with ESMTP id 0A2758FC17 for ; Wed, 12 Nov 2008 00:23:17 +0000 (UTC) (envelope-from pprocacci@datapipe.com) Received: from [10.5.21.122] (adsl-241-169-92.bna.bellsouth.net[74.241.169.92]) by isp.att.net (frfwmhc06) with ESMTP id <20081112002316H0600ruv37e>; Wed, 12 Nov 2008 00:23:16 +0000 X-Originating-IP: [74.241.169.92] Message-ID: <491A21E8.2080209@datapipe.com> Date: Tue, 11 Nov 2008 18:23:04 -0600 From: Paul Procacci User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: freebsd-questions@freebsd.org References: <491A17B9.7080705@datapipe.com> In-Reply-To: <491A17B9.7080705@datapipe.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Tomcat Debugging (OT most likely) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2008 00:23:17 -0000 Paul Procacci wrote: > Hey all, > > This question is probably best suited for tomcat-users, but being this > is an a FBSD machine I'd figure I'd ask here first. I have a client > running apache w/ tomcat serving thousands of requests per second. It > works, but with one caveat, pages load extremely slowly. When I first > saw the request, I figured this client was running into some kind of > resource bottleneck, but that's where my problems started. I'm unable > to find any resource that is being starved. I turn to `the mighty > list` for guidance. > > This particular machine is running FreeBSD 6.2. It has a bce network > card that currently connected to a GigE switch with a 10G uplink. > Even with all this available bandwidth, this client's application > *currently* is not doing any more than 50Mb/sec. :: Rule out network > congestion. > > Moving along to the system ram :: 2 Gigs are in the machine, with very > little usage: > Mem: 724M Active, 874M Inact, 338M Wired, 64M Cache, 112M Buf, 4944K Free > > A dual core cpu is in the machine, again with very little usage: > CPU states: 8.6% user, 0.0% nice, 3.9% system, 2.8% interrupt, > 84.7% idle > > Disk throughput is negligable at this time (50KB/sec :: 3tps). > > I did start off mentioning this machine uses tomcat...so let me > continue. We've setup the java process to use libthr via libmap.conf: > ############# > cat /etc/libmap.conf > [java] > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > ############# > > And added the following to rc.conf:: > ############### > tomcat55_java_opts="-XX:MaxPermSize=512m -Xmx512M" > ############### > > Apache Configuration: > ################### > MaxClients 1024 > ################### > > Tomcat Configuration: > ################### > maxThreads="1200" > ################### > > ---------------------------------------------------------------- > So onto my question...the slowness being encountered. My initial > thought was the slowness was due to quite possibly the cpu being > hammered by numerous requests. After checking however that's not the > case, then I looked at the networking equipment with my networking > team, that's ok too. (I think) That leaves a) thread contention > somewhere b) apache misconfiguration and c) tomcat configuration. > > Both myself and 2 other admins have looked over the apache > configuration and tomcat configuration and we believe that side of > things is probably ok. That's leaves weird contention in the kernel > or userland mutexes or something along those lines. > > Here is our current connection count on the external interface :: > ################################ > sockstat -4cp 80 | wc -l > 997 > ################################ > > Here is our connection count on localhost (from apache to tomcat > process and vica versa) :: > ################################ > sockstat -4cp 8009 | wc -l > 1679 > ################################ > ---------------------------------------------------------------- > > And finally.....top output: > > last pid: 57747; load averages: 0.79, 1.12, > 0.87 > up 153+17:56:53 18:36:57 > <#--snip --> > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 57460 www 96 0 1385M 432M ucond 0 2:44 2.88% java > 57460 www 96 0 1385M 432M ucond 0 2:44 2.73% java > 57460 www 96 0 1385M 432M select 0 2:44 0.15% java > 57460 www 96 0 1385M 432M select 1 2:44 0.15% java > <#--snip--> > > The above lines repeat up to the number of threads we have. `states` > are in ucond and select. > ---------------------------------------------------------------- > > Hopefully I hven't bored anyone....yet, and would appreciate any > guildance. Maybe good tools for debugged kernel threads, or a simple > slap in the face will do nicely. > > Thanks in advance, > Paul > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" List, Apologies, but I sever my head in shame as I have found the problem. In one word: KeepAlives. Thanks!