From owner-freebsd-questions@FreeBSD.ORG Wed Jul 18 22:40:55 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4696016A4CE for ; Wed, 18 Jul 2007 22:40:55 +0000 (UTC) (envelope-from x0dapara@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.231]) by mx1.freebsd.org (Postfix) with ESMTP id EBF4413C4B8 for ; Wed, 18 Jul 2007 22:40:54 +0000 (UTC) (envelope-from x0dapara@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so283166nzf for ; Wed, 18 Jul 2007 15:40:54 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=TopHEri9Z6j4UTAuzsks/tJ2G21eG4lPSPp+fGr8RyYJ1lGnt5uS4FCquFs3p9Tsfhwe+u+WkVFcVxDIc8/bjydgsgBAUCpF8cYaDtZAO4wWS8OTzIyDsU+CCkECJv5/SeBJdxiUKxQgqkTR/l0yvG0gtXNYNHnxNK+SXLQIyso= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=dUBxkEKNORz5vuaU4CaGxdaPiyjcwFRA+fzV0Cgm9kD4sfJu8glUByNTZmZBq70du+2hgfSRlMRRst39gxg7IhiJ49N54lDW/X+szJuQSHYwR0mOTBd6jKGQIhIejWcze5tQAV6kh2KgFQ/W2UXpNZCVJmOVAAO8QBo+uxfGDGI= Received: by 10.142.103.6 with SMTP id a6mr159973wfc.1184796778243; Wed, 18 Jul 2007 15:12:58 -0700 (PDT) Received: by 10.143.44.2 with HTTP; Wed, 18 Jul 2007 15:12:57 -0700 (PDT) Message-ID: <2f0146460707181512x3841af57l588e4d6e67bd5884@mail.gmail.com> Date: Wed, 18 Jul 2007 23:12:57 +0100 From: "Michael Vaughn" To: freebsd-questions@freebsd.org, freebsd-performance@freebsd.org, freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help! X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2007 22:40:55 -0000 Hello everyone, I am contacting -performance, -questions, and -hackers in the hope someone helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 uname: FreeBSD 6.2-STABLE Fri Jun 22 12:17:03 UTC 2007 amd64 installed php modules: php5-5.2.3 PHP Scripting Language (Apache Module and CLI) php5-gd-5.2.3 The gd shared extension for php php5-mysql-5.2.3 The mysql shared extension for php php5-pcre-5.2.3 The pcre shared extension for php php5-session-5.2.3 The session shared extension for php php5-simplexml-5.2.3 The simplexml shared extension for php php5-tokenizer-5.2.3 The tokenizer shared extension for php php5-xml-5.2.3 The xml shared extension for php apache version: apache-2.2.4_2 Version 2.2 of Apache web server with prefork MPM. system: real memory = 5100273664 (4864 MB) avail memory = 4120178688 (3929 MB) CPU: Intel(R) Xeon(TM) CPU 2.66GHz (2666.78-MHz K8-class CPU) Logical CPUs per core: 2 FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs The problem: Right after starting apache, the loads on the server will climb to 10-40's and the application will become unacceptably slow. This will go on until few users are using the said application. (note: other servers running older FreeBSD versions on dual cpus running the same code don't exhibit this system% problem) top shows more than 60% of the CPU time is spent on system: CPU states: 19.9% user, 0.0% nice, 73.7% system, 1.7% interrupt, 4.7% idle Mem: 398M Active, 2226M Inact, 253M Wired, 202M Cache, 214M Buf, 567M Free The apache processes look like: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56882 www 1 103 0 139M 17516K select 0 0:03 12.66% httpd 56862 www 1 100 0 139M 21168K CPU2 6 0:06 11.87% httpd 56830 www 1 99 0 138M 19684K select 2 0:09 10.76% httpd 56887 www 1 105 0 139M 17488K select 6 0:01 10.49% httpd 56852 www 1 99 0 138M 20352K select 4 0:06 10.26% httpd 56889 www 1 106 0 139M 17548K select 6 0:01 10.04% httpd 56894 www 1 109 0 139M 17024K select 6 0:01 9.79% httpd 56839 www 1 99 0 138M 21216K select 6 0:06 9.36% httpd 56866 www 1 99 0 138M 17664K select 6 0:04 9.36% httpd 56890 www 1 108 0 138M 16180K select 4 0:01 9.29% httpd 56848 www 1 99 0 138M 20460K select 2 0:06 9.27% httpd 56865 www 1 99 0 138M 18920K select 2 0:05 9.23% httpd 56883 www 1 102 0 138M 16744K select 4 0:02 8.99% httpd 56870 www 1 100 0 139M 18440K select 2 0:03 8.86% httpd 56850 www 1 98 0 138M 21284K select 6 0:05 8.84% httpd 56860 www 1 99 0 138M 19584K select 0 0:05 8.70% httpd 56864 www 1 99 0 139M 18028K select 2 0:04 8.23% httpd 56854 www 1 99 0 138M 20696K select 6 0:05 8.23% httpd 56853 www 1 98 0 138M 19564K select 4 0:06 8.11% httpd 56835 www 1 98 0 139M 20276K CPU6 4 0:07 8.10% httpd 56849 www 1 98 0 138M 19532K select 0 0:05 7.95% httpd 56851 www 1 98 0 139M 20252K select 4 0:05 7.35% httpd 56888 www 1 4 0 139M 17100K sbwait 6 0:01 7.31% httpd 56869 www 1 100 0 139M 18632K select 4 0:02 6.75% httpd 56861 www 1 98 0 139M 18404K select 0 0:04 6.58% httpd 56863 www 1 98 0 139M 20220K select 2 0:03 6.40% httpd 56867 www 1 99 0 138M 17452K select 6 0:03 6.39% httpd 56868 www 1 99 0 138M 18376K select 0 0:03 6.20% httpd 56893 www 1 107 0 138M 12964K select 0 0:00 5.62% httpd 56878 www 1 100 0 138M 16732K select 6 0:02 5.27% httpd 56881 www 1 100 0 138M 16288K select 6 0:01 2.18% httpd I had to lower MaxClients on apache substancially from 128 to 32, or loads would quickly go to 40+. (Other servers with dual cpus instead of quad and apache 1.3 on freebsd 6.0 don't have this problem) vmstat 1: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 0 1 0 1380860 787212 1365 0 0 0 1312 1 0 0 486 559 842 13 22 65 1 1 0 1384588 787128 2724 0 0 0 2581 0 0 88 3038 82956 48776 19 38 43 4 1 0 1399232 782936 3328 0 0 0 2112 0 0 97 3592 101093 66497 24 50 26 0 1 2 1400200 781628 3726 0 0 0 2910 0 0 99 3529 100289 81531 23 58 19 19 1 0 1404000 778556 2263 0 0 0 1141 0 0 62 2964 73572 101432 19 76 5 15 1 1 1402452 776800 2499 0 0 0 1714 0 7 74 2965 68441 102276 19 78 3 15 1 0 1401548 777112 2213 0 0 0 2103 0 0 42 2491 105584 109418 15 79 6 8 1 1 1403324 778856 2606 0 0 0 2748 0 0 84 2996 75288 91676 22 76 2 0 1 3 1396864 781344 2764 0 0 0 3010 0 0 86 3393 90765 85952 25 70 5 1 2 0 1395520 782604 2774 0 0 0 2978 0 0 79 3195 88251 92623 20 63 17 6 1 0 1396096 781832 2641 0 0 0 2195 0 1 82 3347 96322 55942 21 42 37 iostat 1: tty ad4 ad6 ad8 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 28 13.94 4 0.06 16.13 48 0.75 13.94 4 0.06 13 0 21 1 65 0 231 0.00 0 0.00 16.00 68 1.06 0.00 0 0.00 19 0 74 1 5 0 77 0.00 0 0.00 16.00 90 1.40 0.00 0 0.00 17 0 77 2 4 0 77 0.50 1 0.00 16.00 46 0.72 0.50 1 0.00 14 0 82 1 4 0 77 0.00 0 0.00 16.00 83 1.30 0.00 0 0.00 21 0 65 2 12 0 77 0.00 0 0.00 16.00 37 0.58 0.00 0 0.00 18 0 76 1 5 0 77 0.00 0 0.00 16.00 82 1.28 0.00 0 0.00 20 0 74 2 4 0 77 0.00 0 0.00 16.00 68 1.06 0.00 0 0.00 21 0 47 2 30 0 77 0.00 0 0.00 16.00 61 0.95 0.00 0 0.00 20 0 33 1 46 0 77 0.00 0 0.00 16.00 71 1.11 0.00 0 0.00 21 0 44 2 33 0 77 0.00 0 0.00 16.00 62 0.97 0.00 0 0.00 22 0 46 2 30 The kernel is custom built, with the following added (relevant) options: options PMAP_SHPGPERPROC=4096 options ACCEPT_FILTER_HTTP /boot/loader.conf: kern.maxproc=32768 kern.ipc.shmmni=4096 kern.ipc.shmseg=2048 kern.ipc.semmns=960 kern.ipc.semmni=160 kern.ipc.semume=160 kern.ipc.semmnu=480 mysqld is also running on this server, altough it never goes past 8-10% CPU usage. It is separated from apache on I/O, via jails and gmirror: mirror/gm0 COMPLETE ad4 ad8 mirror/gm1 COMPLETE ad6 ad10 Now this web application isn't the best code out there, but this is a quad cpu server and it's performing a lot worse than some servers I have running with 6.0 with apache 1.3 for over 400 days. Am I the only one getting terrible performance with apache2 on FreeBSD 6 ? p.s: cc me, I am not subscribed to any of the lists. Regards, Mark