From owner-freebsd-apache@FreeBSD.ORG Wed Feb 15 16:15:30 2006 Return-Path: X-Original-To: freebsd-apache@freebsd.org Delivered-To: freebsd-apache@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 60A4016A422 for ; Wed, 15 Feb 2006 16:15:30 +0000 (GMT) (envelope-from bsdlists@celeritystorm.com) Received: from mail.celeritystorm.com (213-247-62-79.ip.netshark.nl [213.247.62.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9E5A243D4C for ; Wed, 15 Feb 2006 16:15:29 +0000 (GMT) (envelope-from bsdlists@celeritystorm.com) Received: by mail.celeritystorm.com (Postfix, from userid 106) id 5558B3D80FE; Wed, 15 Feb 2006 17:39:12 +0100 (CET) Received: from [192.168.0.1] (unknown [81.84.174.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.celeritystorm.com (Postfix) with ESMTP id BD7C23D7A51 for ; Wed, 15 Feb 2006 17:39:08 +0100 (CET) Message-ID: <43F3539E.5080007@celeritystorm.com> Date: Wed, 15 Feb 2006 16:15:26 +0000 From: BSD User-Agent: Mozilla Thunderbird 0.7.2 (X11/20040724) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-apache@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Apache 1.3 + PHP 5 on FreeBSD 6 -- bad performance under load X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2006 16:15:30 -0000 Hello list, I've been having trouble identifying what's causing my apache server to take about 5 seconds to reply to new http requests. The machine in question is a Dual Xeon 2.8GHz with 4GB RAM, running apache 1.3.33_4 and PHP5 on 6.0-RELEASE-p4. It is a very busy PHP-based site. I have done a lot of tweaking on this server, but when the load goes a bit higher, it'll take about 5 seconds to answer to a standard GET / HTTP/1.1 via telnet (no PHP involved, since I'm not sending valid HTTP headers and it will reply with a 400 bad request). It is a problem with apache, as telnetting to the sshd port and hitting ENTER yields an instant "Protocol mismatch". -- no delay whatsoever. Both CPUs are around 65% idle after some PHP tweaks, so it's not CPU starvation either. I have accept filters enabled on the kernel, and apache is using them; also: options KVA_PAGES=768 options PMAP_SHPGPERPROC=5120 - to avoid that "colleting PV entries - suggest increasing PMAP_SHPGPERPROC". So far I've been unable to pinpoint what's causing apache a ~5 sec delay to process the GET. A timestamped tcpdump shows the following: Establishing the connection: 02/15/2006 16:06:44.232973 192.168.0.1.53578 > APACHE_SERVER.80: S 3958791824:3958791824(0) win 65535 (DF) [tos 0x10] 02/15/2006 16:06:44.387141 APACHE_SERVER.80 > 192.168.0.1.53578: S 2261546731:2261546731(0) ack 3958791825 win 65535 (DF) 02/15/2006 16:06:44.387348 192.168.0.1.53578 > APACHE_SERVER.80: . ack 1 win 33304 (DF) [tos 0x10] 02/15/2006 16:06:44.536819 APACHE_SERVER.80 > 192.168.0.1.53578: . ack 1 win 32942 (DF) GET: (note: delay from previous step was induced by manually switching terminals and typing) 02/15/2006 16:06:49.148143 192.168.0.1.53578 > APACHE_SERVER.80: P 1:17(16) ack 1 win 33304 (DF) [tos 0x10] 02/15/2006 16:06:49.395079 APACHE_SERVER.80 > 192.168.0.1.53578: . ack 17 win 32938 (DF) 02/15/2006 16:06:49.395284 192.168.0.1.53578 > APACHE_SERVER.80: P 17:19(2) ack 1 win 33304 (DF) [tos 0x10] 02/15/2006 16:06:49.642127 APACHE_SERVER.80 > 192.168.0.1.53578: . ack 19 win 32937 (DF) "400 bad request" reply: 02/15/2006 16:06:59.769277 APACHE_SERVER.80 > 192.168.0.1.53578: P 1:481(480) ack 19 win 32942 (DF) 02/15/2006 16:06:59.769453 APACHE_SERVER.80 > 192.168.0.1.53578: F 481:481(0) ack 19 win 32942 (DF) 02/15/2006 16:06:59.769543 192.168.0.1.53578 > APACHE_SERVER.80: . ack 482 win 33064 (DF) [tos 0x10] 02/15/2006 16:06:59.770548 192.168.0.1.53578 > APACHE_SERVER.80: F 19:19(0) ack 482 win 33304 (DF) [tos 0x10] 02/15/2006 16:06:59.923720 APACHE_SERVER.80 > 192.168.0.1.53578: . ack 20 win 32941 (DF) As you can see, it took 10 seconds to reply to the GET, yet the accepting of the connection was instant. I have more than enough maxfiles, maxproc, maxfilesperproc, nmbclusters etc. I guess the connection is so promptly answered because of the http accept filters built in the kernel. As a final note, the MaxClients I'm using are not nearly enough (around 200 connections/second, according to pfstat graphics vs MaxClients 98). The reason for this is that trying to match MaxClients with the real load resulted in two 0.0% idle processors, since the PHP code is a bit heavy. I've opted to disable KeepAlive and lower MaxClients instead. Could this be the delay I'm seeing ? I think so, but I need more opinions, so feel free to reply.