From owner-freebsd-questions@FreeBSD.ORG Fri Dec 2 00:10:10 2005 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BB85616A41F for ; Fri, 2 Dec 2005 00:10:10 +0000 (GMT) (envelope-from freebsd@meijome.net) Received: from sigma.octantis.com.au (ns2.octantis.com.au [207.44.189.124]) by mx1.FreeBSD.org (Postfix) with ESMTP id B8F9543D5E for ; Fri, 2 Dec 2005 00:10:09 +0000 (GMT) (envelope-from freebsd@meijome.net) Received: (qmail 12081 invoked from network); 2 Dec 2005 11:10:08 +1100 Received: from andromeda.lef.com.au (HELO ?10.168.101.24?) (210.8.93.2) by sigma.octantis.com.au with (DHE-RSA-AES256-SHA encrypted) SMTP; 2 Dec 2005 11:10:08 +1100 Message-ID: <438F9019.7020309@meijome.net> Date: Fri, 02 Dec 2005 11:06:49 +1100 From: Norberto Meijome User-Agent: Mozilla Thunderbird 1.0.7 (Windows/20050923) X-Accept-Language: en-us, en MIME-Version: 1.0 To: "N.J. Thomas" References: <20051201193813.GG15171@ayvali.org> In-Reply-To: <20051201193813.GG15171@ayvali.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-questions@freebsd.org Subject: Re: overloaded webserver: nfs wait issue? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2005 00:10:10 -0000 N.J. Thomas wrote: > We have a website with moderately high traffic, load balanced among 3 > webservers. > > During peak traffic times however (when the volume is higher than > normal), the load shoots up to over a 100, and the site crawls to its > knees. > > We set up a script to take snapshots of top every 20 seconds. Here is > what it looks like when everthing is normal: > > 127 > last pid: 12003; load averages: 0.93, 1.36, 1.35 up 41+04:22:14 14:00:23 > 243 processes: 12 running, 230 sleeping, 1 zombie > > Mem: 222M Active, 74M Inact, 186M Wired, 16M Cache, 111M Buf, 503M Free > Swap: 2048M Total, 16M Used, 2032M Free > > [...] > When the load shoots up, the number of http clients hits Apache's > MaxClients setting, here is what top shows: > > last pid: 12407; load averages: 87.84, 51.91, 27.52 up 41+04:40:51 14:19:00 > 268 processes: 2 running, 266 sleeping > > Mem: 715M Active, 68M Inact, 187M Wired, 29M Cache, 111M Buf, 2100K Free > Swap: 2048M Total, 272M Used, 1776M Free, 13% Inuse > [...] > > The state of all the httpd prcesses are "nfsrcv". Does this mean the > bottleneck is at the NFS server that hosts the htdocs (and PHP scripts) > or just that the server is low on memory? Hi Thomas, What's your MaxClients set to? Please define your values for "lot of traffic". What CPU? RAM (512MB seems a bit low nowadays)? Disks? I dont think i can give much advice on the NFS side of things but in the meantime I would : - increase # of MaxClients (the default is RIDICULOUSLY small, specially in 1.3. You will probably have to recompile with a new max. - Look at what the PHP scripts do : i.e., is there anything under your control that can be improved? - Install a PHP accelerator (like IonCube's, or Zend's). Or some reverse proxies if you feel like playing with the whole design of your architecture. - if you use mod_gzip, make sure you dont' compress on the fly - it takes a huge hit on the CPU. (Hint: You can pre-compress the files ) - You RAM seems OK ... you may want to tweak some sysctl or memory settings in Apache (I seem to remember in 1.3 some to do with MMap, but i could be wrong) ... or just add more RAM. Check vmstat (or systat -vm 1) to see how much swapping is going on. - Make sure you have tweaked your network sysctl settings (like send + receive buffers, depending on the kind of traffic you get), reduce your close timeouts,etc. - make use of the kernel modules accf_http.ko and accf_data.ko. - disable from Apache ANYTHING that is NOT needed. Enable stuff only as needed on a per virtual host basis. Hope this helps. Beto