Date: Mon, 16 Jun 2008 16:57:03 +0100 From: Stut <stuttle@gmail.com> To: freebsd-questions@freebsd.org Subject: Re: High load, lots of free memory and processes in devfs state Message-ID: <67B08235-6026-44A6-816F-4C862A84566D@gmail.com> In-Reply-To: <CD3881B1-2A55-48A4-907F-986017DDC09B@gmail.com> References: <CD3881B1-2A55-48A4-907F-986017DDC09B@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Does anybody have any ideas about this problem? So far today I've tried moving memcache off this machine, played with a few settings in httpd.conf but nothing seems to make a difference. Any help would be greatly appreciated. I'm rapidly losing what's left of my hair!! -Stut On 15 Jun 2008, at 18:18, Stut wrote: > Hi, > > I'm having a serious problem with one of my FreeBSD servers. It runs > FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the > hardware and it's all OK. There's approximately 400 Apache processes > running and a 2GB memcached instance. > > Up until last weekend this server was working perfectly at a level > of HTTP traffic higher than it's currently getting. Last week our > database server (separate server) died and had to be rebuilt. While > this was being done this server hosted the database. This has now > been completed and the PHP app is pointing back at the dedicated DB > server. > > Top shows the following... > > last pid: 26838; load averages: 10.22, 14.06, 13.55 up > 2+00:34:47 18:03:43 > 619 processes: 1 running, 618 sleeping > CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, > 70.0% idle > Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, > 1747M Free > Swap: 8192M Total, 124K Used, 8192M Free > > PID UID THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81% > httpd > 26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19% > httpd > 26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85% > httpd > 26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84% > httpd > 26780 80 1 -4 0 79892K 10036K devfs 0 0:00 1.51% > httpd > 26801 80 1 -4 0 81292K 11104K devfs 3 0:00 1.50% > httpd > 26786 80 1 -4 0 80456K 10796K devfs 1 0:00 1.41% > httpd > 26784 80 1 -4 0 79892K 10036K devfs 0 0:00 1.41% > httpd > 26785 80 1 -4 0 81304K 11228K devfs 0 0:00 1.41% > httpd > 26763 80 1 -4 0 83220K 13752K devfs 1 0:00 1.24% > httpd > ...etc... > > As you can see there's plenty of free memory and the CPU is 70% idle > yet the load is sky high. When it's like this it's impossible to do > anything - every command can take anything from a few seconds to a > few minutes to respond - and the web user experience is shot to > pieces. I can't find a reference that explains what the devfs state > indicates. > > SA shows the following... > > root@harold:~# sa > 150107 242633.24re 1548.32cp 13avio 801k > 31174 192785.02re 1488.13cp 32avio 502k httpd* > 7047 6286.71re 21.53cp 67avio 2854k php > 7703 164.83re 14.65cp 3avio 743k convert > 8 19.82re 9.47cp 5016avio 1k gzip > 7703 177.66re 7.70cp 2avio 1352k composite > 4 18.90re 3.69cp 1avio 3k mysqldump > 9 0.80re 0.76cp 23avio 144k bzip2 > 6 4.16re 0.53cp 22998avio 1731k find > 2890 1126.31re 0.45cp 0avio 175988k perl* > 5415 40.83re 0.32cp 11avio 123676k rateup > 3009 14948.72re 0.32cp 3avio 9512k smtp > 247 1812.85re 0.27cp 440avio 945k cleanup > 7 163.69re 0.18cp 0avio 105k top > 167 1809.71re 0.11cp 183avio 1117k pickup > 2667 797.34re 0.07cp 0avio 29671k trivial- > rewrite > 1820 96.40re 0.02cp 0avio 141901k ps > 17 1482.98re 0.02cp 1avio 1083k sshd* > 35629 6318.10re 0.01cp 0avio 18016816k sh > 15232 391.68re 0.01cp 0avio 935430k sendmail > 599 1223.30re 0.01cp 28avio 44065k bounce > 7 0.95re 0.01cp 8avio 596k sa > 35 1952.66re 0.01cp 11avio 16013k ***other > 25 0.07re 0.01cp 18avio 1447k rm > 15232 203.94re 0.00cp 7avio 2994436k postdrop > 61 1.20re 0.00cp 5avio 34481k git > 3 123.65re 0.00cp 12avio 796k mail > 92 345.77re 0.00cp 0avio 34405k scache > 31 2.13re 0.00cp 66avio 7115k newsyslog > 6 2140.32re 0.00cp 14avio 6835k bash > 5463 290.01re 0.00cp 0avio 2584397k grep > 5 1479.06re 0.00cp 17avio 3878k sshd > 1830 99.75re 0.00cp 0avio 1221440k wc > 4 2.86re 0.00cp 2avio 13472k less > 3 0.46re 0.00cp 0avio 10048k pgrep > 68 27.44re 0.00cp 0avio 69632k sh* > 3 5.06re 0.00cp 1avio 13120k showq > 59 109.55re 0.00cp 0avio 243008k smtpd > 183 0.54re 0.00cp 2avio 518400k unlink > 2 1.61re 0.00cp 33avio 4608k whereis > 12 0.00re 0.00cp 0avio 35000k adjkerntz > 50 189.34re 0.00cp 0avio 165000k anvil > 364 1.27re 0.00cp 0avio 947300k atrun > 3 0.00re 0.00cp 0avio 53100k awk > 10 0.05re 0.00cp 0avio 298800k bash* > 27 4.15re 0.00cp 0avio 62000k cat > 7 0.00re 0.00cp 0avio 44800k cmp > 40 0.01re 0.00cp 2avio 79100k cp > 2937 5227.93re 0.00cp 0avio 5143200k cron* > 6 0.00re 0.00cp 0avio 22800k date > 183 14.68re 0.00cp 2avio 578900k dd > 5 0.00re 0.00cp 0avio 10600k egrep > 132 241.91re 0.00cp 0avio 552400k error > 6 0.05re 0.00cp 0avio 0k expr > 245 444.67re 0.00cp 1avio 1181500k flush > 3 0.00re 0.00cp 0avio 0k hostname > 8 0.03re 0.00cp 0avio 21000k id > 4 0.04re 0.00cp 0avio 173800k ipcs > 183 0.49re 0.00cp 0avio 670000k jot > 16 0.35re 0.00cp 1avio 65900k ls > 3 1.52re 0.00cp 11avio 16300k man > 2 0.06re 0.00cp 3avio 6800k manpath > 10 0.00re 0.00cp 2avio 29200k mktemp > 2 0.00re 0.00cp 0avio 0k mount > 1287 4.64re 0.00cp 3avio 3717800k mv > 4 0.12re 0.00cp 0avio 17600k postqueue > 57 10.01re 0.00cp 0avio 242300k proxymap > 7 4.19re 0.00cp 0avio 60500k sed > 7 4.18re 0.00cp 0avio 10100k sort > 7 0.05re 0.00cp 0avio 6100k sysctl > 9 0.00re 0.00cp 0avio 15000k tee > 4 26.15re 0.00cp 0avio 9300k time > 2 0.48re 0.00cp 0avio 25400k zcat > > I'm fairly new to SA but I can't see anything in there that > indicates a problem. If I'm missing something please let mw know. > > If anyone has any ideas as to what might be causing this and how to > fix it I'd appreciate it. I'm getting close to giving up and simply > reinstalling it. > > -Stut > > -- > http://stut.net/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?67B08235-6026-44A6-816F-4C862A84566D>