From owner-freebsd-questions@FreeBSD.ORG Mon Jun 16 21:28:09 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B88F1065679 for ; Mon, 16 Jun 2008 21:28:09 +0000 (UTC) (envelope-from stuttle@gmail.com) Received: from ik-out-1112.google.com (ik-out-1112.google.com [66.249.90.176]) by mx1.freebsd.org (Postfix) with ESMTP id 1FAFE8FC16 for ; Mon, 16 Jun 2008 21:28:08 +0000 (UTC) (envelope-from stuttle@gmail.com) Received: by ik-out-1112.google.com with SMTP id c30so4838137ika.3 for ; Mon, 16 Jun 2008 14:28:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=VEinIGEQRREoeLMh2/A0MRgm64MGW5Bd+KM4Wa/LHvg=; b=TEUclphfxraJVCxN1cJb2Q3Uaec8ER3uOz6Bix4WisgWoqi2Sce2cBg7lUA3ovZ5FJ NVJoX6egT4iftAMhufpG7Uq6HDSyOvyPRGNxiC+UH/38dWqSMJdOTFyPkFI92avUD42R 3YgMA7FGkg2z/7J2ZVRykfVMgurBsnUn7FgW0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=t9kOXHYe7kz26cWyKOPPs+eVZHC5/EL8QvqXhEYg6PQhGOXn9J0Bv+DnzoohlDbzlT APQQNPZR9Cw/MUghMt7EYCoXY3N16ce91WwYHOT3+WiJ+p2BAeGsYt+yPBw5sjjWpvJQ OxpHg5DX/7+L3dyxlx83KQADnYgXgsJPArXvc= Received: by 10.210.10.1 with SMTP id 1mr6629625ebj.167.1213651683526; Mon, 16 Jun 2008 14:28:03 -0700 (PDT) Received: from ?192.168.0.101? ( [91.109.249.101]) by mx.google.com with ESMTPS id g11sm10987412gve.8.2008.06.16.14.28.01 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 16 Jun 2008 14:28:02 -0700 (PDT) Message-Id: From: Stut To: "Patrick C" In-Reply-To: <34394a3a0806161420t1d0f3e79q85e34d3dbb5cda68@mail.gmail.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v924) Date: Mon, 16 Jun 2008 22:27:59 +0100 References: <200806161748.m5GHmgkf006333@lurza.secnetix.de> <6E7BD474-CAF1-4017-B19A-7F039E4463BB@gmail.com> <34394a3a0806161420t1d0f3e79q85e34d3dbb5cda68@mail.gmail.com> X-Mailer: Apple Mail (2.924) Cc: freebsd-questions@freebsd.org Subject: Re: High load, lots of free memory and processes in devfs state X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2008 21:28:09 -0000 On 16 Jun 2008, at 22:20, Patrick C wrote: > Is the MySQL daemon still running on that box? I see a mysqldump but > no mysqld. If it is, try doing a shutdown and see if the load > decreases. > > Sounds odd, but I have been having similar issues with MySQL. It's not, no. It used to but it doesn't run now. The mysqldump is a nightly backup. -Stut > 2008/6/16 Stut : >> On 16 Jun 2008, at 18:48, Oliver Fromme wrote: >>> >>> Stut wrote: >>>> >>>> I'm having a serious problem with one of my FreeBSD servers. It >>>> runs >>>> FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the >>>> hardware and it's all OK. There's approximately 400 Apache >>>> processes >>>> running and a 2GB memcached instance. >>>> >>>> Up until last weekend this server was working perfectly at a level >>>> of HTTP traffic higher than it's currently getting. Last week our >>>> database server (separate server) died and had to be rebuilt. While >>>> this was being done this server hosted the database. This has now >>>> been completed and the PHP app is pointing back at the dedicated DB >>>> server. >>>> >>>> Top shows the following... >>>> >>>> last pid: 26838; load averages: 10.22, 14.06, 13.55 up >>>> 2+00:34:47 18:03:43 >>>> 619 processes: 1 running, 618 sleeping >>>> CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, >>>> 70.0% idle >>>> Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, >>>> 1747M Free >>>> Swap: 8192M Total, 124K Used, 8192M Free >>>> >>>> PID UID THR PRI NICE SIZE RES STATE C TIME WCPU >>>> COMMAND >>>> 26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81% >>>> httpd >>>> 26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19% >>>> httpd >>>> 26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85% >>>> httpd >>>> 26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84% >>>> httpd >>>> [...] >>> >>> Please let "vmstat 5" run for a minute ... Anything >>> that looks unusual? >> >> procs memory page disks >> faults cpu >> r b w avm fre flt re pi po fr sr da0 pa0 in sy >> cs us >> sy id >> 1 412 2 8910132 2743568 2230 3 1 0 2106 93 0 0 1098 4733 >> 3025 4 >> 17 79 >> 1 412 0 8841792 2760548 1630 0 0 0 2276 0 21 0 658 3970 >> 6374 3 >> 26 71 >> 0 422 0 8804744 2765076 1349 0 0 0 1394 0 12 0 576 3454 >> 5131 3 >> 26 71 >> 1 421 0 8756808 2778660 1095 0 0 0 1581 0 48 0 574 3076 >> 4415 3 >> 25 72 >> 0 420 0 8684932 2800128 2049 0 0 0 2927 0 45 0 505 2998 >> 3770 3 >> 24 73 >> 1 380 0 8603676 2838452 1149 0 0 0 2842 0 22 0 505 3497 >> 4339 2 >> 26 72 >> 1 21 0 8280260 3097392 7630 0 0 0 19969 0 29 0 1241 8895 >> 6950 10 >> 30 60 >> 1 4 0 8195828 3131992 9113 0 0 0 9742 0 78 0 996 9405 >> 2830 7 >> 19 74 >> 0 20 0 8169288 3121688 2899 0 0 0 2098 0 47 0 1193 7740 >> 2733 4 >> 23 73 >> 11 13 0 8123476 3128688 1947 0 0 0 2160 0 52 0 1161 6231 >> 2617 4 >> 26 70 >> 1 14 0 8067304 3143984 2298 0 0 0 2885 0 57 0 1806 5572 >> 3373 4 >> 25 70 >> 2 17 1 8015924 3156168 2702 0 0 0 3164 0 23 0 1384 8243 >> 2770 6 >> 25 69 >> 1 22 0 7956476 3176376 2013 0 0 0 2879 0 23 0 917 7063 >> 2484 6 >> 25 69 >> 0 35 0 7944760 3150452 4274 0 0 0 2806 0 21 0 1591 8281 >> 3399 7 >> 25 68 >> 1 67 3 7903160 3158776 2043 0 0 0 2442 0 20 0 1095 6405 >> 3605 6 >> 25 70 >> 44 69 0 7872504 3147192 2569 0 0 0 1712 0 81 0 1137 5773 >> 4998 6 >> 26 69 >> 1 146 0 7849632 3135388 2095 0 0 0 1274 0 19 0 869 5550 >> 5466 5 >> 26 69 >> 3 195 2 7825932 3122116 2482 0 0 0 1586 0 15 0 863 5558 >> 6135 5 >> 26 69 >> 1 244 3 7798148 3111624 1609 0 0 0 1226 0 10 0 849 4027 >> 6477 4 >> 27 69 >> 2 273 3 7776772 3102948 2080 0 0 0 1310 0 14 0 604 4376 >> 6527 4 >> 26 70 >> 1 312 2 7768232 3079012 3148 0 0 0 1800 0 12 0 1066 6088 >> 10242 6 >> 28 66 >> 1 340 0 7742680 3068244 2020 0 0 0 1421 0 74 0 729 4407 >> 8204 4 >> 26 70 >> 2 366 0 7740324 3068068 1612 0 0 0 1553 0 11 0 613 3526 >> 6728 3 >> 26 70 >> 1 397 1 7928688 3059900 1886 0 0 0 1344 0 12 0 515 3081 >> 6864 3 >> 27 70 >> 1 400 0 8074560 3008988 4771 0 0 0 1457 0 14 0 950 5309 >> 8996 5 >> 27 68 >> >> Lots of processes blocked - which I guess is what the devfs state >> indicates. >> >>> Have you checked dmesg? >> >> Yes. The only odd thing I can see is the following message, but >> from what >> I've read it's not critical until you get 5 and it's only in there >> once. >> >> "collecting pv entries -- suggest increasing PMAP_SHPGPERPROC| >> >> >>> Is this FreeBSD i386 (32bit) or amd64 (64bit)? >> >> FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: >> Fri Jan 12 >> 08:43:30 UTC 2007 root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/ >> SMP >> amd64 >> >>> Have you considered updating? 6.2-RELEASE isn't the >>> freshest anymore. You might even consider going to >>> 7-stable and using the new ULE scheduler which copes >>> better with SMP servers. >> >> I have, but I'd rather understand what's happening. >> >>>> As you can see there's plenty of free memory and the CPU is 70% >>>> idle >>>> yet the load is sky high. >>> >>> Well, load 10 isn't that much for a 4-way SMP system. >> >> A couple of weeks ago this server was fairly fast, load never >> really going >> beyond 3 and everything was reasonably responsive. Now it regularly >> goes up >> to and beyond a load of 10 (more often than not at the moment) and >> everything is slow. This means our website users are getting a very >> poor >> experience which is reflected in our traffic levels which have >> dropped by >> about 25% since this started happening. >> >> Thanks for your help. Any other ideas? >> >> -Stut >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org >> " >>