Date: Tue, 14 Apr 2009 23:26:11 -0300 From: "Marc G. Fournier" <freebsd@hub.org> To: freebsd-stable@freebsd.org Cc: freebsd-proliant@freebsd.org Subject: 7.1-STABLE Sun Mar 29 01:06:46 ADT 2009 Locks up ... Message-ID: <54F1B57E4F5953BBDCE2F1F3@ganymede.hub.org>
next in thread | raw e-mail | index | archive | help
--==========FBEC849F7CF9A3F6439C========== Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi ... Over the past little while, two of my servers have suddenly started to hang ... servers that up until this started, have been reasonably rock solid ... they are generally within a day of each other for source code, and the hardware on both are pretty much identical (HP Proliant DL360 Servers) ... I have serial console configured on both so that I can do CR ~ ^b to get to DDB ... except, when it hangs, all I get is: "KDB: enter: Break sequence on console" And it hangs there, no prompt. I setup a simple script (see attached) to run every 5 minutes that gathers various pieces of info that I think are pertinent, but most likely don't cover everything ... Whenever this happens, on either machine, vmstat show data *like* (notice the high procs -> w values?): procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 165 106 2 12699168 33840 3080 38 2 2 3082 1623 0 0 337 36961 4731 18 7 75 64 75 4 12761744 23084 46809 623 65 43 19307 116 334 0 1189 83674 11708 70 20 10 1 68 25 12773980 23068 11036 3003 9 36 4055 116 282 0 1336 78346 14869 56 16 28 0 71 25 12774236 23084 186 769 1 5 18 80 249 0 609 9298 5894 5 5 91 5 90 31 12747296 23352 626 2546 5 104 1147 368 281 0 1536 40945 19980 6 5 90 Where procs -> w just seems to keep rising ... note that the output for vmstat *5 minutes before* shows: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 35 121 0 12414692 90552 3080 32 2 1 3090 1403 0 0 337 37022 4730 18 7 75 31 93 0 12314408 62024 36550 414 46 6 34285 27 563 0 916 94851 8813 67 33 0 43 179 0 12270932 23080 24035 101 41 12 13887 36 375 0 766 61969 6945 69 23 7 92 44 0 12265524 119804 2122 2028 1 32 13051 1096092 205 0 558 19460 4561 19 50 32 38 34 0 12330068 89140 30758 103 39 119 37037 2837365 165 0 773 92041 7111 47 53 0 I have one QEMU VPS running on this box, with kqemu running the latest kernel module ... but the other machine experiencing the same issue is only running FreeBSD jails ... Both servers are running SCHED_4BSD, if that matters any ... ? I'm at a loss as to what to look at / for next ... pointers would be greatly appreciated ... I have the various output files that the script generates available if anyone thinks they would be useful ... thank you ... Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (FreeBSD) iEYEARECAAYFAknlRcMACgkQ4QvfyHIvDvNmIgCfSWdT9gug6VCjYM1VVMuv1UkN K28AoK298b6mxEeiddu4BAH0+IpkRsti =q6lD -----END PGP SIGNATURE----- --==========FBEC849F7CF9A3F6439C========== Content-Type: multipart/mixed; boundary="==========9CB08864D0820C7D0E16==========" --==========9CB08864D0820C7D0E16========== Content-Type: application/octet-stream; name="monitor.sh" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="monitor.sh"; size=309 IyEvYmluL3NoCmNkIC92bS9yZXBvcnRzCk1JTj1gL2Jpbi9kYXRlICslTWAKaWYgWyAhIC1kICRN SU4gXQp0aGVuCiAgbWtkaXIgJE1JTgpmaQoKY2QgJE1JTgoKL3Vzci9zYmluL2pscyA+IGphaWxs aXN0Lm91dAovYmluL3BzIC1hdWN4SGwgLU8gamlkID4gcHMub3V0Ci91c3Ivc2Jpbi9wc3RhdCAt cyA+IHN3YXAub3V0Ci91c3IvYmluL3Ztc3RhdCAxIDUgPiB2bXN0YXQub3V0Ci91c3IvYmluL2F3 ayAne3ByaW50ICQxNX0nIC9wcm9jLyovc3RhdHVzIHwgL3Vzci9iaW4vc29ydCB8IC91c3IvYmlu L3VuaXEgLWMgPiB2cHNfZGlzdC5vdXQK --==========9CB08864D0820C7D0E16========== Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit Content-Disposition: attachment; size=196 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (FreeBSD) iEYEABECAAYFAknlRcMACgkQ4QvfyHIvDvM1OgCeJXSM5W+14GOBfVBr7G1LL4z6 V4YAoL2H43yO01J3aEEB6AIL5AjlUrul =or0D -----END PGP SIGNATURE----- --==========9CB08864D0820C7D0E16==========-- --==========FBEC849F7CF9A3F6439C==========--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54F1B57E4F5953BBDCE2F1F3>