Date: Wed, 20 Nov 2002 11:20:19 +0200 From: Ruslan Ermilov <ru@FreeBSD.org> To: Matt Dillon <dillon@FreeBSD.org>, Alan Cox <alc@FreeBSD.org> Cc: stable@FreeBSD.org Subject: vm problems in 4.7? Message-ID: <20021120092019.GA5460@sunbay.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hi!
We've got already two hard lockups after upgrading our production
server from 4.5-STABLE to 4.7-STABLE on November 2nd. The server
was rock-stable before, with the uptime more than 90 days. The
hardware in question did not change, nor did the kernel config.
Today's morning I found the system frozen again.
Below is the ps(1) output from the hands-made panic's dump. There
are a few processes waiting on either vmwait or wdrain which seems
suspicious; is there anything else I can do to diagnose the cause
of the problem?
Script started on Wed Nov 20 11:06:04 2002
# ps -axl -N kernel.5 -M vmcore.5
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
60 238 1 0 2 0 2468 0 select IW+ #C2- 0:00,00 (master)
8 282 1 67 10 0 672 0 wait IW+ #C2- 0:00,00 (sh)
8 284 282 0 10 0 808 0 wait IW+ #C2- 0:00,00 (sh)
0 335 1 126 10 0 652 0 wait IW+ #C2- 0:00,00 (sh)
88 365 335 0 2 0 24992 0 - RW+ #C2- 0:00,00 (mysqld)
0 453 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 452 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 451 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 450 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 449 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 448 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 447 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 446 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 445 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 444 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 443 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 92625 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 92624 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty)
0 85486 1 0 3 0 960 0 - RWs+ #C2 0:00,00 (getty)
0 92621 1 0 3 0 960 0 - RWs+ #C2 0:00,00 (getty)
0 35212 1 0 10 0 636 0 wait IW #C2- 0:00,00 (sh)
0 35213 1 0 10 0 632 0 wait IW #C2- 0:00,00 (sh)
0 35214 1 0 10 0 636 0 wait IW #C2- 0:00,00 (sh)
0 62257 1 184 10 0 652 0 wait IW #C2- 0:00,00 (sh)
0 95598 1 10 10 0 664 0 wait IW #C3- 0:00,00 (sh)
239 29784 29781 0 10 0 1152 0 wait IWs #C3 0:00,00 (bash)
239 75203 29784 0 2 0 2448 0 select IW+ #C3 0:00,00 (ssh)
0 0 0 0 -18 0 0 0 vmwait DLs ?? 0:00,07 (swapper)
0 1 0 0 10 0 548 0 wait ILs ?? 0:12,15 (init)
0 2 0 0 -18 0 0 0 wdrain DL ?? 0:42,18 (pagedaemon)
0 3 0 3 18 0 0 0 psleep DL ?? 0:03,22 (vmdaemon)
0 4 0 0 -18 0 0 0 psleep DL ?? 0:18,01 (bufdaemon)
0 5 0 0 -18 0 0 0 wdrain DL ?? 17:10,78 (syncer)
0 6 0 0 -2 0 0 0 vlruwt DL ?? 0:08,15 (vnlru)
0 49 1 1 28 0 328648 0 pfault DLs ?? 0:11,74 (mount_mfs)
0 59 1 76 18 0 212 0 pause IWs ?? 0:00,00 (adjkerntz)
0 124 1 0 2 0 724 0 select Ss ?? 15:27,37 (natd)
0 144 1 0 -18 0 1028 0 wdrain Ds ?? 2:18,54 (syslogd)
0 150 1 0 2 0 1328 0 select Ss ?? 0:38,50 (ntpd)
1 152 1 0 2 0 964 0 - RWs ?? 0:00,00 (portmap)
0 155 1 0 2 0 1600 0 - RWs ?? 0:00,00 (ypserv)
0 157 1 0 2 0 1096 0 select IWs ?? 0:00,00 (rpc.yppasswdd)
0 159 1 0 2 0 928 0 - RWs ?? 0:00,00 (ypbind)
0 169 1 0 2 0 1104 0 select IWs ?? 0:00,00 (inetd)
0 171 1 0 10 0 1028 0 - RWs ?? 0:00,00 (cron)
0 176 1 0 2 0 2352 0 - RWs ?? 0:00,00 (sshd)
0 202 1 0 2 0 916 0 select IWs ?? 0:00,00 (moused)
8 281 1 0 2 0 11424 0 - RWs ?? 0:00,00 (innd)
72 292 1 0 -18 0 12632 0 wdrain Ds ?? 1:46,71 (ircd)
0 304 1 0 28 0 1972 0 pfault DLs ?? 1:09,04 (dhcpd)
0 315 1 0 2 0 1564 0 select IWs ?? 0:00,00 (kavucc)
0 316 315 0 10 0 1460 0 - RW ?? 0:00,00 (kavucc)
0 328 1 0 -4 0 7196 0 msgwai Is ?? 0:00,00 (kavdaemon)
0 329 1 18 2 0 7244 0 select IWs ?? 0:00,00 (kavdaemon)
0 386 1 0 2 0 3024 0 - RWs ?? 0:00,00 (smbd)
0 388 1 0 2 0 2084 0 - RWs ?? 0:00,00 (nmbd)
0 390 388 126 -6 0 2048 0 piperd I ?? 0:00,00 (nmbd)
0 408 1 0 2 0 3948 0 select IWs ?? 0:00,00 (snmpd)
0 421 1 0 4 0 3680 0 bpf Ss ?? 29:32,05 (trafd)
0 430 1 0 2 0 2696 0 - RWs ?? 0:00,00 (xdm)
53 3636 1 0 -18 0 5544 0 vmwait DLs ?? 6:45,85 (named)
0 11036 1 0 10 0 2864 0 wait IWs ?? 0:00,00 (squid)
65534 11038 11036 0 -18 0 56740 0 wdrain D ?? 36:47,45 (squid)
65534 11049 11038 2 -6 0 872 0 piperd Is ?? 0:00,01 (unlinkd)
65534 11050 11038 0 -4 0 1392 0 msgwai Ss ?? 1:20,28 (diskd)
60 12951 238 0 2 0 23864 0 sbwait IW ?? 0:00,00 (imapd)
60 20264 238 0 2 0 20936 0 - RW ?? 0:00,00 (imapd)
60 20265 238 0 2 0 20944 0 - RW ?? 0:00,00 (imapd)
60 20266 238 0 2 0 21528 0 - RW ?? 0:00,00 (imapd)
8 21386 1 0 2 0 3056 0 - RWs ?? 0:00,00 (nnrpd)
0 22646 95598 0 10 0 180 0 - RW ?? 0:00,00 (sleep)
271 24308 386 0 2 0 3628 0 - RW ?? 0:00,00 (smbd)
271 25446 386 0 2 0 3484 0 - RW ?? 0:00,00 (smbd)
60 26071 238 0 18 0 20848 0 - RW ?? 0:00,00 (imapd)
60 26135 238 0 2 0 22404 0 - RW ?? 0:00,00 (imapd)
60 28952 238 0 2 0 20848 0 - RW ?? 0:00,00 (imapd)
0 29731 386 0 2 0 3492 0 - RW ?? 0:00,00 (smbd)
0 29781 176 0 2 0 2516 0 select IW ?? 0:00,00 (sshd)
60 30285 238 0 2 0 21260 0 - RW ?? 0:00,00 (imapd)
60 30510 238 0 2 0 21464 0 select IW ?? 0:00,00 (imapd)
60 31142 238 0 2 0 20944 0 - RW ?? 0:00,00 (imapd)
60 32295 238 0 2 0 20984 0 select IW ?? 0:00,00 (imapd)
0 32354 386 0 2 0 3500 0 - RW ?? 0:00,00 (smbd)
60 32694 238 0 2 0 20940 0 - RW ?? 0:00,00 (imapd)
60 32839 238 0 2 0 20944 0 select IW ?? 0:00,00 (imapd)
0 32945 386 0 2 0 3492 0 - RW ?? 0:00,00 (smbd)
60 32978 238 0 2 0 21732 0 - RW ?? 0:00,00 (imapd)
60 33426 238 0 18 0 20856 0 - RW ?? 0:00,00 (imapd)
0 34363 62257 184 10 0 180 0 nanslp IW ?? 0:00,00 (sleep)
65534 34364 11038 0 2 0 1972 0 - RWs ?? 0:00,00 (perl)
65534 34365 11038 0 2 0 1964 0 sbwait IWs ?? 0:00,00 (perl)
65534 34366 11038 0 2 0 1964 0 sbwait IWs ?? 0:00,00 (perl)
65534 34367 11038 6 2 0 1908 0 sbwait IWs ?? 0:00,00 (perl)
65534 34368 11038 6 2 0 1908 0 sbwait IWs ?? 0:00,00 (perl)
0 34370 11038 0 2 0 1436 0 sbwait IWs ?? 0:00,00 (pam_auth)
0 34373 11038 7 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_auth)
0 34377 11038 9 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_auth)
0 34382 11038 10 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_auth)
0 34384 11038 11 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_auth)
0 34616 386 0 2 0 3532 0 - RW ?? 0:00,00 (smbd)
60 35524 238 0 2 0 20924 0 - RW ?? 0:00,00 (lmtpd)
60 35569 238 0 2 0 20836 0 - RW ?? 0:00,00 (imapd)
60 35616 238 0 2 0 21000 0 - RW ?? 0:00,00 (imapd)
0 35695 35213 0 10 0 180 0 - RW ?? 0:00,00 (sleep)
0 35699 35212 0 10 0 180 0 - RW ?? 0:00,00 (sleep)
0 35757 36191 0 2 0 3340 0 - RW ?? 0:00,00 (sendmail)
0 35912 386 0 2 0 3516 0 - RW ?? 0:00,00 (smbd)
60 36012 238 0 18 0 20816 0 - RW ?? 0:00,00 (imapd)
60 36059 238 0 18 0 20832 0 - RW ?? 0:00,00 (imapd)
0 36191 1 0 2 0 3228 0 - RWs ?? 0:00,00 (sendmail)
25 36194 1 0 18 0 2784 0 pause IWs ?? 0:00,00 (sendmail)
16032 36289 169 0 2 0 1908 0 select IWs ?? 0:00,00 (cvs)
0 36312 35214 0 10 0 180 0 nanslp IW ?? 0:00,00 (sleep)
0 36411 36191 0 2 0 3328 0 - RW ?? 0:00,00 (sendmail)
0 36417 386 0 28 0 3496 0 pfault DL ?? 0:00,05 (smbd)
60 36497 238 0 2 0 21080 0 - RW ?? 0:00,00 (lmtpd)
0 36500 36191 0 2 0 3328 0 - RW ?? 0:00,00 (sendmail)
60 36508 238 0 2 0 20872 0 select IW ?? 0:00,00 (imapd)
0 36532 36191 0 2 0 3328 0 - RW ?? 0:00,00 (sendmail)
0 36533 171 0 -6 0 1028 0 piperd I ?? 0:00,00 (cron)
0 36536 36533 0 10 0 632 0 wait IWs ?? 0:00,00 (sh)
0 36540 36536 0 -6 0 6800 0 piperd I ?? 0:00,46 (perl)
0 36543 36540 0 -6 0 6800 0 piperd I ?? 0:00,00 (perl)
0 36544 36540 0 -6 0 6800 0 piperd I ?? 0:00,00 (perl)
0 36545 36540 0 -6 0 6800 0 piperd I ?? 0:00,00 (perl)
0 36546 36540 63 35 0 0 0 - Z ?? 0:00,00 (perl)
0 36548 36544 0 10 0 632 0 wait IW ?? 0:00,00 (sh)
0 36549 36545 0 10 0 632 0 wait IW ?? 0:00,00 (sh)
0 36550 36548 0 28 0 456 0 pfault D ?? 0:00,01 (ping)
0 36551 36548 0 -6 0 1084 0 piperd I ?? 0:00,00 (grep)
0 36552 36548 0 -6 0 952 0 piperd I ?? 0:00,00 (sed)
0 36553 36549 0 28 0 456 0 pfault D ?? 0:00,01 (ping)
0 36554 36549 0 -6 0 1084 0 piperd I ?? 0:00,00 (grep)
0 36555 36549 0 -6 0 952 0 piperd I ?? 0:00,00 (sed)
0 36558 36543 0 10 0 632 0 wait IW ?? 0:00,00 (sh)
0 36559 36558 0 28 0 456 0 pfault D ?? 0:00,01 (ping)
0 36560 36558 0 -6 0 1084 0 piperd I ?? 0:00,00 (grep)
0 36561 36558 0 -6 0 952 0 piperd I ?? 0:00,00 (sed)
60 36571 238 0 18 0 20836 0 - RW ?? 0:00,00 (imapd)
8 36718 284 0 10 0 180 0 - RW ?? 0:00,00 (sleep)
0 36730 1 0 28 0 3992 0 pfault DL ?? 0:00,08 (sendmail)
8 36731 21386 0 2 4 3436 0 - RWN ?? 0:00,00 (nnrpd)
8 36732 21386 0 2 4 3436 0 - RWN ?? 0:00,00 (nnrpd)
282 36744 386 0 2 0 3520 0 - RW ?? 0:00,00 (smbd)
258 36783 386 0 2 0 3520 0 - RW ?? 0:00,00 (smbd)
16032 36789 36289 3 -18 0 197736 0 wdrain D ?? 0:03,26 (cvs)
140 36813 63421 0 28 0 2132 0 pfault DL ?? 0:00,00 (netsaint)
53 36814 3636 0 28 0 5544 0 - RV ?? 0:00,00 (named)
0 41193 1 0 18 0 1108 0 lockf IWs ?? 0:00,00 (saslauthd)
0 41194 41193 0 18 0 1112 0 lockf IW ?? 0:00,00 (saslauthd)
0 41195 41193 0 18 0 1112 0 lockf IW ?? 0:00,00 (saslauthd)
0 41196 41193 0 18 0 1108 0 lockf IW ?? 0:00,00 (saslauthd)
0 41197 41193 0 2 0 1108 0 accept IW ?? 0:00,00 (saslauthd)
0 48938 1 0 2 0 1092 0 accept IWs ?? 0:00,00 (pwcheck_pam)
60 52333 238 0 2 0 22408 0 select IW ?? 0:00,00 (imapd)
60 54291 238 0 2 0 23164 0 - RW ?? 0:00,00 (imapd)
60 54740 238 0 2 0 23132 0 - RW ?? 0:00,00 (imapd)
140 63421 1 0 28 0 2132 0 pfault DLs ?? 2:21,97 (netsaint)
60 64293 238 0 2 0 21084 0 - RW ?? 0:00,00 (imapd)
60 66556 238 0 2 0 21580 0 select IW ?? 0:00,00 (imapd)
60 66570 238 0 2 0 20984 0 - RW ?? 0:00,00 (imapd)
60 66776 238 0 2 0 20972 0 - RW ?? 0:00,00 (imapd)
60 67112 238 0 2 0 20960 0 - RW ?? 0:00,00 (imapd)
0 72176 386 0 2 0 3616 0 - RW ?? 0:00,00 (smbd)
0 72959 386 0 2 0 3624 0 - RW ?? 0:00,00 (smbd)
0 77142 430 0 2 0 2872 0 - RWs ?? 0:00,00 (xdm)
8 79532 281 0 2 4 1464 0 - RWN ?? 0:00,00 (innfeed)
8 79533 281 28 2 4 3336 0 sbwait IWN ?? 0:00,00 (perl)
8 79534 281 0 2 4 3376 0 sbwait IWN ?? 0:00,00 (perl)
0 80687 1 0 3 0 956 0 siodcd IW ?? 0:00,00 (getty)
72 86360 292 0 2 0 3324 0 - RW ?? 0:00,00 (servlink)
0 87916 386 0 2 0 3576 0 - RW ?? 0:00,00 (smbd)
60 89204 238 0 2 0 20980 0 - RW ?? 0:00,00 (imapd)
60 90019 238 0 2 0 21064 0 - RW ?? 0:00,00 (imapd)
0 98940 1 0 2 0 6584 0 select Ss ?? 0:19,09 (httpd)
80 98941 98940 0 2 0 6380 0 - RW ?? 0:00,00 (httpd)
80 98942 98941 0 2 0 10832 0 accept IW ?? 0:00,00 (lexacal.fcgi)
80 98943 98940 0 18 0 6948 0 lockf IW ?? 0:00,00 (httpd)
80 98944 98940 0 18 0 7012 0 lockf IW ?? 0:00,00 (httpd)
80 98945 98940 0 18 0 7000 0 lockf IW ?? 0:00,00 (httpd)
80 98946 98940 0 2 0 6984 0 - RW ?? 0:00,00 (httpd)
80 98947 98940 0 18 0 6984 0 lockf IW ?? 0:00,00 (httpd)
80 98948 98941 0 2 0 10832 0 accept IW ?? 0:00,00 (lexacal.fcgi)
80 98950 98940 0 18 0 6876 0 lockf IW ?? 0:00,00 (httpd)
80 98951 98941 0 2 0 10832 0 accept IW ?? 0:00,00 (lexacal.fcgi)
80 98957 98941 1 2 0 10832 0 accept IW ?? 0:00,00 (lexacal.fcgi)
80 98961 98941 1 2 0 10832 0 accept IW ?? 0:00,00 (lexacal.fcgi)
80 99098 98940 0 18 0 6992 0 lockf IW ?? 0:00,00 (httpd)
80 99770 98940 0 18 0 7036 0 lockf IW ?? 0:00,00 (httpd)
80 99772 98940 0 18 0 6912 0 lockf IW ?? 0:00,00 (httpd)
80 99773 98940 0 18 0 7132 0 lockf IW ?? 0:00,00 (httpd)
#
Script done on Wed Nov 20 11:06:26 2002
Cheers,
--
Ruslan Ermilov Sysadmin and DBA,
ru@sunbay.com Sunbay Software AG,
ru@FreeBSD.org FreeBSD committer,
+380.652.512.251 Simferopol, Ukraine
http://www.FreeBSD.org The Power To Serve
http://www.oracle.com Enabling The Information Age
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (FreeBSD)
iD8DBQE921PTUkv4P6juNwoRAseKAJ97Hcs7kKhz2WibKyRbY2NXtoFksgCdG2uU
o0BN/80FrTrNgCIwPZOGw1Y=
=gB83
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021120092019.GA5460>