From owner-freebsd-stable Wed Nov 20 1:20:47 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 58F1837B401; Wed, 20 Nov 2002 01:20:40 -0800 (PST) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9F77243E4A; Wed, 20 Nov 2002 01:20:34 -0800 (PST) (envelope-from ru@whale.sunbay.crimea.ua) Received: from whale.sunbay.crimea.ua (ru@localhost [127.0.0.1]) by whale.sunbay.crimea.ua (8.12.6/8.12.6/Sunbay) with ESMTP id gAK9KJhC007885 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 20 Nov 2002 11:20:22 +0200 (EET) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.12.6/8.12.6/Submit) id gAK9KJ12007880; Wed, 20 Nov 2002 11:20:19 +0200 (EET) Date: Wed, 20 Nov 2002 11:20:19 +0200 From: Ruslan Ermilov To: Matt Dillon , Alan Cox Cc: stable@FreeBSD.org Subject: vm problems in 4.7? Message-ID: <20021120092019.GA5460@sunbay.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LZvS9be/3tNcYl/X" Content-Disposition: inline User-Agent: Mutt/1.5.1i Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi! We've got already two hard lockups after upgrading our production server from 4.5-STABLE to 4.7-STABLE on November 2nd. The server was rock-stable before, with the uptime more than 90 days. The hardware in question did not change, nor did the kernel config. Today's morning I found the system frozen again. Below is the ps(1) output from the hands-made panic's dump. There are a few processes waiting on either vmwait or wdrain which seems suspicious; is there anything else I can do to diagnose the cause of the problem? Script started on Wed Nov 20 11:06:04 2002 # ps -axl -N kernel.5 -M vmcore.5 UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND 60 238 1 0 2 0 2468 0 select IW+ #C2- 0:00,00 (master) 8 282 1 67 10 0 672 0 wait IW+ #C2- 0:00,00 (sh) 8 284 282 0 10 0 808 0 wait IW+ #C2- 0:00,00 (sh) 0 335 1 126 10 0 652 0 wait IW+ #C2- 0:00,00 (sh) 88 365 335 0 2 0 24992 0 - RW+ #C2- 0:00,00 (mysqld) 0 453 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 452 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 451 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 450 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 449 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 448 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 447 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 446 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 445 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 444 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 443 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 92625 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 92624 1 0 3 0 960 0 ttyin IWs+ #C2 0:00,00 (getty) 0 85486 1 0 3 0 960 0 - RWs+ #C2 0:00,00 (getty) 0 92621 1 0 3 0 960 0 - RWs+ #C2 0:00,00 (getty) 0 35212 1 0 10 0 636 0 wait IW #C2- 0:00,00 (sh) 0 35213 1 0 10 0 632 0 wait IW #C2- 0:00,00 (sh) 0 35214 1 0 10 0 636 0 wait IW #C2- 0:00,00 (sh) 0 62257 1 184 10 0 652 0 wait IW #C2- 0:00,00 (sh) 0 95598 1 10 10 0 664 0 wait IW #C3- 0:00,00 (sh) 239 29784 29781 0 10 0 1152 0 wait IWs #C3 0:00,00 (bash) 239 75203 29784 0 2 0 2448 0 select IW+ #C3 0:00,00 (ssh) 0 0 0 0 -18 0 0 0 vmwait DLs ?? 0:00,07 (swappe= r) 0 1 0 0 10 0 548 0 wait ILs ?? 0:12,15 (init) 0 2 0 0 -18 0 0 0 wdrain DL ?? 0:42,18 (pageda= emon) 0 3 0 3 18 0 0 0 psleep DL ?? 0:03,22 (vmdaem= on) 0 4 0 0 -18 0 0 0 psleep DL ?? 0:18,01 (bufdae= mon) 0 5 0 0 -18 0 0 0 wdrain DL ?? 17:10,78 (syncer) 0 6 0 0 -2 0 0 0 vlruwt DL ?? 0:08,15 (vnlru) 0 49 1 1 28 0 328648 0 pfault DLs ?? 0:11,74 (mount= _mfs) 0 59 1 76 18 0 212 0 pause IWs ?? 0:00,00 (adjker= ntz) 0 124 1 0 2 0 724 0 select Ss ?? 15:27,37 (natd) 0 144 1 0 -18 0 1028 0 wdrain Ds ?? 2:18,54 (syslog= d) 0 150 1 0 2 0 1328 0 select Ss ?? 0:38,50 (ntpd) 1 152 1 0 2 0 964 0 - RWs ?? 0:00,00 (portma= p) 0 155 1 0 2 0 1600 0 - RWs ?? 0:00,00 (ypserv) 0 157 1 0 2 0 1096 0 select IWs ?? 0:00,00 (rpc.yp= passwdd) 0 159 1 0 2 0 928 0 - RWs ?? 0:00,00 (ypbind) 0 169 1 0 2 0 1104 0 select IWs ?? 0:00,00 (inetd) 0 171 1 0 10 0 1028 0 - RWs ?? 0:00,00 (cron) 0 176 1 0 2 0 2352 0 - RWs ?? 0:00,00 (sshd) 0 202 1 0 2 0 916 0 select IWs ?? 0:00,00 (moused) 8 281 1 0 2 0 11424 0 - RWs ?? 0:00,00 (innd) 72 292 1 0 -18 0 12632 0 wdrain Ds ?? 1:46,71 (ircd) 0 304 1 0 28 0 1972 0 pfault DLs ?? 1:09,04 (dhcpd) 0 315 1 0 2 0 1564 0 select IWs ?? 0:00,00 (kavucc) 0 316 315 0 10 0 1460 0 - RW ?? 0:00,00 (kavucc) 0 328 1 0 -4 0 7196 0 msgwai Is ?? 0:00,00 (kavdae= mon) 0 329 1 18 2 0 7244 0 select IWs ?? 0:00,00 (kavdae= mon) 0 386 1 0 2 0 3024 0 - RWs ?? 0:00,00 (smbd) 0 388 1 0 2 0 2084 0 - RWs ?? 0:00,00 (nmbd) 0 390 388 126 -6 0 2048 0 piperd I ?? 0:00,00 (nmbd) 0 408 1 0 2 0 3948 0 select IWs ?? 0:00,00 (snmpd) 0 421 1 0 4 0 3680 0 bpf Ss ?? 29:32,05 (trafd) 0 430 1 0 2 0 2696 0 - RWs ?? 0:00,00 (xdm) 53 3636 1 0 -18 0 5544 0 vmwait DLs ?? 6:45,85 (named) 0 11036 1 0 10 0 2864 0 wait IWs ?? 0:00,00 (squid) 65534 11038 11036 0 -18 0 56740 0 wdrain D ?? 36:47,45 (squid) 65534 11049 11038 2 -6 0 872 0 piperd Is ?? 0:00,01 (unlink= d) 65534 11050 11038 0 -4 0 1392 0 msgwai Ss ?? 1:20,28 (diskd) 60 12951 238 0 2 0 23864 0 sbwait IW ?? 0:00,00 (imapd) 60 20264 238 0 2 0 20936 0 - RW ?? 0:00,00 (imapd) 60 20265 238 0 2 0 20944 0 - RW ?? 0:00,00 (imapd) 60 20266 238 0 2 0 21528 0 - RW ?? 0:00,00 (imapd) 8 21386 1 0 2 0 3056 0 - RWs ?? 0:00,00 (nnrpd) 0 22646 95598 0 10 0 180 0 - RW ?? 0:00,00 (sleep) 271 24308 386 0 2 0 3628 0 - RW ?? 0:00,00 (smbd) 271 25446 386 0 2 0 3484 0 - RW ?? 0:00,00 (smbd) 60 26071 238 0 18 0 20848 0 - RW ?? 0:00,00 (imapd) 60 26135 238 0 2 0 22404 0 - RW ?? 0:00,00 (imapd) 60 28952 238 0 2 0 20848 0 - RW ?? 0:00,00 (imapd) 0 29731 386 0 2 0 3492 0 - RW ?? 0:00,00 (smbd) 0 29781 176 0 2 0 2516 0 select IW ?? 0:00,00 (sshd) 60 30285 238 0 2 0 21260 0 - RW ?? 0:00,00 (imapd) 60 30510 238 0 2 0 21464 0 select IW ?? 0:00,00 (imapd) 60 31142 238 0 2 0 20944 0 - RW ?? 0:00,00 (imapd) 60 32295 238 0 2 0 20984 0 select IW ?? 0:00,00 (imapd) 0 32354 386 0 2 0 3500 0 - RW ?? 0:00,00 (smbd) 60 32694 238 0 2 0 20940 0 - RW ?? 0:00,00 (imapd) 60 32839 238 0 2 0 20944 0 select IW ?? 0:00,00 (imapd) 0 32945 386 0 2 0 3492 0 - RW ?? 0:00,00 (smbd) 60 32978 238 0 2 0 21732 0 - RW ?? 0:00,00 (imapd) 60 33426 238 0 18 0 20856 0 - RW ?? 0:00,00 (imapd) 0 34363 62257 184 10 0 180 0 nanslp IW ?? 0:00,00 (sleep) 65534 34364 11038 0 2 0 1972 0 - RWs ?? 0:00,00 (perl) 65534 34365 11038 0 2 0 1964 0 sbwait IWs ?? 0:00,00 (perl) 65534 34366 11038 0 2 0 1964 0 sbwait IWs ?? 0:00,00 (perl) 65534 34367 11038 6 2 0 1908 0 sbwait IWs ?? 0:00,00 (perl) 65534 34368 11038 6 2 0 1908 0 sbwait IWs ?? 0:00,00 (perl) 0 34370 11038 0 2 0 1436 0 sbwait IWs ?? 0:00,00 (pam_au= th) 0 34373 11038 7 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_au= th) 0 34377 11038 9 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_au= th) 0 34382 11038 10 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_au= th) 0 34384 11038 11 2 0 1152 0 sbwait IWs ?? 0:00,00 (pam_au= th) 0 34616 386 0 2 0 3532 0 - RW ?? 0:00,00 (smbd) 60 35524 238 0 2 0 20924 0 - RW ?? 0:00,00 (lmtpd) 60 35569 238 0 2 0 20836 0 - RW ?? 0:00,00 (imapd) 60 35616 238 0 2 0 21000 0 - RW ?? 0:00,00 (imapd) 0 35695 35213 0 10 0 180 0 - RW ?? 0:00,00 (sleep) 0 35699 35212 0 10 0 180 0 - RW ?? 0:00,00 (sleep) 0 35757 36191 0 2 0 3340 0 - RW ?? 0:00,00 (sendma= il) 0 35912 386 0 2 0 3516 0 - RW ?? 0:00,00 (smbd) 60 36012 238 0 18 0 20816 0 - RW ?? 0:00,00 (imapd) 60 36059 238 0 18 0 20832 0 - RW ?? 0:00,00 (imapd) 0 36191 1 0 2 0 3228 0 - RWs ?? 0:00,00 (sendma= il) 25 36194 1 0 18 0 2784 0 pause IWs ?? 0:00,00 (sendma= il) 16032 36289 169 0 2 0 1908 0 select IWs ?? 0:00,00 (cvs) 0 36312 35214 0 10 0 180 0 nanslp IW ?? 0:00,00 (sleep) 0 36411 36191 0 2 0 3328 0 - RW ?? 0:00,00 (sendma= il) 0 36417 386 0 28 0 3496 0 pfault DL ?? 0:00,05 (smbd) 60 36497 238 0 2 0 21080 0 - RW ?? 0:00,00 (lmtpd) 0 36500 36191 0 2 0 3328 0 - RW ?? 0:00,00 (sendma= il) 60 36508 238 0 2 0 20872 0 select IW ?? 0:00,00 (imapd) 0 36532 36191 0 2 0 3328 0 - RW ?? 0:00,00 (sendma= il) 0 36533 171 0 -6 0 1028 0 piperd I ?? 0:00,00 (cron) 0 36536 36533 0 10 0 632 0 wait IWs ?? 0:00,00 (sh) 0 36540 36536 0 -6 0 6800 0 piperd I ?? 0:00,46 (perl) 0 36543 36540 0 -6 0 6800 0 piperd I ?? 0:00,00 (perl) 0 36544 36540 0 -6 0 6800 0 piperd I ?? 0:00,00 (perl) 0 36545 36540 0 -6 0 6800 0 piperd I ?? 0:00,00 (perl) 0 36546 36540 63 35 0 0 0 - Z ?? 0:00,00 (perl) 0 36548 36544 0 10 0 632 0 wait IW ?? 0:00,00 (sh) 0 36549 36545 0 10 0 632 0 wait IW ?? 0:00,00 (sh) 0 36550 36548 0 28 0 456 0 pfault D ?? 0:00,01 (ping) 0 36551 36548 0 -6 0 1084 0 piperd I ?? 0:00,00 (grep) 0 36552 36548 0 -6 0 952 0 piperd I ?? 0:00,00 (sed) 0 36553 36549 0 28 0 456 0 pfault D ?? 0:00,01 (ping) 0 36554 36549 0 -6 0 1084 0 piperd I ?? 0:00,00 (grep) 0 36555 36549 0 -6 0 952 0 piperd I ?? 0:00,00 (sed) 0 36558 36543 0 10 0 632 0 wait IW ?? 0:00,00 (sh) 0 36559 36558 0 28 0 456 0 pfault D ?? 0:00,01 (ping) 0 36560 36558 0 -6 0 1084 0 piperd I ?? 0:00,00 (grep) 0 36561 36558 0 -6 0 952 0 piperd I ?? 0:00,00 (sed) 60 36571 238 0 18 0 20836 0 - RW ?? 0:00,00 (imapd) 8 36718 284 0 10 0 180 0 - RW ?? 0:00,00 (sleep) 0 36730 1 0 28 0 3992 0 pfault DL ?? 0:00,08 (sendma= il) 8 36731 21386 0 2 4 3436 0 - RWN ?? 0:00,00 (nnrpd) 8 36732 21386 0 2 4 3436 0 - RWN ?? 0:00,00 (nnrpd) 282 36744 386 0 2 0 3520 0 - RW ?? 0:00,00 (smbd) 258 36783 386 0 2 0 3520 0 - RW ?? 0:00,00 (smbd) 16032 36789 36289 3 -18 0 197736 0 wdrain D ?? 0:03,26 (cvs) 140 36813 63421 0 28 0 2132 0 pfault DL ?? 0:00,00 (netsai= nt) 53 36814 3636 0 28 0 5544 0 - RV ?? 0:00,00 (named) 0 41193 1 0 18 0 1108 0 lockf IWs ?? 0:00,00 (saslau= thd) 0 41194 41193 0 18 0 1112 0 lockf IW ?? 0:00,00 (saslau= thd) 0 41195 41193 0 18 0 1112 0 lockf IW ?? 0:00,00 (saslau= thd) 0 41196 41193 0 18 0 1108 0 lockf IW ?? 0:00,00 (saslau= thd) 0 41197 41193 0 2 0 1108 0 accept IW ?? 0:00,00 (saslau= thd) 0 48938 1 0 2 0 1092 0 accept IWs ?? 0:00,00 (pwchec= k_pam) 60 52333 238 0 2 0 22408 0 select IW ?? 0:00,00 (imapd) 60 54291 238 0 2 0 23164 0 - RW ?? 0:00,00 (imapd) 60 54740 238 0 2 0 23132 0 - RW ?? 0:00,00 (imapd) 140 63421 1 0 28 0 2132 0 pfault DLs ?? 2:21,97 (netsai= nt) 60 64293 238 0 2 0 21084 0 - RW ?? 0:00,00 (imapd) 60 66556 238 0 2 0 21580 0 select IW ?? 0:00,00 (imapd) 60 66570 238 0 2 0 20984 0 - RW ?? 0:00,00 (imapd) 60 66776 238 0 2 0 20972 0 - RW ?? 0:00,00 (imapd) 60 67112 238 0 2 0 20960 0 - RW ?? 0:00,00 (imapd) 0 72176 386 0 2 0 3616 0 - RW ?? 0:00,00 (smbd) 0 72959 386 0 2 0 3624 0 - RW ?? 0:00,00 (smbd) 0 77142 430 0 2 0 2872 0 - RWs ?? 0:00,00 (xdm) 8 79532 281 0 2 4 1464 0 - RWN ?? 0:00,00 (innfee= d) 8 79533 281 28 2 4 3336 0 sbwait IWN ?? 0:00,00 (perl) 8 79534 281 0 2 4 3376 0 sbwait IWN ?? 0:00,00 (perl) 0 80687 1 0 3 0 956 0 siodcd IW ?? 0:00,00 (getty) 72 86360 292 0 2 0 3324 0 - RW ?? 0:00,00 (servli= nk) 0 87916 386 0 2 0 3576 0 - RW ?? 0:00,00 (smbd) 60 89204 238 0 2 0 20980 0 - RW ?? 0:00,00 (imapd) 60 90019 238 0 2 0 21064 0 - RW ?? 0:00,00 (imapd) 0 98940 1 0 2 0 6584 0 select Ss ?? 0:19,09 (httpd) 80 98941 98940 0 2 0 6380 0 - RW ?? 0:00,00 (httpd) 80 98942 98941 0 2 0 10832 0 accept IW ?? 0:00,00 (lexaca= l.fcgi) 80 98943 98940 0 18 0 6948 0 lockf IW ?? 0:00,00 (httpd) 80 98944 98940 0 18 0 7012 0 lockf IW ?? 0:00,00 (httpd) 80 98945 98940 0 18 0 7000 0 lockf IW ?? 0:00,00 (httpd) 80 98946 98940 0 2 0 6984 0 - RW ?? 0:00,00 (httpd) 80 98947 98940 0 18 0 6984 0 lockf IW ?? 0:00,00 (httpd) 80 98948 98941 0 2 0 10832 0 accept IW ?? 0:00,00 (lexaca= l.fcgi) 80 98950 98940 0 18 0 6876 0 lockf IW ?? 0:00,00 (httpd) 80 98951 98941 0 2 0 10832 0 accept IW ?? 0:00,00 (lexaca= l.fcgi) 80 98957 98941 1 2 0 10832 0 accept IW ?? 0:00,00 (lexaca= l.fcgi) 80 98961 98941 1 2 0 10832 0 accept IW ?? 0:00,00 (lexaca= l.fcgi) 80 99098 98940 0 18 0 6992 0 lockf IW ?? 0:00,00 (httpd) 80 99770 98940 0 18 0 7036 0 lockf IW ?? 0:00,00 (httpd) 80 99772 98940 0 18 0 6912 0 lockf IW ?? 0:00,00 (httpd) 80 99773 98940 0 18 0 7132 0 lockf IW ?? 0:00,00 (httpd) #=20 Script done on Wed Nov 20 11:06:26 2002 Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --LZvS9be/3tNcYl/X Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE921PTUkv4P6juNwoRAseKAJ97Hcs7kKhz2WibKyRbY2NXtoFksgCdG2uU o0BN/80FrTrNgCIwPZOGw1Y= =gB83 -----END PGP SIGNATURE----- --LZvS9be/3tNcYl/X-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message