From owner-freebsd-current@FreeBSD.ORG Thu Jul 9 14:21:27 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D33161065670 for ; Thu, 9 Jul 2009 14:21:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 187E78FC18 for ; Thu, 9 Jul 2009 14:21:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id n69ELL86003248 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 9 Jul 2009 17:21:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id n69ELLsl087172 for ; Thu, 9 Jul 2009 17:21:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id n69ELLNd087137 for freebsd-current@freebsd.org; Thu, 9 Jul 2009 17:21:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 9 Jul 2009 17:21:21 +0300 From: Kostik Belousov To: freebsd-current@freebsd.org Message-ID: <20090709142121.GS55190@deviant.kiev.zoral.com.ua> References: <20090709060556.GA27373@rwpc12.mby.riverwillow.net.au> <20090709073054.GB27373@rwpc12.mby.riverwillow.net.au> <20090709085242.GC27373@rwpc12.mby.riverwillow.net.au> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FiqEyLLt06qkB6ow" Content-Disposition: inline In-Reply-To: <20090709085242.GC27373@rwpc12.mby.riverwillow.net.au> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Subject: Re: Process stuck in vmmaps on 8.0-BETA1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2009 14:21:28 -0000 --FiqEyLLt06qkB6ow Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 09, 2009 at 06:52:42PM +1000, John Marshall wrote: > On Thu, 09 Jul 2009, 17:30 +1000, John Marshall wrote: > > On Thu, 09 Jul 2009, 10:42 +0400, pluknet wrote: > > > 2009/7/9 John Marshall : > > > > After upgrading... > > > > - boot new kernel to single-user > > > > - make installworld > > > > - make delete-old > > > > - make delete-old-libs > > > > - mergemaster > > > > - reboot > > > > > > > > I re-built a few of my applications. I noticed a problem with ntpd > > > > 4.2.4p7. The build was fine, it started fine, but got stuck in vmm= aps > > > > and I couldn't kill it. Stopping the operating system appears to b= e the > > > > only remedy. I have re-built a few times (starting with 'make > > > > distclean') just to make sure. > > > > > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME = COMMAND > > > > 0 791 1 0 44 0 4944 4920 vmmaps Ds ?? 0:00.01 = ntpd > > > > > > >=20 > > > Can you place here 'procstat -k 791', where 791 is pid of ntpd? > > > It'd be nice also if you go through all ddb steps described in > > > Debugging Deadlocks chapter of FreeBSD Developers' Handbook. > >=20 > > Here is some procstat output. I'm just rebuilding the kernel with the > > debugging options enabled - not something I've ever done before. > >=20 > > rwsrv05# procstat 2788 > > PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM= =20 > > 2788 1 2788 2788 0 1 john vmmaps FreeBSD ELF32 ntpd= =20 > > rwsrv05# procstat -k 2788 > > PID TID COMM TDNAME KSTACK = =20 > > 2788 100164 ntpd - mi_switch sleepq_switch = sleepq_wait _sleep vm_map_unlock_and_wait vm_map_delete vm_map_fixed vm_mma= p mmap syscall Xint0x80_syscall=20 > > rwsrv05# procstat -v 2788 > > PID START END PRT RES PRES REF SHD FL TP PATH > > 2788 0x8048000 0x807e000 r-x 54 60 2 1 CN vn /usr/local/bin/= ntpd > > 2788 0x807e000 0x8080000 rw- 2 0 1 0 C- vn /usr/local/bin/= ntpd > > 2788 0x8080000 0x8100000 rw- 128 0 1 0 C- df=20 > > 2788 0x2807e000 0x280ab000 r-x 45 0 171 75 CN vn /libexec/ld-elf= .so.1 > > 2788 0x280ab000 0x280ad000 rw- 2 0 1 0 C- vn /libexec/ld-elf= .so.1 > > 2788 0x280ad000 0x280c0000 rw- 19 0 1 0 C- df=20 > > 2788 0x280c0000 0x280d7000 r-x 23 0 1 0 CN vn /lib/libm.so.5 > > 2788 0x280d7000 0x280d8000 r-x 1 0 1 0 CN vn /lib/libm.so.5 > > 2788 0x280d8000 0x280d9000 rw- 1 0 1 0 C- vn /lib/libm.so.5 > > 2788 0x280d9000 0x28211000 r-x 312 0 1 0 CN vn /lib/libcrypto.= so.5 > > 2788 0x28211000 0x28212000 r-x 1 0 1 0 CN vn /lib/libcrypto.= so.5 > > 2788 0x28212000 0x2822a000 rw- 24 0 1 0 C- vn /lib/libcrypto.= so.5 > > 2788 0x2822a000 0x2822c000 rw- 2 0 1 0 C- df=20 > > 2788 0x2822c000 0x28232000 r-x 6 0 1 0 CN vn /lib/libkvm.so.4 > > 2788 0x28232000 0x28233000 r-x 1 0 1 0 CN vn /lib/libkvm.so.4 > > 2788 0x28233000 0x28234000 rw- 1 0 1 0 C- vn /lib/libkvm.so.4 > > 2788 0x28234000 0x2824c000 r-x 24 0 1 0 CN vn /usr/lib/libelf= .so.1 > > 2788 0x2824c000 0x2824d000 r-x 1 0 1 0 CN vn /usr/lib/libelf= .so.1 > > 2788 0x2824d000 0x2824e000 rw- 1 0 1 0 C- vn /usr/lib/libelf= .so.1 > > 2788 0x2824e000 0x28251000 r-x 3 0 15 10 CN vn /usr/lib/librt.= so.1 > > 2788 0x28251000 0x28252000 r-x 1 0 1 0 CN vn /usr/lib/librt.= so.1 > > 2788 0x28252000 0x28253000 rw- 1 0 1 0 C- vn /usr/lib/librt.= so.1 > > 2788 0x28253000 0x28260000 r-x 13 0 1 0 CN vn /lib/libmd.so.4 > > 2788 0x28260000 0x28261000 r-x 1 0 1 0 CN vn /lib/libmd.so.4 > > 2788 0x28261000 0x28262000 rw- 1 0 1 0 C- vn /lib/libmd.so.4 > > 2788 0x28262000 0x28351000 r-x 239 0 1 0 CN vn /lib/libc.so.7 > > 2788 0x28351000 0x28352000 r-x 1 0 1 0 CN vn /lib/libc.so.7 > > 2788 0x28352000 0x28358000 rw- 6 0 1 0 C- vn /lib/libc.so.7 > > 2788 0x28358000 0x2836e000 rw- 22 0 1 0 C- df=20 > > 2788 0x2836e000 0x2837a000 --- 0 0 0 0 -- --=20 > > 2788 0x28400000 0x28500000 rw- 256 0 1 0 C- df=20 > > 2788 0xbfbe0000 0xbfc00000 rwx 32 0 1 0 C- df=20 > > rwsrv05#=20 >=20 > OK, now that I've rebuilt the kernel with the debugging options not > commented out, I'm getting a number of 'lock order reversal' messages > printed on the console: is that normal? >=20 > From the Debugging Deadlocks chapter to which I was referred by pluknet > (above) it appears that I need to enter 'sysctl debug.kdb.enter=3D1' or > 'sysctl debug.kdb.panic=3D1' after I get the process into the desired > 'stuck' state. If I enter either of those commands, the system reboots. > Now *I'm* stuck. Since you have mostly working system, and interesting information most easy accessible by kgdb, attach it to the live kernel: kgdb /dev/mem =46rom there, switch to the stuck process, process do bt find the frame for vm_map_delete, and print the entry: p entry Also, I need to see the information you posted earlier, namely, procstat -k and -v output for the process. --FiqEyLLt06qkB6ow Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkpV/OAACgkQC3+MBN1Mb4hMnQCfYV77E+BucFdo3ScitgWSPW3m zOwAn1nXO/9ns8JWl59RJqh9+dRSFedE =2J9a -----END PGP SIGNATURE----- --FiqEyLLt06qkB6ow--