From owner-freebsd-current@FreeBSD.ORG Fri Jul 10 08:24:05 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 012EC106564A for ; Fri, 10 Jul 2009 08:24:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 505528FC51 for ; Fri, 10 Jul 2009 08:24:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id n6A8NvGk074197 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 10 Jul 2009 11:23:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id n6A8NuLB064841 for ; Fri, 10 Jul 2009 11:23:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id n6A8Nusd064840 for freebsd-current@freebsd.org; Fri, 10 Jul 2009 11:23:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 10 Jul 2009 11:23:56 +0300 From: Kostik Belousov To: freebsd-current@freebsd.org Message-ID: <20090710082356.GZ55190@deviant.kiev.zoral.com.ua> References: <20090709060556.GA27373@rwpc12.mby.riverwillow.net.au> <20090709073054.GB27373@rwpc12.mby.riverwillow.net.au> <20090709085242.GC27373@rwpc12.mby.riverwillow.net.au> <20090709142121.GS55190@deviant.kiev.zoral.com.ua> <20090710035849.GB31950@rwpc12.mby.riverwillow.net.au> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SqEPCaGsKgsesFh5" Content-Disposition: inline In-Reply-To: <20090710035849.GB31950@rwpc12.mby.riverwillow.net.au> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Subject: Re: Process stuck in vmmaps on 8.0-BETA1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jul 2009 08:24:05 -0000 --SqEPCaGsKgsesFh5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 10, 2009 at 01:58:49PM +1000, John Marshall wrote: > On Thu, 09 Jul 2009, 17:21 +0300, Kostik Belousov wrote: > > On Thu, Jul 09, 2009 at 06:52:42PM +1000, John Marshall wrote: > > > OK, now that I've rebuilt the kernel with the debugging options not > > > commented out, I'm getting a number of 'lock order reversal' messages > > > printed on the console: is that normal? > > >=20 > > > From the Debugging Deadlocks chapter to which I was referred by plukn= et > > > (above) it appears that I need to enter 'sysctl debug.kdb.enter=3D1' = or > > > 'sysctl debug.kdb.panic=3D1' after I get the process into the desired > > > 'stuck' state. If I enter either of those commands, the system reboo= ts. > > > Now *I'm* stuck. > >=20 > > Since you have mostly working system, and interesting information most > > easy accessible by kgdb, attach it to the live kernel: > > kgdb /dev/mem > >=20 > > From there, switch to the stuck process, > > process >=20 > I tried that... >=20 > (kgdb) process 1373 > Undefined command: "process". Try "help". >=20 > It took me several more hours to discover "proc" which I assume is what > you meant? Yes. >=20 > > do > > bt > > find the frame for vm_map_delete, and print the entry: > > p entry >=20 > I have no idea which number(s) to plug in here. I hope I guessed the > right one. >=20 > > Also, I need to see the information you posted earlier, namely, procstat > > -k and -v output for the process. >=20 > rwsrv05# procstat 1373 > PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM = =20 > 1373 1 1373 1373 0 1 john vmmaps FreeBSD ELF32 ntpd = =20 > rwsrv05# procstat -k 1373 > PID TID COMM TDNAME KSTACK = =20 > 1373 100168 ntpd - mi_switch sleepq_switch sl= eepq_wait _sleep vm_map_unlock_and_wait vm_map_delete vm_map_fixed vm_mmap = mmap syscall Xint0x80_syscall=20 > rwsrv05# procstat -v 1373 > PID START END PRT RES PRES REF SHD FL TP PATH > 1373 0x8048000 0x807e000 r-x 54 60 2 1 CN vn /usr/local/bin/nt= pd > 1373 0x807e000 0x8080000 rw- 2 0 1 0 C- vn /usr/local/bin/nt= pd > 1373 0x8080000 0x8100000 rw- 128 0 1 0 C- df=20 > 1373 0x2807e000 0x280ab000 r-x 45 0 143 62 CN vn /libexec/ld-elf.s= o.1 > 1373 0x280ab000 0x280ad000 rw- 2 0 1 0 C- vn /libexec/ld-elf.s= o.1 > 1373 0x280ad000 0x280c0000 rw- 19 0 1 0 C- df=20 > 1373 0x280c0000 0x280d7000 r-x 23 0 1 0 CN vn /lib/libm.so.5 > 1373 0x280d7000 0x280d8000 r-x 1 0 1 0 CN vn /lib/libm.so.5 > 1373 0x280d8000 0x280d9000 rw- 1 0 1 0 C- vn /lib/libm.so.5 > 1373 0x280d9000 0x28211000 r-x 312 0 1 0 CN vn /lib/libcrypto.so= .5 > 1373 0x28211000 0x28212000 r-x 1 0 1 0 CN vn /lib/libcrypto.so= .5 > 1373 0x28212000 0x2822a000 rw- 24 0 1 0 C- vn /lib/libcrypto.so= .5 > 1373 0x2822a000 0x2822c000 rw- 2 0 1 0 C- df=20 > 1373 0x2822c000 0x28232000 r-x 6 0 1 0 CN vn /lib/libkvm.so.4 > 1373 0x28232000 0x28233000 r-x 1 0 1 0 CN vn /lib/libkvm.so.4 > 1373 0x28233000 0x28234000 rw- 1 0 1 0 C- vn /lib/libkvm.so.4 > 1373 0x28234000 0x2824c000 r-x 24 0 1 0 CN vn /usr/lib/libelf.s= o.1 > 1373 0x2824c000 0x2824d000 r-x 1 0 1 0 CN vn /usr/lib/libelf.s= o.1 > 1373 0x2824d000 0x2824e000 rw- 1 0 1 0 C- vn /usr/lib/libelf.s= o.1 > 1373 0x2824e000 0x28251000 r-x 3 0 15 10 CN vn /usr/lib/librt.so= .1 > 1373 0x28251000 0x28252000 r-x 1 0 1 0 CN vn /usr/lib/librt.so= .1 > 1373 0x28252000 0x28253000 rw- 1 0 1 0 C- vn /usr/lib/librt.so= .1 > 1373 0x28253000 0x28260000 r-x 13 0 1 0 CN vn /lib/libmd.so.4 > 1373 0x28260000 0x28261000 r-x 1 0 1 0 CN vn /lib/libmd.so.4 > 1373 0x28261000 0x28262000 rw- 1 0 1 0 C- vn /lib/libmd.so.4 > 1373 0x28262000 0x28351000 r-x 239 0 1 0 CN vn /lib/libc.so.7 > 1373 0x28351000 0x28352000 r-x 1 0 1 0 CN vn /lib/libc.so.7 > 1373 0x28352000 0x28358000 rw- 6 0 1 0 C- vn /lib/libc.so.7 > 1373 0x28358000 0x2836e000 rw- 22 0 1 0 C- df=20 > 1373 0x2836e000 0x2837a000 --- 0 0 0 0 -- --=20 > 1373 0x28400000 0x28500000 rw- 256 0 1 0 C- df=20 > 1373 0xbfbe0000 0xbfc00000 rwx 32 0 1 0 C- df=20 >=20 > rwsrv05# kgdb /spare/obj8/usr/src/sys/RWSRV05/kernel.debug /dev/mem > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you = are > welcome to change it and/or distribute copies of it under certain conditi= ons. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for detail= s. > This GDB was configured as "i386-marcel-freebsd"... > #0 sched_switch (td=3D0xc08ad090, newtd=3D0xc4d4d900, flags=3D260) > at /usr/src/sys/kern/sched_ule.c:1864 > 1864 cpuid =3D PCPU_GET(cpuid); > Ready to go. Enter 'tr' to connect to the remote target > with /dev/cuad0, 'tr /dev/cuad1' to connect to a different port > or 'trf portno' to connect to the remote target with the firewire > interface. portno defaults to 5556. >=20 > Type 'getsyms' after connection to load kld symbols. >=20 > If you're debugging a local system, you can use 'kldsyms' instead > to load the kld symbols. That's a less obnoxious interface. > (kgdb) proc 1373 > [Switching to thread 154 (Thread 100168)]#0 sched_switch (td=3D0xc577624= 0, newtd=3D0xc4d4db40, flags=3D0x104) > at /usr/src/sys/kern/sched_ule.c:1864 > 1864 cpuid =3D PCPU_GET(cpuid); > (kgdb) bt 1373 > #0 sched_switch (td=3D0xc5776240, newtd=3D0xc4d4db40, flags=3D0x104) at = /usr/src/sys/kern/sched_ule.c:1864 > During symbol reading, Incomplete CFI data; unspecified registers at 0xc0= 5fe876. > #1 0xc05e572f in mi_switch (flags=3D0x104, newtd=3D0x0) at /usr/src/sys/= kern/kern_synch.c:444 > #2 0xc06147fc in sleepq_switch (wchan=3D0xc5a5f338, pri=3D0x44) at /usr/= src/sys/kern/subr_sleepqueue.c:505 > #3 0xc0615495 in sleepq_wait (wchan=3D0xc5a5f338, pri=3D0x44) at /usr/sr= c/sys/kern/subr_sleepqueue.c:584 > #4 0xc05e5bd9 in _sleep (ident=3D0xc5a5f338, lock=3D0xc0a243a4, priority= =3D0x244, wmesg=3D0xc08357af "vmmaps", timo=3D0x0) > at /usr/src/sys/kern/kern_synch.c:232 > #5 0xc075f8d7 in vm_map_unlock_and_wait (map=3D0xc5a5f2b8, timo=3D0x0) a= t /usr/src/sys/vm/vm_map.c:638 > #6 0xc075f987 in vm_map_delete (map=3D0xc5a5f2b8, start=3D0x2836e000, en= d=3D0x28374000) at /usr/src/sys/vm/vm_map.c:2703 This frame (vm_map_delete). > #7 0xc076136e in vm_map_fixed (map=3D0xc5a5f2b8, object=3D0xc52ffc38, of= fset=3D0x0, start=3D0x2836e000, length=3D0x6000,=20 > prot=3D0x5, max=3D0x7, cow=3D0x112) at /usr/src/sys/vm/vm_map.c:1367 > #8 0xc0763a48 in vm_mmap (map=3D0xc5a5f2b8, addr=3D0xe7840c70, size=3D0x= 6000, prot=3DVariable "prot" is not available. > ) at /usr/src/sys/vm/vm_mmap.c:1439 > #9 0xc07641ef in mmap (td=3D0xc5776240, uap=3D0xe7840cf8) at /usr/src/sy= s/vm/vm_mmap.c:390 > #10 0xc07b955f in syscall (frame=3D0xe7840d38) at /usr/src/sys/i386/i386/= trap.c:1073 > #11 0xc079dff0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception= .s:261 > #12 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) p *0xc5a5f2b8=20 > $1 =3D 0xc58cb048 This is useless. You need to do either p *entry or p *(struct vm_map_entry *)
>=20 > --=20 > John Marshall --SqEPCaGsKgsesFh5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkpW+pwACgkQC3+MBN1Mb4gk7gCgjoPkx+e6gr7g7GRVnPrA4ECS 6FcAnRw6HMf7TIXSVGC1YyBSWod6eepu =FEZj -----END PGP SIGNATURE----- --SqEPCaGsKgsesFh5--