Date: Fri, 10 Jul 2009 21:42:34 +1000 From: John Marshall <john.marshall@riverwillow.com.au> To: freebsd-current@freebsd.org Subject: Re: Process stuck in vmmaps on 8.0-BETA1 Message-ID: <20090710114234.GF32316@rwpc12.mby.riverwillow.net.au> In-Reply-To: <1247214510.2437.1693.camel@strangepork.london.mintel.ad> References: <20090709060556.GA27373@rwpc12.mby.riverwillow.net.au> <a31046fc0907082342g1343b138g2fc38fda890b582a@mail.gmail.com> <20090709073054.GB27373@rwpc12.mby.riverwillow.net.au> <20090709085242.GC27373@rwpc12.mby.riverwillow.net.au> <20090709142121.GS55190@deviant.kiev.zoral.com.ua> <20090710035849.GB31950@rwpc12.mby.riverwillow.net.au> <1247214510.2437.1693.camel@strangepork.london.mintel.ad>
next in thread | previous in thread | raw e-mail | index | archive | help
--KuLpqunXa7jZSBt+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, 10 Jul 2009, 09:28 +0100, Tom Evans wrote:
> On Fri, 2009-07-10 at 13:58 +1000, John Marshall wrote:
> > On Thu, 09 Jul 2009, 17:21 +0300, Kostik Belousov wrote:
> > > Since you have mostly working system, and interesting information most
> > > easy accessible by kgdb, attach it to the live kernel:
> > > kgdb <path to kernel.debug> /dev/mem
> > >=20
> > > From there, switch to the stuck process,
> > > process <pid>
> >=20
> > I tried that...
> >=20
> > (kgdb) process 1373
> > Undefined command: "process". Try "help".
> >=20
> > It took me several more hours to discover "proc" which I assume is what
> > you meant?
> >=20
> > > do
> > > bt
> > > find the frame for vm_map_delete, and print the entry:
> > > p entry
> >=20
> > I have no idea which number(s) to plug in here. I hope I guessed the
> > right one.
> >=20
> > > Also, I need to see the information you posted earlier, namely, procs=
tat
> > > -k and -v output for the process.
> >=20
> At this stage, you want to type 'f 6', to go to frame 6, (the
> vm_map_delete function) and then type 'p *entry' to print the struct
> pointed to by the local variable entry in that frame.
Thank you Tom! You could see I was struggling. Now, hopefully, I can
provide something useful this time.
rwsrv05# procstat 1270
PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM =
=20
1270 1 1270 1270 0 1 john vmmaps FreeBSD ELF32 ntpd =
=20
rwsrv05# procstat -k 1270
PID TID COMM TDNAME KSTACK =
=20
1270 100184 ntpd - mi_switch sleepq_switch slee=
pq_wait _sleep vm_map_unlock_and_wait vm_map_delete vm_map_fixed vm_mmap mm=
ap syscall Xint0x80_syscall=20
rwsrv05# procstat -v 1270
PID START END PRT RES PRES REF SHD FL TP PATH
1270 0x8048000 0x807e000 r-x 54 60 2 1 CN vn /usr/local/bin/ntpd
1270 0x807e000 0x8080000 rw- 2 0 1 0 C- vn /usr/local/bin/ntpd
1270 0x8080000 0x8100000 rw- 128 0 1 0 C- df=20
1270 0x2807e000 0x280ab000 r-x 45 0 170 75 CN vn /libexec/ld-elf.so.1
1270 0x280ab000 0x280ad000 rw- 2 0 1 0 C- vn /libexec/ld-elf.so.1
1270 0x280ad000 0x280c0000 rw- 19 0 1 0 C- df=20
1270 0x280c0000 0x280d7000 r-x 23 0 1 0 CN vn /lib/libm.so.5
1270 0x280d7000 0x280d8000 r-x 1 0 1 0 CN vn /lib/libm.so.5
1270 0x280d8000 0x280d9000 rw- 1 0 1 0 C- vn /lib/libm.so.5
1270 0x280d9000 0x28211000 r-x 312 0 1 0 CN vn /lib/libcrypto.so.5
1270 0x28211000 0x28212000 r-x 1 0 1 0 CN vn /lib/libcrypto.so.5
1270 0x28212000 0x2822a000 rw- 24 0 1 0 C- vn /lib/libcrypto.so.5
1270 0x2822a000 0x2822c000 rw- 2 0 1 0 C- df=20
1270 0x2822c000 0x28232000 r-x 6 0 1 0 CN vn /lib/libkvm.so.4
1270 0x28232000 0x28233000 r-x 1 0 1 0 CN vn /lib/libkvm.so.4
1270 0x28233000 0x28234000 rw- 1 0 1 0 C- vn /lib/libkvm.so.4
1270 0x28234000 0x2824c000 r-x 24 0 1 0 CN vn /usr/lib/libelf.so.1
1270 0x2824c000 0x2824d000 r-x 1 0 1 0 CN vn /usr/lib/libelf.so.1
1270 0x2824d000 0x2824e000 rw- 1 0 1 0 C- vn /usr/lib/libelf.so.1
1270 0x2824e000 0x28251000 r-x 3 0 15 10 CN vn /usr/lib/librt.so.1
1270 0x28251000 0x28252000 r-x 1 0 1 0 CN vn /usr/lib/librt.so.1
1270 0x28252000 0x28253000 rw- 1 0 1 0 C- vn /usr/lib/librt.so.1
1270 0x28253000 0x28260000 r-x 13 0 1 0 CN vn /lib/libmd.so.4
1270 0x28260000 0x28261000 r-x 1 0 1 0 CN vn /lib/libmd.so.4
1270 0x28261000 0x28262000 rw- 1 0 1 0 C- vn /lib/libmd.so.4
1270 0x28262000 0x28351000 r-x 239 0 1 0 CN vn /lib/libc.so.7
1270 0x28351000 0x28352000 r-x 1 0 1 0 CN vn /lib/libc.so.7
1270 0x28352000 0x28358000 rw- 6 0 1 0 C- vn /lib/libc.so.7
1270 0x28358000 0x2836e000 rw- 22 0 1 0 C- df=20
1270 0x2836e000 0x2837a000 --- 0 0 0 0 -- --=20
1270 0x28400000 0x28500000 rw- 256 0 1 0 C- df=20
1270 0xbfbe0000 0xbfc00000 rwx 32 0 1 0 C- df=20
rwsrv05# kgdb kernel.debug /dev/mem
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
#0 sched_switch (td=3D0xc08af410, newtd=3D0xc4d4db40, flags=3D260)
at /usr/src/sys/kern/sched_ule.c:1864
1864 cpuid =3D PCPU_GET(cpuid);
Ready to go. Enter 'tr' to connect to the remote target
with /dev/cuad0, 'tr /dev/cuad1' to connect to a different port
or 'trf portno' to connect to the remote target with the firewire
interface. portno defaults to 5556.
Type 'getsyms' after connection to load kld symbols.
If you're debugging a local system, you can use 'kldsyms' instead
to load the kld symbols. That's a less obnoxious interface.
(kgdb) proc 1270
[Switching to thread 168 (Thread 100184)]#0 sched_switch (td=3D0xc592fd80,=
newtd=3D0xc4d4db40, flags=3D0x104)
at /usr/src/sys/kern/sched_ule.c:1864
1864 cpuid =3D PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=3D0xc592fd80, newtd=3D0xc4d4db40, flags=3D0x104) at /u=
sr/src/sys/kern/sched_ule.c:1864
During symbol reading, Incomplete CFI data; unspecified registers at 0xc060=
09d6.
#1 0xc05e788f in mi_switch (flags=3D0x104, newtd=3D0x0) at /usr/src/sys/ke=
rn/kern_synch.c:444
#2 0xc061695c in sleepq_switch (wchan=3D0xc5909f00, pri=3D0x44) at /usr/sr=
c/sys/kern/subr_sleepqueue.c:505
#3 0xc06175f5 in sleepq_wait (wchan=3D0xc5909f00, pri=3D0x44) at /usr/src/=
sys/kern/subr_sleepqueue.c:584
#4 0xc05e7d39 in _sleep (ident=3D0xc5909f00, lock=3D0xc0a26724, priority=
=3D0x244, wmesg=3D0xc0837aa0 "vmmaps", timo=3D0x0)
at /usr/src/sys/kern/kern_synch.c:232
#5 0xc0761a37 in vm_map_unlock_and_wait (map=3D0xc5909e80, timo=3D0x0) at =
/usr/src/sys/vm/vm_map.c:638
#6 0xc0761ae7 in vm_map_delete (map=3D0xc5909e80, start=3D0x2836e000, end=
=3D0x28374000) at /usr/src/sys/vm/vm_map.c:2703
#7 0xc07634ce in vm_map_fixed (map=3D0xc5909e80, object=3D0xc5254990, offs=
et=3D0x0, start=3D0x2836e000, length=3D0x6000,=20
prot=3D0x5, max=3D0x7, cow=3D0x112) at /usr/src/sys/vm/vm_map.c:1367
#8 0xc0765ba8 in vm_mmap (map=3D0xc5909e80, addr=3D0xe787ac70, size=3D0x60=
00, prot=3DVariable "prot" is not available.
) at /usr/src/sys/vm/vm_mmap.c:1439
#9 0xc076634f in mmap (td=3D0xc592fd80, uap=3D0xe787acf8) at /usr/src/sys/=
vm/vm_mmap.c:390
#10 0xc07bb6bf in syscall (frame=3D0xe787ad38) at /usr/src/sys/i386/i386/tr=
ap.c:1073
#11 0xc07a0150 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s=
:261
#12 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) f 6
#6 0xc0761ae7 in vm_map_delete (map=3D0xc5909e80, start=3D0x2836e000, end=
=3D0x28374000) at /usr/src/sys/vm/vm_map.c:2703
2703 (void) vm_map_unlock_and_wait(map, 0);
(kgdb) p *entry
$1 =3D {
prev =3D 0xc5812b40,=20
next =3D 0xc59ec7e0,=20
left =3D 0xc5812b40,=20
right =3D 0xc59ec7e0,=20
start =3D 0x2836e000,=20
end =3D 0x2837a000,=20
avail_ssize =3D 0x0,=20
adj_free =3D 0x86000,=20
max_free =3D 0x976e0000,=20
object =3D {
vm_object =3D 0x0,=20
sub_map =3D 0x0
},=20
offset =3D 0x0,=20
eflags =3D 0x600,=20
protection =3D 0x0,=20
max_protection =3D 0x7,=20
inheritance =3D 0x1,=20
wired_count =3D 0xffffffff,=20
lastr =3D 0x1c,=20
uip =3D 0x0
}
(kgdb)=20
--=20
John Marshall
--KuLpqunXa7jZSBt+
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (FreeBSD)
iEYEARECAAYFAkpXKSoACgkQw/tAaKKahKLGrACffJpDtsJM7iwHCQz+hFCPAX6t
HW4AoLvMgKIMZ3qTRrHyTdoDJSN/znzk
=ab+c
-----END PGP SIGNATURE-----
--KuLpqunXa7jZSBt+--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090710114234.GF32316>