Date: Fri, 5 Jan 2007 15:08:57 +0000 From: Ceri Davies <ceri@submonkey.net> To: Robert Watson <rwatson@FreeBSD.org> Cc: stable@FreeBSD.org Subject: Re: (audit?) Panic in 6.2-PRERELEASE Message-ID: <20070105150857.GC7088@submonkey.net> In-Reply-To: <20070105133028.F98541@fledge.watson.org> References: <20070105111954.GA51511@submonkey.net> <20070105120539.H46119@fledge.watson.org> <20070105131528.GB7088@submonkey.net> <20070105133028.F98541@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--NKoe5XOeduwbEQHU
Content-Type: multipart/mixed; boundary="VrqPEDrXMn8OVzN4"
Content-Disposition: inline
--VrqPEDrXMn8OVzN4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Jan 05, 2007 at 01:34:04PM +0000, Robert Watson wrote:
>=20
> On Fri, 5 Jan 2007, Ceri Davies wrote:
>=20
> >>Much as I would love to trust the contents of ub there, I suspect they=
=20
> >>can't be trusted. Could you print the contents of *fp in kern_fstat() =
in=20
> >>both of those stacks? I'd particularly like to know the value of=20
> >>fp->f_type, and then depending on the type, possibly the contents of=20
> >>*(struct vnode *)fp->f_vnode for DTYPE_VNODE/TYPE_FIFO or *(struct sock=
et=20
> >>*)fp->f_data in the case of DTYPE_SOCKET.
> >
> >Can you tell me how to get at *fp given that the stack trace shows fstat=
()=20
> >and not kern_fstat()? Sorry if I'm being dumb but I don't know how to=
=20
> >step into the kern_fstat() call from fstat().
>=20
> It could be that the stack is hosed losing the frame, or maybe it's inlin=
ed=20
> (more likely the former I think, as kern_fstat() is a symbol used elsewhe=
re=20
> in the kernel). The best bet may be to use the file descriptor number=20
> (uap->fd) to pull the struct file reference out of the process. Somethin=
g=20
> on the order of (td->td_proc->p_fd->fd_ofiles[fd]) should return the righ=
t=20
> struct file *.
OK, got it. They're both sockets, data in the attachments.
> How reproduceable is this?
So far it's happened this morning and yesterday morning. I haven't seen
it before that. I don't know the cause so I can't reproduce it at will,
but the logs don't give any indication. Chances are that it will happen
again tomorrow, but we'll see.
Thanks,
Ceri
--=20
That must be wonderful! I don't understand it at all.
-- Moliere
--VrqPEDrXMn8OVzN4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=panic1
Content-Transfer-Encoding: quoted-printable
{root@shrike}-{~} # kgdb /usr/obj/usr/src/sys/SHRIKE/kernel.debug /var/cras=
h/vmcore.29
kgdb: kvm_nlist(_stopped_cpus):=20
kgdb: kvm_nlist(_stoppcbs):=20
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:=
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address =3D 0x53892047
fault code =3D supervisor write, page not present
instruction pointer =3D 0x20:0xc05cda7c
stack pointer =3D 0x28:0xd610dc48
frame pointer =3D 0x28:0xd610dc60
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, def32 1, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 65381 (imapd)
trap number =3D 12
panic: page fault
Uptime: 5d19h44m40s
Dumping 503 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327=
311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) up 8
#8 0xc04c470d in fstat (td=3D0xc2eeb180, uap=3D0xd610dc74) at /usr/src/sys=
/kern/kern_descrip.c:1075
1075 error =3D kern_fstat(td, uap->fd, &ub);
(kgdb) p *td->td_proc->p_fd->fd_ofiles
$1 =3D (struct file *) 0xc32f73f0
(kgdb) p*$1
$2 =3D {f_list =3D {le_next =3D 0xc32ddd38, le_prev =3D 0xc4062048}, f_type=
=3D 2, f_data =3D 0xc38f62c8,=20
f_flag =3D 3, f_mtxp =3D 0xc2a67154, f_ops =3D 0xc06b1040, f_cred =3D 0xc=
2e4a580, f_count =3D 3,=20
f_vnode =3D 0x0, f_offset =3D 0, f_vnread_flags =3D 0, f_gcflag =3D 0, f_=
msgcount =3D 0, f_seqcount =3D 0,=20
f_nextoff =3D 0, f_label =3D 0x0}
(kgdb) p $2->f_data
$3 =3D (void *) 0xc38f62c8
(kgdb) p *(struct socket *)$2->f_data
$4 =3D {so_count =3D 1, so_type =3D 1, so_options =3D 4, so_linger =3D 0, s=
o_state =3D 2, so_qstate =3D 0,=20
so_pcb =3D 0xc38eaec4, so_proto =3D 0xc06b8148, so_head =3D 0x0, so_incom=
p =3D {tqh_first =3D 0x0,=20
tqh_last =3D 0x0}, so_comp =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, s=
o_list =3D {tqe_next =3D 0x0,=20
tqe_prev =3D 0xc2e5087c}, so_qlen =3D 0, so_incqlen =3D 0, so_qlimit =
=3D 0, so_timeo =3D 0,=20
so_error =3D 0, so_sigio =3D 0x0, so_oobmark =3D 0, so_aiojobq =3D {tqh_f=
irst =3D 0x0,=20
tqh_last =3D 0xc38f6310}, so_rcv =3D {sb_sel =3D {si_thrlist =3D {tqe_n=
ext =3D 0x0,=20
tqe_prev =3D 0xc2eeb1b0}, si_thread =3D 0x0, si_note =3D {kl_list =
=3D {slh_first =3D 0x0},=20
kl_lock =3D 0xc04cd13c <knlist_mtx_lock>, kl_unlock =3D 0xc04cd170 =
<knlist_mtx_unlock>,=20
kl_locked =3D 0xc04cd1ac <knlist_mtx_locked>, kl_lockarg =3D 0xc38f=
633c}, si_flags =3D 0},=20
sb_mtx =3D {mtx_object =3D {lo_class =3D 0xc06ad4c4, lo_name =3D 0xc068=
133e "so_rcv",=20
lo_type =3D 0xc068133e "so_rcv", lo_flags =3D 196608, lo_list =3D {=
tqe_next =3D 0x0,=20
tqe_prev =3D 0x0}, lo_witness =3D 0x0}, mtx_lock =3D 4, mtx_recur=
se =3D 0}, sb_state =3D 0,=20
sb_mb =3D 0x0, sb_mbtail =3D 0x0, sb_lastrecord =3D 0x0, sb_cc =3D 0, s=
b_hiwat =3D 66608, sb_mbcnt =3D 0,=20
sb_mbmax =3D 262144, sb_ctl =3D 0, sb_lowat =3D 1, sb_timeo =3D 0, sb_f=
lags =3D 0}, so_snd =3D {sb_sel =3D {
si_thrlist =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, si_thread =3D 0x=
0, si_note =3D {kl_list =3D {
slh_first =3D 0x0}, kl_lock =3D 0xc04cd13c <knlist_mtx_lock>,=20
kl_unlock =3D 0xc04cd170 <knlist_mtx_unlock>, kl_locked =3D 0xc04cd=
1ac <knlist_mtx_locked>,=20
kl_lockarg =3D 0xc38f63b4}, si_flags =3D 0}, sb_mtx =3D {mtx_object=
=3D {lo_class =3D 0xc06ad4c4,=20
lo_name =3D 0xc0681337 "so_snd", lo_type =3D 0xc0681337 "so_snd", l=
o_flags =3D 196608,=20
lo_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, lo_witness =3D 0x=
0}, mtx_lock =3D 4,=20
mtx_recurse =3D 0}, sb_state =3D 0, sb_mb =3D 0x0, sb_mbtail =3D 0x0,=
sb_lastrecord =3D 0x0,=20
sb_cc =3D 0, sb_hiwat =3D 33304, sb_mbcnt =3D 0, sb_mbmax =3D 262144, s=
b_ctl =3D 0, sb_lowat =3D 2048,=20
sb_timeo =3D 0, sb_flags =3D 0}, so_upcall =3D 0, so_upcallarg =3D 0x0,=
so_cred =3D 0xc2a7ad00,=20
so_label =3D 0x0, so_peerlabel =3D 0x0, so_gencnt =3D 92385, so_emuldata =
=3D 0x0, so_accf =3D 0x0}
(kgdb)
--VrqPEDrXMn8OVzN4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=panic2
Content-Transfer-Encoding: quoted-printable
{root@shrike}=1B[m-{~} # kgdb /usr/obj/usr/src/sys/SHRIKE/kernel.debug /var=
/crash/vmcore.30
kgdb: kvm_nlist(_stopped_cpus):=20
kgdb: kvm_nlist(_stoppcbs):=20
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:=
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address =3D 0x53892047
fault code =3D supervisor write, page not present
instruction pointer =3D 0x20:0xc05cda7c
stack pointer =3D 0x28:0xd617ec48
frame pointer =3D 0x28:0xd617ec60
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, def32 1, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 9943 (imapd)
trap number =3D 12
panic: page fault
Uptime: 22h39m3s
Dumping 503 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327=
311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) up 8
#8 0xc04c470d in fstat (td=3D0xc3109300, uap=3D0xd617ec74) at /usr/src/sys=
/kern/kern_descrip.c:1075
1075 error =3D kern_fstat(td, uap->fd, &ub);
(kgdb) p *td->td_proc->p_fd->fd_ofiles
$1 =3D (struct file *) 0xc33fd1f8
(kgdb) p *$1
$2 =3D {f_list =3D {le_next =3D 0xc30a6678, le_prev =3D 0xc3790b88}, f_type=
=3D 2, f_data =3D 0xc347f590,=20
f_flag =3D 3, f_mtxp =3D 0xc2a67a30, f_ops =3D 0xc06b1040, f_cred =3D 0xc=
3592a80, f_count =3D 3,=20
f_vnode =3D 0x0, f_offset =3D 0, f_vnread_flags =3D 0, f_gcflag =3D 0, f_=
msgcount =3D 0, f_seqcount =3D 0,=20
f_nextoff =3D 0, f_label =3D 0x0}
(kgdb) p *(struct socket *)$2->f_data
$3 =3D {so_count =3D 1, so_type =3D 1, so_options =3D 4, so_linger =3D 0, s=
o_state =3D 2, so_qstate =3D 0,=20
so_pcb =3D 0xc317b168, so_proto =3D 0xc06b8148, so_head =3D 0x0, so_incom=
p =3D {tqh_first =3D 0x0,=20
tqh_last =3D 0x0}, so_comp =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, s=
o_list =3D {tqe_next =3D 0x0,=20
tqe_prev =3D 0xc2e5ab44}, so_qlen =3D 0, so_incqlen =3D 0, so_qlimit =
=3D 0, so_timeo =3D 0,=20
so_error =3D 0, so_sigio =3D 0x0, so_oobmark =3D 0, so_aiojobq =3D {tqh_f=
irst =3D 0x0,=20
tqh_last =3D 0xc347f5d8}, so_rcv =3D {sb_sel =3D {si_thrlist =3D {tqe_n=
ext =3D 0x0,=20
tqe_prev =3D 0xc3109330}, si_thread =3D 0x0, si_note =3D {kl_list =
=3D {slh_first =3D 0x0},=20
kl_lock =3D 0xc04cd13c <knlist_mtx_lock>, kl_unlock =3D 0xc04cd170 =
<knlist_mtx_unlock>,=20
kl_locked =3D 0xc04cd1ac <knlist_mtx_locked>, kl_lockarg =3D 0xc347=
f604}, si_flags =3D 0},=20
sb_mtx =3D {mtx_object =3D {lo_class =3D 0xc06ad4c4, lo_name =3D 0xc068=
133e "so_rcv",=20
lo_type =3D 0xc068133e "so_rcv", lo_flags =3D 196608, lo_list =3D {=
tqe_next =3D 0x0,=20
tqe_prev =3D 0x0}, lo_witness =3D 0x0}, mtx_lock =3D 4, mtx_recur=
se =3D 0}, sb_state =3D 0,=20
sb_mb =3D 0x0, sb_mbtail =3D 0x0, sb_lastrecord =3D 0x0, sb_cc =3D 0, s=
b_hiwat =3D 66608, sb_mbcnt =3D 0,=20
sb_mbmax =3D 262144, sb_ctl =3D 0, sb_lowat =3D 1, sb_timeo =3D 0, sb_f=
lags =3D 0}, so_snd =3D {sb_sel =3D {
si_thrlist =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, si_thread =3D 0x=
0, si_note =3D {kl_list =3D {
slh_first =3D 0x0}, kl_lock =3D 0xc04cd13c <knlist_mtx_lock>,=20
kl_unlock =3D 0xc04cd170 <knlist_mtx_unlock>, kl_locked =3D 0xc04cd=
1ac <knlist_mtx_locked>,=20
kl_lockarg =3D 0xc347f67c}, si_flags =3D 0}, sb_mtx =3D {mtx_object=
=3D {lo_class =3D 0xc06ad4c4,=20
lo_name =3D 0xc0681337 "so_snd", lo_type =3D 0xc0681337 "so_snd", l=
o_flags =3D 196608,=20
lo_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, lo_witness =3D 0x=
0}, mtx_lock =3D 4,=20
mtx_recurse =3D 0}, sb_state =3D 0, sb_mb =3D 0x0, sb_mbtail =3D 0x0,=
sb_lastrecord =3D 0x0,=20
sb_cc =3D 0, sb_hiwat =3D 33304, sb_mbcnt =3D 0, sb_mbmax =3D 262144, s=
b_ctl =3D 0, sb_lowat =3D 2048,=20
sb_timeo =3D 0, sb_flags =3D 0}, so_upcall =3D 0, so_upcallarg =3D 0x0,=
so_cred =3D 0xc2a7ad00,=20
so_label =3D 0x0, so_peerlabel =3D 0x0, so_gencnt =3D 22107, so_emuldata =
=3D 0x0, so_accf =3D 0x0}
(kgdb)
--VrqPEDrXMn8OVzN4--
--NKoe5XOeduwbEQHU
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (FreeBSD)
iD8DBQFFnmoJocfcwTS3JF8RAvRrAJ0cIWI5KunoJMHvGiGdyFp3FfNYAgCgtMY7
FcV0jf4O/FUWBUijhF8d+4U=
=sSv7
-----END PGP SIGNATURE-----
--NKoe5XOeduwbEQHU--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070105150857.GC7088>