From owner-freebsd-stable@FreeBSD.ORG Fri Jan 5 15:09:09 2007 Return-Path: X-Original-To: stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A65AE16A403; Fri, 5 Jan 2007 15:09:09 +0000 (UTC) (envelope-from ceri@submonkey.net) Received: from shrike.submonkey.net (cpc2-cdif2-0-0-cust107.cdif.cable.ntl.com [81.104.168.108]) by mx1.freebsd.org (Postfix) with ESMTP id EE5FD13C478; Fri, 5 Jan 2007 15:09:08 +0000 (UTC) (envelope-from ceri@submonkey.net) Received: from ceri by shrike.submonkey.net with local (Exim 4.64 (FreeBSD)) (envelope-from ) id 1H2qgj-000JmE-OU; Fri, 05 Jan 2007 15:08:57 +0000 Date: Fri, 5 Jan 2007 15:08:57 +0000 From: Ceri Davies To: Robert Watson Message-ID: <20070105150857.GC7088@submonkey.net> References: <20070105111954.GA51511@submonkey.net> <20070105120539.H46119@fledge.watson.org> <20070105131528.GB7088@submonkey.net> <20070105133028.F98541@fledge.watson.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NKoe5XOeduwbEQHU" Content-Disposition: inline In-Reply-To: <20070105133028.F98541@fledge.watson.org> X-PGP: finger ceri@FreeBSD.org User-Agent: Mutt/1.5.13 (2006-08-11) Sender: Ceri Davies Cc: stable@FreeBSD.org Subject: Re: (audit?) Panic in 6.2-PRERELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jan 2007 15:09:09 -0000 --NKoe5XOeduwbEQHU Content-Type: multipart/mixed; boundary="VrqPEDrXMn8OVzN4" Content-Disposition: inline --VrqPEDrXMn8OVzN4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jan 05, 2007 at 01:34:04PM +0000, Robert Watson wrote: >=20 > On Fri, 5 Jan 2007, Ceri Davies wrote: >=20 > >>Much as I would love to trust the contents of ub there, I suspect they= =20 > >>can't be trusted. Could you print the contents of *fp in kern_fstat() = in=20 > >>both of those stacks? I'd particularly like to know the value of=20 > >>fp->f_type, and then depending on the type, possibly the contents of=20 > >>*(struct vnode *)fp->f_vnode for DTYPE_VNODE/TYPE_FIFO or *(struct sock= et=20 > >>*)fp->f_data in the case of DTYPE_SOCKET. > > > >Can you tell me how to get at *fp given that the stack trace shows fstat= ()=20 > >and not kern_fstat()? Sorry if I'm being dumb but I don't know how to= =20 > >step into the kern_fstat() call from fstat(). >=20 > It could be that the stack is hosed losing the frame, or maybe it's inlin= ed=20 > (more likely the former I think, as kern_fstat() is a symbol used elsewhe= re=20 > in the kernel). The best bet may be to use the file descriptor number=20 > (uap->fd) to pull the struct file reference out of the process. Somethin= g=20 > on the order of (td->td_proc->p_fd->fd_ofiles[fd]) should return the righ= t=20 > struct file *. OK, got it. They're both sockets, data in the attachments. > How reproduceable is this? So far it's happened this morning and yesterday morning. I haven't seen it before that. I don't know the cause so I can't reproduce it at will, but the logs don't give any indication. Chances are that it will happen again tomorrow, but we'll see. Thanks, Ceri --=20 That must be wonderful! I don't understand it at all. -- Moliere --VrqPEDrXMn8OVzN4 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=panic1 Content-Transfer-Encoding: quoted-printable {root@shrike}-{~} # kgdb /usr/obj/usr/src/sys/SHRIKE/kernel.debug /var/cras= h/vmcore.29 kgdb: kvm_nlist(_stopped_cpus):=20 kgdb: kvm_nlist(_stoppcbs):=20 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0x53892047 fault code =3D supervisor write, page not present instruction pointer =3D 0x20:0xc05cda7c stack pointer =3D 0x28:0xd610dc48 frame pointer =3D 0x28:0xd610dc60 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 65381 (imapd) trap number =3D 12 panic: page fault Uptime: 5d19h44m40s Dumping 503 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327= 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) up 8 #8 0xc04c470d in fstat (td=3D0xc2eeb180, uap=3D0xd610dc74) at /usr/src/sys= /kern/kern_descrip.c:1075 1075 error =3D kern_fstat(td, uap->fd, &ub); (kgdb) p *td->td_proc->p_fd->fd_ofiles $1 =3D (struct file *) 0xc32f73f0 (kgdb) p*$1 $2 =3D {f_list =3D {le_next =3D 0xc32ddd38, le_prev =3D 0xc4062048}, f_type= =3D 2, f_data =3D 0xc38f62c8,=20 f_flag =3D 3, f_mtxp =3D 0xc2a67154, f_ops =3D 0xc06b1040, f_cred =3D 0xc= 2e4a580, f_count =3D 3,=20 f_vnode =3D 0x0, f_offset =3D 0, f_vnread_flags =3D 0, f_gcflag =3D 0, f_= msgcount =3D 0, f_seqcount =3D 0,=20 f_nextoff =3D 0, f_label =3D 0x0} (kgdb) p $2->f_data $3 =3D (void *) 0xc38f62c8 (kgdb) p *(struct socket *)$2->f_data $4 =3D {so_count =3D 1, so_type =3D 1, so_options =3D 4, so_linger =3D 0, s= o_state =3D 2, so_qstate =3D 0,=20 so_pcb =3D 0xc38eaec4, so_proto =3D 0xc06b8148, so_head =3D 0x0, so_incom= p =3D {tqh_first =3D 0x0,=20 tqh_last =3D 0x0}, so_comp =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, s= o_list =3D {tqe_next =3D 0x0,=20 tqe_prev =3D 0xc2e5087c}, so_qlen =3D 0, so_incqlen =3D 0, so_qlimit = =3D 0, so_timeo =3D 0,=20 so_error =3D 0, so_sigio =3D 0x0, so_oobmark =3D 0, so_aiojobq =3D {tqh_f= irst =3D 0x0,=20 tqh_last =3D 0xc38f6310}, so_rcv =3D {sb_sel =3D {si_thrlist =3D {tqe_n= ext =3D 0x0,=20 tqe_prev =3D 0xc2eeb1b0}, si_thread =3D 0x0, si_note =3D {kl_list = =3D {slh_first =3D 0x0},=20 kl_lock =3D 0xc04cd13c , kl_unlock =3D 0xc04cd170 = ,=20 kl_locked =3D 0xc04cd1ac , kl_lockarg =3D 0xc38f= 633c}, si_flags =3D 0},=20 sb_mtx =3D {mtx_object =3D {lo_class =3D 0xc06ad4c4, lo_name =3D 0xc068= 133e "so_rcv",=20 lo_type =3D 0xc068133e "so_rcv", lo_flags =3D 196608, lo_list =3D {= tqe_next =3D 0x0,=20 tqe_prev =3D 0x0}, lo_witness =3D 0x0}, mtx_lock =3D 4, mtx_recur= se =3D 0}, sb_state =3D 0,=20 sb_mb =3D 0x0, sb_mbtail =3D 0x0, sb_lastrecord =3D 0x0, sb_cc =3D 0, s= b_hiwat =3D 66608, sb_mbcnt =3D 0,=20 sb_mbmax =3D 262144, sb_ctl =3D 0, sb_lowat =3D 1, sb_timeo =3D 0, sb_f= lags =3D 0}, so_snd =3D {sb_sel =3D { si_thrlist =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, si_thread =3D 0x= 0, si_note =3D {kl_list =3D { slh_first =3D 0x0}, kl_lock =3D 0xc04cd13c ,=20 kl_unlock =3D 0xc04cd170 , kl_locked =3D 0xc04cd= 1ac ,=20 kl_lockarg =3D 0xc38f63b4}, si_flags =3D 0}, sb_mtx =3D {mtx_object= =3D {lo_class =3D 0xc06ad4c4,=20 lo_name =3D 0xc0681337 "so_snd", lo_type =3D 0xc0681337 "so_snd", l= o_flags =3D 196608,=20 lo_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, lo_witness =3D 0x= 0}, mtx_lock =3D 4,=20 mtx_recurse =3D 0}, sb_state =3D 0, sb_mb =3D 0x0, sb_mbtail =3D 0x0,= sb_lastrecord =3D 0x0,=20 sb_cc =3D 0, sb_hiwat =3D 33304, sb_mbcnt =3D 0, sb_mbmax =3D 262144, s= b_ctl =3D 0, sb_lowat =3D 2048,=20 sb_timeo =3D 0, sb_flags =3D 0}, so_upcall =3D 0, so_upcallarg =3D 0x0,= so_cred =3D 0xc2a7ad00,=20 so_label =3D 0x0, so_peerlabel =3D 0x0, so_gencnt =3D 92385, so_emuldata = =3D 0x0, so_accf =3D 0x0} (kgdb) --VrqPEDrXMn8OVzN4 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=panic2 Content-Transfer-Encoding: quoted-printable {root@shrike}=1B[m-{~} # kgdb /usr/obj/usr/src/sys/SHRIKE/kernel.debug /var= /crash/vmcore.30 kgdb: kvm_nlist(_stopped_cpus):=20 kgdb: kvm_nlist(_stoppcbs):=20 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0x53892047 fault code =3D supervisor write, page not present instruction pointer =3D 0x20:0xc05cda7c stack pointer =3D 0x28:0xd617ec48 frame pointer =3D 0x28:0xd617ec60 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 9943 (imapd) trap number =3D 12 panic: page fault Uptime: 22h39m3s Dumping 503 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327= 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) up 8 #8 0xc04c470d in fstat (td=3D0xc3109300, uap=3D0xd617ec74) at /usr/src/sys= /kern/kern_descrip.c:1075 1075 error =3D kern_fstat(td, uap->fd, &ub); (kgdb) p *td->td_proc->p_fd->fd_ofiles $1 =3D (struct file *) 0xc33fd1f8 (kgdb) p *$1 $2 =3D {f_list =3D {le_next =3D 0xc30a6678, le_prev =3D 0xc3790b88}, f_type= =3D 2, f_data =3D 0xc347f590,=20 f_flag =3D 3, f_mtxp =3D 0xc2a67a30, f_ops =3D 0xc06b1040, f_cred =3D 0xc= 3592a80, f_count =3D 3,=20 f_vnode =3D 0x0, f_offset =3D 0, f_vnread_flags =3D 0, f_gcflag =3D 0, f_= msgcount =3D 0, f_seqcount =3D 0,=20 f_nextoff =3D 0, f_label =3D 0x0} (kgdb) p *(struct socket *)$2->f_data $3 =3D {so_count =3D 1, so_type =3D 1, so_options =3D 4, so_linger =3D 0, s= o_state =3D 2, so_qstate =3D 0,=20 so_pcb =3D 0xc317b168, so_proto =3D 0xc06b8148, so_head =3D 0x0, so_incom= p =3D {tqh_first =3D 0x0,=20 tqh_last =3D 0x0}, so_comp =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, s= o_list =3D {tqe_next =3D 0x0,=20 tqe_prev =3D 0xc2e5ab44}, so_qlen =3D 0, so_incqlen =3D 0, so_qlimit = =3D 0, so_timeo =3D 0,=20 so_error =3D 0, so_sigio =3D 0x0, so_oobmark =3D 0, so_aiojobq =3D {tqh_f= irst =3D 0x0,=20 tqh_last =3D 0xc347f5d8}, so_rcv =3D {sb_sel =3D {si_thrlist =3D {tqe_n= ext =3D 0x0,=20 tqe_prev =3D 0xc3109330}, si_thread =3D 0x0, si_note =3D {kl_list = =3D {slh_first =3D 0x0},=20 kl_lock =3D 0xc04cd13c , kl_unlock =3D 0xc04cd170 = ,=20 kl_locked =3D 0xc04cd1ac , kl_lockarg =3D 0xc347= f604}, si_flags =3D 0},=20 sb_mtx =3D {mtx_object =3D {lo_class =3D 0xc06ad4c4, lo_name =3D 0xc068= 133e "so_rcv",=20 lo_type =3D 0xc068133e "so_rcv", lo_flags =3D 196608, lo_list =3D {= tqe_next =3D 0x0,=20 tqe_prev =3D 0x0}, lo_witness =3D 0x0}, mtx_lock =3D 4, mtx_recur= se =3D 0}, sb_state =3D 0,=20 sb_mb =3D 0x0, sb_mbtail =3D 0x0, sb_lastrecord =3D 0x0, sb_cc =3D 0, s= b_hiwat =3D 66608, sb_mbcnt =3D 0,=20 sb_mbmax =3D 262144, sb_ctl =3D 0, sb_lowat =3D 1, sb_timeo =3D 0, sb_f= lags =3D 0}, so_snd =3D {sb_sel =3D { si_thrlist =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, si_thread =3D 0x= 0, si_note =3D {kl_list =3D { slh_first =3D 0x0}, kl_lock =3D 0xc04cd13c ,=20 kl_unlock =3D 0xc04cd170 , kl_locked =3D 0xc04cd= 1ac ,=20 kl_lockarg =3D 0xc347f67c}, si_flags =3D 0}, sb_mtx =3D {mtx_object= =3D {lo_class =3D 0xc06ad4c4,=20 lo_name =3D 0xc0681337 "so_snd", lo_type =3D 0xc0681337 "so_snd", l= o_flags =3D 196608,=20 lo_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, lo_witness =3D 0x= 0}, mtx_lock =3D 4,=20 mtx_recurse =3D 0}, sb_state =3D 0, sb_mb =3D 0x0, sb_mbtail =3D 0x0,= sb_lastrecord =3D 0x0,=20 sb_cc =3D 0, sb_hiwat =3D 33304, sb_mbcnt =3D 0, sb_mbmax =3D 262144, s= b_ctl =3D 0, sb_lowat =3D 2048,=20 sb_timeo =3D 0, sb_flags =3D 0}, so_upcall =3D 0, so_upcallarg =3D 0x0,= so_cred =3D 0xc2a7ad00,=20 so_label =3D 0x0, so_peerlabel =3D 0x0, so_gencnt =3D 22107, so_emuldata = =3D 0x0, so_accf =3D 0x0} (kgdb) --VrqPEDrXMn8OVzN4-- --NKoe5XOeduwbEQHU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFFnmoJocfcwTS3JF8RAvRrAJ0cIWI5KunoJMHvGiGdyFp3FfNYAgCgtMY7 FcV0jf4O/FUWBUijhF8d+4U= =sSv7 -----END PGP SIGNATURE----- --NKoe5XOeduwbEQHU--