Date: Tue, 6 May 2014 16:34:34 -0400 From: John Baldwin <jhb@freebsd.org> To: sbruno@freebsd.org Cc: freebsd-stable@freebsd.org Subject: Re: stable/10 panic Message-ID: <201405061634.34868.jhb@freebsd.org> In-Reply-To: <1399315572.77984.2.camel@powernoodle.corp.yahoo.com> References: <1398097892.1101.6.camel@powernoodle.corp.yahoo.com> <201405051348.13320.jhb@freebsd.org> <1399315572.77984.2.camel@powernoodle.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, May 05, 2014 2:46:12 pm Sean Bruno wrote:
> On Mon, 2014-05-05 at 13:48 -0400, John Baldwin wrote:
> > On Monday, April 21, 2014 12:31:32 pm Sean Bruno wrote:
> > > We're seeing this *a lot* on our qmail based hosts during our stable/10
> > > rollout. We're running stable/10 from around svn rev 261579 (mid
> > > february) with one or two patches strewn in.
> > >
> > > sean
> > >
> > >
> > > GNU gdb 6.1.1 [FreeBSD]
> > > Copyright 2004 Free Software Foundation, Inc.
> > > GDB is free software, covered by the GNU General Public License, and you
are
> > > welcome to change it and/or distribute copies of it under certain
conditions.
> > > Type "show copying" to see the conditions.
> > > There is absolutely no warranty for GDB. Type "show warranty" for
details.
> > > This GDB was configured as "amd64-marcel-freebsd"...
> > >
> > > Unread portion of the kernel message buffer:
> > > panic: page fault
> > > cpuid = 5
> > > KDB: stack backtrace:
> > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe048b9b12a0
> > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe048b9b1350
> > > vpanic() at vpanic+0x126/frame 0xfffffe048b9b1390
> > > panic() at panic+0x43/frame 0xfffffe048b9b13f0
> > > trap_fatal() at trap_fatal+0x3a2/frame 0xfffffe048b9b1450
> > > trap_pfault() at trap_pfault+0x22f/frame 0xfffffe048b9b14f0
> > > trap() at trap+0x670/frame 0xfffffe048b9b1710
> > > calltrap() at calltrap+0x8/frame 0xfffffe048b9b1710
> > > --- trap 0xc, rip = 0xffffffff805e759b, rsp = 0xfffffe048b9b17d0, rbp =
0xfffffe048b9b17e0 ---
> > > __mtx_assert() at __mtx_assert+0x3b/frame 0xfffffe048b9b17e0
> > > knote() at knote+0x39/frame 0xfffffe048b9b1830
> > > pipeclose() at pipeclose+0xbe/frame 0xfffffe048b9b1860
> >
> > Can you show which line this is via kgdb? Can you also 'p *cpipe'
> > and 'p *ppipe'?
> >
>
>
> Looks like its in here:
> (kgdb) whe
> #0 doadump (textdump=1) at pcpu.h:219
> #1 0xffffffff805fb2a7 in kern_reboot (howto=260)
> at ../../../kern/kern_shutdown.c:452
> #2 0xffffffff805fb7b5 in vpanic (fmt=<value optimized out>, ap=<value
> optimized out>) at ../../../kern/kern_shutdown.c:759
> #3 0xffffffff805fb803 in panic (fmt=<value optimized out>)
> at ../../../kern/kern_shutdown.c:688
> #4 0xffffffff80977852 in trap_fatal (frame=<value optimized out>,
> eva=<value optimized out>) at ../../../amd64/amd64/trap.c:882
> #5 0xffffffff80977a8f in trap_pfault (frame=0x0, usermode=<value
> optimized out>) at ../../../amd64/amd64/trap.c:699
> #6 0xffffffff809772a0 in trap (frame=0xfffffe048c3df6e0)
> at ../../../amd64/amd64/trap.c:463
> #7 0xffffffff8095c7e2 in calltrap ()
> at ../../../amd64/amd64/exception.S:232
> #8 0xffffffff805e759b in __mtx_assert (c=0x18, what=4,
> file=0xffffffff80ca0bcb "../../../kern/kern_event.c", line=1960)
> at ../../../kern/kern_mutex.c:791
> #9 0xffffffff805c2099 in knote (list=0xfffff8003498aae0, hint=0,
> lockflags=1) at ../../../kern/kern_event.c:1822
> #10 0xffffffff806510fe in pipeclose (cpipe=0xfffff8003498aa18)
> at ../../../kern/sys_pipe.c:1655
> #11 0xffffffff80651019 in pipe_dtor (dpipe=<value optimized out>)
> at ../../../kern/sys_pipe.c:395
> #12 0xffffffff80559924 in fifo_close (ap=<value optimized out>)
> at ../../../fs/fifofs/fifo_vnops.c:115
> #13 0xffffffff80a6ba6a in VOP_CLOSE_APV (vop=<value optimized out>,
> a=<value optimized out>) at vnode_if.c:535
> #14 0xffffffff806acc09 in vn_close (vp=0xfffff8032000cce8, flags=6,
> file_cred=0xfffff8040400d200, td=0xfffff804044f9490) at vnode_if.h:225
> #15 0xffffffff806abad8 in vn_closefile (fp=0xfffff803b2f3c410,
> td=0xfffff804044f9490) at ../../../kern/vfs_vnops.c:1481
> #16 0xffffffff805b8789 in _fdrop (fp=0xfffff803b2f3c410, td=0x4) at
> file.h:342
> #17 0xffffffff805bb0e1 in closef (fp=0xfffff803b2f3c410,
> td=0xfffff804044f9490) at ../../../kern/kern_descrip.c:2415
> #18 0xffffffff805b8bf0 in closefp (fdp=0xfffff800263f5000, fd=<value
> optimized out>, fp=0xfffff803b2f3c410, td=0xfffff804044f9490,
> holdleaders=<value optimized out>) at ../../../kern/kern_descrip.c:1257
> #19 0xffffffff80a332b5 in ia32_syscall (frame=0xfffffe048c3dfbc0) at
> subr_syscall.c:135
> #20 0xffffffff8095cdc5 in Xint0x80_syscall () at ia32_exception.S:73
> #21 0x00000000210f7804 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language: auto; currently minimal
>
>
>
> ----------------
>
> (kgdb) f 10
> #10 0xffffffff806510fe in pipeclose (cpipe=0xfffff8003498aa18)
> at ../../../kern/sys_pipe.c:1655
> 1655 ../../../kern/sys_pipe.c: No such file or directory.
> in ../../../kern/sys_pipe.c
> (kgdb) p *cpipe
> $1 = {pipe_buffer = {cnt = 0, in = 0, out = 0, size = 0, buffer = 0x0},
> pipe_map = {cnt = 0, pos = 0, npages = 0, ms = {0x0 <repeats 17
> times>}}, pipe_sel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0},
> si_note = {kl_list = {
> slh_first = 0x0}, kl_lock = 0, kl_unlock = 0, kl_assert_locked =
> 0xffffffff805c25c0 <knlist_mtx_assert_locked>, kl_assert_unlocked =
> 0xffffffff805c25e0 <knlist_mtx_assert_unlocked>, kl_lockarg = 0x0},
> si_mtx = 0x0},
> pipe_atime = {tv_sec = 1399120887, tv_nsec = 0}, pipe_mtime = {tv_sec
> = 1399120887, tv_nsec = 0}, pipe_ctime = {tv_sec = 1399120887, tv_nsec =
> 0}, pipe_sigio = 0x0, pipe_peer = 0xfffff8003498a8b8, pipe_pair =
> 0xfffff8003498a8b8,
> pipe_state = 2432, pipe_busy = 0, pipe_present = 3, pipe_wgen = 0,
> pipe_ino = 4294967295}
>
> (kgdb) p *ppipe
> $2 = {pipe_buffer = {cnt = 0, in = 0, out = 0, size = 0, buffer = 0x0},
> pipe_map = {cnt = 0, pos = 0, npages = 0, ms = {0x0 <repeats 17
> times>}}, pipe_sel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0},
> si_note = {kl_list = {
> slh_first = 0x0}, kl_lock = 0, kl_unlock = 0, kl_assert_locked =
> 0xffffffff805c25c0 <knlist_mtx_assert_locked>, kl_assert_unlocked =
> 0xffffffff805c25e0 <knlist_mtx_assert_unlocked>, kl_lockarg = 0x0},
> si_mtx = 0x0},
> pipe_atime = {tv_sec = 1399120887, tv_nsec = 0}, pipe_mtime = {tv_sec
> = 1399120887, tv_nsec = 0}, pipe_ctime = {tv_sec = 1399120887, tv_nsec =
> 0}, pipe_sigio = 0x0, pipe_peer = 0xfffff8003498a8b8, pipe_pair =
> 0xfffff8003498a8b8,
> pipe_state = 2432, pipe_busy = 0, pipe_present = 3, pipe_wgen = 0,
> pipe_ino = 4294967295}
So the knlist for both of these has already been destroyed (knlist_destroy
clears kl_lock, but not the assert function pointers). Note that pipe_present
is set to PIPE_FINALIZED for both pipes here. Can you do an 'l' at frame
10 to see exactly which line is being called? Also, it seems like 'cpipe'
and 'pipe' might be the same.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201405061634.34868.jhb>