Date: Wed, 16 Dec 2015 12:21:16 +0100 From: Fabian Keil <freebsd-listen@fabiankeil.de> To: Konstantin Belousov <kostikbel@gmail.com> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: fork_findpid() - Fatal trap 12: page fault while in kernel mode Message-ID: <20151216122116.09e1b27d@fabiankeil.de> In-Reply-To: <20151216104227.GS3625@kib.kiev.ua> References: <20151215174238.2d7cc3bb@fabiankeil.de> <20151216104227.GS3625@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_/vSk480ceOOzu_o.Qu1u6K0o
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
Konstantin Belousov <kostikbel@gmail.com> wrote:
> On Tue, Dec 15, 2015 at 05:42:38PM +0100, Fabian Keil wrote:
> > I've seen the following panic a couple of times in the last three
> > months, usually while poudriere was running and with sh being the
> > current process.
> >=20
> > This one is from a system based on r290926 running with
> > kern.randompid=3D9001 and forking frequently (>1000 forks/second)
> > due to poudriere and afl-fuzz:
> >=20
> > Fatal trap 12: page fault while in kernel mode
> > cpuid =3D 1; apic id =3D 04
> > fault virtual address =3D 0x618b00a8
> > fault code =3D supervisor read data, page not present
> > instruction pointer =3D 0x20:0xffffffff80909158
> > stack pointer =3D 0x28:0xfffffe011e03b940
> > frame pointer =3D 0x28:0xfffffe011e03b960
> > code segment =3D base 0x0, limit 0xfffff, type 0x1b
> > =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags =3D interrupt enabled, resume, IOPL =3D 0
> > current process =3D 71325 (sh)
> > trap number =3D 12
> > panic: page fault
> > cpuid =3D 1
> > KDB: stack backtrace:
> > [...]
> > Uptime: 13d20h43m20s
> > [...]
> > (kgdb) where
> > #0 doadump (textdump=3D1) at pcpu.h:221
> > #1 0xffffffff8094a923 in kern_reboot (howto=3D260) at /usr/src/sys/ker=
n/kern_shutdown.c:364
> > #2 0xffffffff8094ae8b in vpanic (fmt=3D<value optimized out>, ap=3D<va=
lue optimized out>) at /usr/src/sys/kern/kern_shutdown.c:757
> > #3 0xffffffff8094acc3 in panic (fmt=3D0x0) at /usr/src/sys/kern/kern_s=
hutdown.c:688
> > #4 0xffffffff80c2fbb1 in trap_fatal (frame=3D<value optimized out>, ev=
a=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:834
> > #5 0xffffffff80c2fda4 in trap_pfault (frame=3D0xfffffe011e03b890, user=
mode=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:684
> > #6 0xffffffff80c2f55e in trap (frame=3D0xfffffe011e03b890) at /usr/src=
/sys/amd64/amd64/trap.c:435
> > #7 0xffffffff80c120a7 in calltrap () at /usr/src/sys/amd64/amd64/excep=
tion.S:234
> > #8 0xffffffff80909158 in fork_findpid (flags=3D<value optimized out>) =
at /usr/src/sys/kern/kern_fork.c:281 =20
> It is the values of *p and *(p->p_pgrp) that are needed, from the frame 8.
Unfortunately it's not available and apparently I removed the attempts
to get it from the previous output.
#8 0xffffffff80909158 in fork_findpid (flags=3D<value optimized out>) at /=
usr/src/sys/kern/kern_fork.c:281
warning: Source file is more recent than executable.
281 (p->p_pgrp !=3D NULL &&
Current language: auto; currently minimal
(kgdb) p p
No symbol "p" in current context.
(kgdb) p trypid
$1 =3D <value optimized out>
(kgdb) p pidchecked
$2 =3D 99999
(kgdb) p lastpid
$3 =3D 51281
allproc is available and the first one matches lastpid and has an invalid
p_pgrp, but due to trypid being optimized out as well, it's not obvious
(to me) that it's the right process.
(kgdb) p *allproc->lh_first
$4 =3D {p_list =3D {le_next =3D 0xfffff800a99e4548, le_prev =3D 0xffffffff8=
13f3cc8}, p_threads =3D {tqh_first =3D 0xfffff801162819a0, tqh_last =3D 0xf=
ffff801162819b0}, p_slock =3D {lock_object =3D {
lo_name =3D 0xffffffff80e22449 "process slock", lo_flags =3D 53706752=
0, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 4}, p_ucred =3D 0xfffff=
8009d070000, p_fd =3D 0x0, p_fdtol =3D 0x0, p_stats =3D 0xfffff800299e5800,=
=20
p_limit =3D 0x0, p_limco =3D {c_links =3D {le =3D {le_next =3D 0x0, le_pr=
ev =3D 0x0}, sle =3D {sle_next =3D 0x0}, tqe =3D {tqe_next =3D 0x0, tqe_pre=
v =3D 0x0}}, c_time =3D 0, c_precision =3D 0, c_arg =3D 0x0, c_func =3D 0,=
=20
c_lock =3D 0xfffff800304df120, c_flags =3D 0, c_iflags =3D 0, c_cpu =3D=
0}, p_sigacts =3D 0x0, p_flag =3D 268443648, p_flag2 =3D 0, p_state =3D PR=
S_NEW, p_pid =3D 51281, p_hash =3D {le_next =3D 0x0,=20
le_prev =3D 0xfffffe0000c8a288}, p_pglist =3D {le_next =3D 0x0, le_prev=
=3D 0xfffff800aa94d618}, p_pptr =3D 0xfffff800aa94d548, p_sibling =3D {le_=
next =3D 0x0, le_prev =3D 0xfffff800aa94d640}, p_children =3D {
lh_first =3D 0x0}, p_reaper =3D 0xfffff800029a5548, p_reaplist =3D {lh_=
first =3D 0x0}, p_reapsibling =3D {le_next =3D 0xfffff8007e713548, le_prev =
=3D 0xfffff80023df1110}, p_mtx =3D {lock_object =3D {
lo_name =3D 0xffffffff80e2243c "process lock", lo_flags =3D 558039040=
, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 18446735280470265856}, p=
_statmtx =3D {lock_object =3D {lo_name =3D 0xffffffff80e22457 "pstatl",=20
lo_flags =3D 537067520, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =
=3D 4}, p_itimmtx =3D {lock_object =3D {lo_name =3D 0xffffffff80e2245e "pit=
iml", lo_flags =3D 537067520, lo_data =3D 0, lo_witness =3D 0x0},=20
mtx_lock =3D 4}, p_profmtx =3D {lock_object =3D {lo_name =3D 0xffffffff=
80e22465 "pprofl", lo_flags =3D 537067520, lo_data =3D 0, lo_witness =3D 0x=
0}, mtx_lock =3D 4}, p_ksi =3D 0xfffff80126950380, p_sigqueue =3D {
sq_signals =3D {__bits =3D 0xfffff800304df1a8}, sq_kill =3D {__bits =3D=
0xfffff800304df1b8}, sq_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff8=
00304df1c8}, sq_proc =3D 0xfffff800304df000, sq_flags =3D 1}, p_oppid =3D 0=
,=20
p_vmspace =3D 0x0, p_swtick =3D 3344683412, p_cowgen =3D 0, p_realtimer =
=3D {it_interval =3D {tv_sec =3D 0, tv_usec =3D 0}, it_value =3D {tv_sec =
=3D 0, tv_usec =3D 0}}, p_ru =3D {ru_utime =3D {tv_sec =3D 0, tv_usec =3D 0=
}, ru_stime =3D {
tv_sec =3D 0, tv_usec =3D 0}, ru_maxrss =3D 0, ru_ixrss =3D 0, ru_idr=
ss =3D 0, ru_isrss =3D 0, ru_minflt =3D 63, ru_majflt =3D 0, ru_nswap =3D 0=
, ru_inblock =3D 1, ru_oublock =3D 1, ru_msgsnd =3D 0, ru_msgrcv =3D 0,=20
ru_nsignals =3D 0, ru_nvcsw =3D 2, ru_nivcsw =3D 3}, p_rux =3D {rux_run=
time =3D 1704892, rux_uticks =3D 0, rux_sticks =3D 0, rux_iticks =3D 0, rux=
_uu =3D 0, rux_su =3D 0, rux_tu =3D 0}, p_crux =3D {rux_runtime =3D 0,=20
rux_uticks =3D 0, rux_sticks =3D 0, rux_iticks =3D 0, rux_uu =3D 0, rux=
_su =3D 0, rux_tu =3D 0}, p_profthreads =3D 0, p_exitthreads =3D 0, p_trace=
flag =3D 0, p_tracevp =3D 0x0, p_tracecred =3D 0x0, p_textvp =3D 0x0, p_loc=
k =3D 0,=20
p_sigiolst =3D {slh_first =3D 0x0}, p_sigparent =3D 20, p_sig =3D 0, p_co=
de =3D 0, p_stops =3D 0, p_stype =3D 0, p_step =3D 0 '\0', p_pfsflags =3D 0=
'\0', p_nlminfo =3D 0x0, p_aioinfo =3D 0x0, p_singlethread =3D 0x0,=20
p_suspcount =3D 0, p_xthread =3D 0xfffff801162819a0, p_boundary_count =3D=
0, p_pendingcnt =3D 0, p_itimers =3D 0x0, p_procdesc =3D 0x0, p_treeflag =
=3D 0, p_magic =3D 3203398350, p_osrel =3D 1100090,=20
p_comm =3D 0xfffff800304df3c4 "privoxy", p_pgrp =3D 0x618b0080, p_sysent =
=3D 0xffffffff8118f9f8, p_args =3D 0x0, p_cpulimit =3D 9223372036854775807,=
p_nice =3D 0 '\0', p_fibnum =3D 0, p_reapsubtree =3D 28, p_xexit =3D 0,=20
p_xsig =3D 0, p_klist =3D {kl_list =3D {slh_first =3D 0x0}, kl_lock =3D 0=
xffffffff808fc960 <knlist_mtx_lock>, kl_unlock =3D 0xffffffff808fc9c0 <knli=
st_mtx_unlock>,=20
kl_assert_locked =3D 0xffffffff808fca30 <knlist_mtx_assert_locked>, kl_=
assert_unlocked =3D 0xffffffff808fca40 <knlist_mtx_assert_unlocked>, kl_loc=
karg =3D 0xfffff800304df120}, p_numthreads =3D 1, p_md =3D {
md_ldt =3D 0x0, md_ldt_sd =3D {sd_lolimit =3D 0, sd_lobase =3D 0, sd_ty=
pe =3D 0, sd_dpl =3D 0, sd_p =3D 0, sd_hilimit =3D 0, sd_xx0 =3D 0, sd_gran=
=3D 0, sd_hibase =3D 0, sd_xx1 =3D 0, sd_mbz =3D 0, sd_xx2 =3D 0}}, p_itca=
llout =3D {
c_links =3D {le =3D {le_next =3D 0x0, le_prev =3D 0x0}, sle =3D {sle_ne=
xt =3D 0x0}, tqe =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}}, c_time =3D 0, c=
_precision =3D 0, c_arg =3D 0x0, c_func =3D 0, c_lock =3D 0xfffff800304df12=
0,=20
c_flags =3D 0, c_iflags =3D 0, c_cpu =3D 0}, p_acflag =3D 1, p_peers =
=3D 0x0, p_leader =3D 0xfffff800304df000, p_emuldata =3D 0x0, p_label =3D 0=
x0, p_sched =3D 0xfffff800304df548, p_ktr =3D {stqh_first =3D 0x0,=20
stqh_last =3D 0xfffff800304df4d0}, p_mqnotifier =3D {lh_first =3D 0x0},=
p_dtrace =3D 0xfffff80087571840, p_pwait =3D {cv_description =3D 0xfffffff=
f80e22d2a "ppwait", cv_waiters =3D 0}, p_dbgwait =3D {
cv_description =3D 0xffffffff80e22d31 "dbgwait", cv_waiters =3D 0}, p_p=
rev_runtime =3D 0, p_racct =3D 0x0, p_throttled =3D 0 '\0', p_vm_dom_policy=
=3D {seq =3D 2, p =3D {policy =3D VM_POLICY_NONE, domain =3D -1}},=20
p_orphan =3D {le_next =3D 0x0, le_prev =3D 0x0}, p_orphans =3D {lh_first =
=3D 0x0}}
(kgdb) p *allproc->lh_first->p_pgrp
Cannot access memory at address 0x618b0080
I've changed p's declaration to static so hopefully its value will
be available the next time the panic occurs, but it may take a while
until that happens.
Fabian
--Sig_/vSk480ceOOzu_o.Qu1u6K0o
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlZxSS0ACgkQBYqIVf93VJ23fgCeMHGTrR44tucnv8TYIDVjDNJK
DLMAn2AzABXMx4DB9NiWgVi9ib7J5l+z
=jq7+
-----END PGP SIGNATURE-----
--Sig_/vSk480ceOOzu_o.Qu1u6K0o--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151216122116.09e1b27d>