From owner-freebsd-current@freebsd.org Wed Dec 16 13:28:10 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5BDDAA487F9 for ; Wed, 16 Dec 2015 13:28:10 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.31.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E95C215FA for ; Wed, 16 Dec 2015 13:28:09 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.181.63] (helo=fabiankeil.de) by smtprelay03.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1a9A7g-000847-Df; Wed, 16 Dec 2015 12:19:56 +0100 Date: Wed, 16 Dec 2015 12:21:16 +0100 From: Fabian Keil To: Konstantin Belousov Cc: FreeBSD Current Subject: Re: fork_findpid() - Fatal trap 12: page fault while in kernel mode Message-ID: <20151216122116.09e1b27d@fabiankeil.de> In-Reply-To: <20151216104227.GS3625@kib.kiev.ua> References: <20151215174238.2d7cc3bb@fabiankeil.de> <20151216104227.GS3625@kib.kiev.ua> Reply-To: FreeBSD Current MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/vSk480ceOOzu_o.Qu1u6K0o"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Dec 2015 13:28:10 -0000 --Sig_/vSk480ceOOzu_o.Qu1u6K0o Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Konstantin Belousov wrote: > On Tue, Dec 15, 2015 at 05:42:38PM +0100, Fabian Keil wrote: > > I've seen the following panic a couple of times in the last three > > months, usually while poudriere was running and with sh being the > > current process. > >=20 > > This one is from a system based on r290926 running with > > kern.randompid=3D9001 and forking frequently (>1000 forks/second) > > due to poudriere and afl-fuzz: > >=20 > > Fatal trap 12: page fault while in kernel mode > > cpuid =3D 1; apic id =3D 04 > > fault virtual address =3D 0x618b00a8 > > fault code =3D supervisor read data, page not present > > instruction pointer =3D 0x20:0xffffffff80909158 > > stack pointer =3D 0x28:0xfffffe011e03b940 > > frame pointer =3D 0x28:0xfffffe011e03b960 > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > current process =3D 71325 (sh) > > trap number =3D 12 > > panic: page fault > > cpuid =3D 1 > > KDB: stack backtrace: > > [...] > > Uptime: 13d20h43m20s > > [...] > > (kgdb) where > > #0 doadump (textdump=3D1) at pcpu.h:221 > > #1 0xffffffff8094a923 in kern_reboot (howto=3D260) at /usr/src/sys/ker= n/kern_shutdown.c:364 > > #2 0xffffffff8094ae8b in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:757 > > #3 0xffffffff8094acc3 in panic (fmt=3D0x0) at /usr/src/sys/kern/kern_s= hutdown.c:688 > > #4 0xffffffff80c2fbb1 in trap_fatal (frame=3D, ev= a=3D) at /usr/src/sys/amd64/amd64/trap.c:834 > > #5 0xffffffff80c2fda4 in trap_pfault (frame=3D0xfffffe011e03b890, user= mode=3D) at /usr/src/sys/amd64/amd64/trap.c:684 > > #6 0xffffffff80c2f55e in trap (frame=3D0xfffffe011e03b890) at /usr/src= /sys/amd64/amd64/trap.c:435 > > #7 0xffffffff80c120a7 in calltrap () at /usr/src/sys/amd64/amd64/excep= tion.S:234 > > #8 0xffffffff80909158 in fork_findpid (flags=3D) = at /usr/src/sys/kern/kern_fork.c:281 =20 > It is the values of *p and *(p->p_pgrp) that are needed, from the frame 8. Unfortunately it's not available and apparently I removed the attempts to get it from the previous output. #8 0xffffffff80909158 in fork_findpid (flags=3D) at /= usr/src/sys/kern/kern_fork.c:281 warning: Source file is more recent than executable. 281 (p->p_pgrp !=3D NULL && Current language: auto; currently minimal (kgdb) p p No symbol "p" in current context. (kgdb) p trypid $1 =3D (kgdb) p pidchecked $2 =3D 99999 (kgdb) p lastpid $3 =3D 51281 allproc is available and the first one matches lastpid and has an invalid p_pgrp, but due to trypid being optimized out as well, it's not obvious (to me) that it's the right process. (kgdb) p *allproc->lh_first $4 =3D {p_list =3D {le_next =3D 0xfffff800a99e4548, le_prev =3D 0xffffffff8= 13f3cc8}, p_threads =3D {tqh_first =3D 0xfffff801162819a0, tqh_last =3D 0xf= ffff801162819b0}, p_slock =3D {lock_object =3D { lo_name =3D 0xffffffff80e22449 "process slock", lo_flags =3D 53706752= 0, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 4}, p_ucred =3D 0xfffff= 8009d070000, p_fd =3D 0x0, p_fdtol =3D 0x0, p_stats =3D 0xfffff800299e5800,= =20 p_limit =3D 0x0, p_limco =3D {c_links =3D {le =3D {le_next =3D 0x0, le_pr= ev =3D 0x0}, sle =3D {sle_next =3D 0x0}, tqe =3D {tqe_next =3D 0x0, tqe_pre= v =3D 0x0}}, c_time =3D 0, c_precision =3D 0, c_arg =3D 0x0, c_func =3D 0,= =20 c_lock =3D 0xfffff800304df120, c_flags =3D 0, c_iflags =3D 0, c_cpu =3D= 0}, p_sigacts =3D 0x0, p_flag =3D 268443648, p_flag2 =3D 0, p_state =3D PR= S_NEW, p_pid =3D 51281, p_hash =3D {le_next =3D 0x0,=20 le_prev =3D 0xfffffe0000c8a288}, p_pglist =3D {le_next =3D 0x0, le_prev= =3D 0xfffff800aa94d618}, p_pptr =3D 0xfffff800aa94d548, p_sibling =3D {le_= next =3D 0x0, le_prev =3D 0xfffff800aa94d640}, p_children =3D { lh_first =3D 0x0}, p_reaper =3D 0xfffff800029a5548, p_reaplist =3D {lh_= first =3D 0x0}, p_reapsibling =3D {le_next =3D 0xfffff8007e713548, le_prev = =3D 0xfffff80023df1110}, p_mtx =3D {lock_object =3D { lo_name =3D 0xffffffff80e2243c "process lock", lo_flags =3D 558039040= , lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 18446735280470265856}, p= _statmtx =3D {lock_object =3D {lo_name =3D 0xffffffff80e22457 "pstatl",=20 lo_flags =3D 537067520, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock = =3D 4}, p_itimmtx =3D {lock_object =3D {lo_name =3D 0xffffffff80e2245e "pit= iml", lo_flags =3D 537067520, lo_data =3D 0, lo_witness =3D 0x0},=20 mtx_lock =3D 4}, p_profmtx =3D {lock_object =3D {lo_name =3D 0xffffffff= 80e22465 "pprofl", lo_flags =3D 537067520, lo_data =3D 0, lo_witness =3D 0x= 0}, mtx_lock =3D 4}, p_ksi =3D 0xfffff80126950380, p_sigqueue =3D { sq_signals =3D {__bits =3D 0xfffff800304df1a8}, sq_kill =3D {__bits =3D= 0xfffff800304df1b8}, sq_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff8= 00304df1c8}, sq_proc =3D 0xfffff800304df000, sq_flags =3D 1}, p_oppid =3D 0= ,=20 p_vmspace =3D 0x0, p_swtick =3D 3344683412, p_cowgen =3D 0, p_realtimer = =3D {it_interval =3D {tv_sec =3D 0, tv_usec =3D 0}, it_value =3D {tv_sec = =3D 0, tv_usec =3D 0}}, p_ru =3D {ru_utime =3D {tv_sec =3D 0, tv_usec =3D 0= }, ru_stime =3D { tv_sec =3D 0, tv_usec =3D 0}, ru_maxrss =3D 0, ru_ixrss =3D 0, ru_idr= ss =3D 0, ru_isrss =3D 0, ru_minflt =3D 63, ru_majflt =3D 0, ru_nswap =3D 0= , ru_inblock =3D 1, ru_oublock =3D 1, ru_msgsnd =3D 0, ru_msgrcv =3D 0,=20 ru_nsignals =3D 0, ru_nvcsw =3D 2, ru_nivcsw =3D 3}, p_rux =3D {rux_run= time =3D 1704892, rux_uticks =3D 0, rux_sticks =3D 0, rux_iticks =3D 0, rux= _uu =3D 0, rux_su =3D 0, rux_tu =3D 0}, p_crux =3D {rux_runtime =3D 0,=20 rux_uticks =3D 0, rux_sticks =3D 0, rux_iticks =3D 0, rux_uu =3D 0, rux= _su =3D 0, rux_tu =3D 0}, p_profthreads =3D 0, p_exitthreads =3D 0, p_trace= flag =3D 0, p_tracevp =3D 0x0, p_tracecred =3D 0x0, p_textvp =3D 0x0, p_loc= k =3D 0,=20 p_sigiolst =3D {slh_first =3D 0x0}, p_sigparent =3D 20, p_sig =3D 0, p_co= de =3D 0, p_stops =3D 0, p_stype =3D 0, p_step =3D 0 '\0', p_pfsflags =3D 0= '\0', p_nlminfo =3D 0x0, p_aioinfo =3D 0x0, p_singlethread =3D 0x0,=20 p_suspcount =3D 0, p_xthread =3D 0xfffff801162819a0, p_boundary_count =3D= 0, p_pendingcnt =3D 0, p_itimers =3D 0x0, p_procdesc =3D 0x0, p_treeflag = =3D 0, p_magic =3D 3203398350, p_osrel =3D 1100090,=20 p_comm =3D 0xfffff800304df3c4 "privoxy", p_pgrp =3D 0x618b0080, p_sysent = =3D 0xffffffff8118f9f8, p_args =3D 0x0, p_cpulimit =3D 9223372036854775807,= p_nice =3D 0 '\0', p_fibnum =3D 0, p_reapsubtree =3D 28, p_xexit =3D 0,=20 p_xsig =3D 0, p_klist =3D {kl_list =3D {slh_first =3D 0x0}, kl_lock =3D 0= xffffffff808fc960 , kl_unlock =3D 0xffffffff808fc9c0 ,=20 kl_assert_locked =3D 0xffffffff808fca30 , kl_= assert_unlocked =3D 0xffffffff808fca40 , kl_loc= karg =3D 0xfffff800304df120}, p_numthreads =3D 1, p_md =3D { md_ldt =3D 0x0, md_ldt_sd =3D {sd_lolimit =3D 0, sd_lobase =3D 0, sd_ty= pe =3D 0, sd_dpl =3D 0, sd_p =3D 0, sd_hilimit =3D 0, sd_xx0 =3D 0, sd_gran= =3D 0, sd_hibase =3D 0, sd_xx1 =3D 0, sd_mbz =3D 0, sd_xx2 =3D 0}}, p_itca= llout =3D { c_links =3D {le =3D {le_next =3D 0x0, le_prev =3D 0x0}, sle =3D {sle_ne= xt =3D 0x0}, tqe =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}}, c_time =3D 0, c= _precision =3D 0, c_arg =3D 0x0, c_func =3D 0, c_lock =3D 0xfffff800304df12= 0,=20 c_flags =3D 0, c_iflags =3D 0, c_cpu =3D 0}, p_acflag =3D 1, p_peers = =3D 0x0, p_leader =3D 0xfffff800304df000, p_emuldata =3D 0x0, p_label =3D 0= x0, p_sched =3D 0xfffff800304df548, p_ktr =3D {stqh_first =3D 0x0,=20 stqh_last =3D 0xfffff800304df4d0}, p_mqnotifier =3D {lh_first =3D 0x0},= p_dtrace =3D 0xfffff80087571840, p_pwait =3D {cv_description =3D 0xfffffff= f80e22d2a "ppwait", cv_waiters =3D 0}, p_dbgwait =3D { cv_description =3D 0xffffffff80e22d31 "dbgwait", cv_waiters =3D 0}, p_p= rev_runtime =3D 0, p_racct =3D 0x0, p_throttled =3D 0 '\0', p_vm_dom_policy= =3D {seq =3D 2, p =3D {policy =3D VM_POLICY_NONE, domain =3D -1}},=20 p_orphan =3D {le_next =3D 0x0, le_prev =3D 0x0}, p_orphans =3D {lh_first = =3D 0x0}} (kgdb) p *allproc->lh_first->p_pgrp Cannot access memory at address 0x618b0080 I've changed p's declaration to static so hopefully its value will be available the next time the panic occurs, but it may take a while until that happens. Fabian --Sig_/vSk480ceOOzu_o.Qu1u6K0o Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlZxSS0ACgkQBYqIVf93VJ23fgCeMHGTrR44tucnv8TYIDVjDNJK DLMAn2AzABXMx4DB9NiWgVi9ib7J5l+z =jq7+ -----END PGP SIGNATURE----- --Sig_/vSk480ceOOzu_o.Qu1u6K0o--