From owner-freebsd-current@FreeBSD.ORG Tue Apr 22 15:36:43 2008 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9A601065674 for ; Tue, 22 Apr 2008 15:36:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from anti-4.kiev.sovam.com (anti-4.kiev.sovam.com [62.64.120.202]) by mx1.freebsd.org (Postfix) with ESMTP id 310468FC20 for ; Tue, 22 Apr 2008 15:36:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by anti-4.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JoKXx-000E0t-AI for current@freebsd.org; Tue, 22 Apr 2008 18:36:41 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m3MFaZDc086061 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Apr 2008 18:36:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m3MFaSEl078470; Tue, 22 Apr 2008 18:36:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m3MFaSgB078464; Tue, 22 Apr 2008 18:36:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 22 Apr 2008 18:36:27 +0300 From: Kostik Belousov To: John-Mark Gurney Message-ID: <20080422153627.GS18958@deviant.kiev.zoral.com.ua> References: <20080302152523.GO57756@deviant.kiev.zoral.com.ua> <20080302195334.GI96595@funkthat.com> <20080302200647.GQ57756@deviant.kiev.zoral.com.ua> <20080303003705.GJ96595@funkthat.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KRG9W6ukdHdpcXku" Content-Disposition: inline In-Reply-To: <20080303003705.GJ96595@funkthat.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: 417be1d66bc02b3cfc7f91380aac6f56 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2690 [Apr 22 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: current@freebsd.org Subject: Re: knlist_cleardel() panic X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Apr 2008 15:36:43 -0000 --KRG9W6ukdHdpcXku Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Mar 02, 2008 at 04:37:06PM -0800, John-Mark Gurney wrote: > Kostik Belousov wrote this message on Sun, Mar 02, 2008 at 22:06 +0200: > > On Sun, Mar 02, 2008 at 11:53:34AM -0800, John-Mark Gurney wrote: > > > Kostik Belousov wrote this message on Sun, Mar 02, 2008 at 17:25 +020= 0: > > > > The panic below was already reported, but now I got it on my desktop > > > > and was able to investigate further. > > > >=20 > > > > #5 0xc06daf36 in trap (frame=3D0xe8093b1c) > > > > at /usr/bsd/src/sys/i386/i386/trap.c:490 > > > > #6 0xc06c0b4b in calltrap () at /usr/bsd/src/sys/i386/i386/excepti= on.s:139 > > > > #7 0xc0493968 in knlist_cleardel (knl=3D0xcabec128, td=3D0x0, islo= cked=3D1,=20 > > > > killkn=3D0) at atomic.h:149 > > > > #8 0xc04f520c in pipeclose (cpipe=3D0xcabec0b8) > > > > at /usr/bsd/src/sys/kern/sys_pipe.c:1508 > > > > #9 0xc04f5320 in pipe_close (fp=3D0xc5ce8630, td=3D0xcac01aa0) > > > > at /usr/bsd/src/sys/kern/sys_pipe.c:1425 > > > > #10 0xc0489442 in fdrop (fp=3D0xc5ce8630, td=3D0xcac01aa0) at file.= h:297 > > > > #11 0xc048accf in closef (fp=3D0xc5ce8630, td=3D0xcac01aa0) > > > > at /usr/bsd/src/sys/kern/kern_descrip.c:1958 > > > > #12 0xc048b1ff in kern_close (td=3D0xcac01aa0, fd=3D10) > > > > at /usr/bsd/src/sys/kern/kern_descrip.c:1054 > > > > #13 0xc048b2da in close (td=3D0xcac01aa0, uap=3D0xe8093cfc) > > > > at /usr/bsd/src/sys/kern/kern_descrip.c:1006 > > > > ---Type to continue, or q to quit--- > > > > #14 0xc06da865 in syscall (frame=3D0xe8093d38) > > > > at /usr/bsd/src/sys/i386/i386/trap.c:1035 > > > > #15 0xc06c0bb0 in Xint0x80_syscall () > > > > at /usr/bsd/src/sys/i386/i386/exception.s:196 > > > >=20 > > > > At the frame 8, we have > > > > (kgdb) p/x *(knl->kl_list->slh_first) > > > > $9 =3D {kn_link =3D {sle_next =3D 0x0}, kn_selnext =3D {sle_next = =3D 0x0}, > > > > kn_knlist =3D 0x0, kn_tqe =3D {tqe_next =3D 0xc58de484, tqe_prev = =3D 0xc5e9ab20}, > > > > kn_kq =3D 0x0, kn_kevent =3D {ident =3D 0x0, filter =3D 0x0, flag= s =3D 0x0, > > > > fflags =3D 0x0, data =3D 0x0, udata =3D 0x0}, kn_status =3D 0x2= 0, > > > > kn_sfflags =3D 0x0, kn_sdata =3D 0x0, kn_ptr =3D {p_fp =3D 0x0, p= _proc =3D 0x0, > > > > p_aio =3D 0x0, p_lio =3D 0x0}, kn_fop =3D 0x0, kn_hook =3D 0x0} > > > >=20 > > > > The knote is KN_MARKER, and the kn_kq is NULL. The result is that K= Q_LOCK > > > > in the knlist_cleardel()::SLIST_FOREACH_SAFE() loop dereferences NU= LL and > > > > panics. > > > >=20 > > > > Does the following change makes any sense? > > >=20 > > > I thought this was a bug, but upon further examination, there is > > > something wrong... a KN_MARKER knote should never be on the knlist.. > > > It is only ever added to kq's even list, never to an object's list... > > >=20 > > > If you could walk the knl->kl_list through kn_selnext, and ensure > > > that the kn w/ kN_MARKER exists would be good... It should be > > > last one as kn_selnext is NULL... > >=20 > > As I shown above, the KN_MARKER was found on the cpipe->pipe_sel.si_not= e, > >=20 > > (kgdb) p cpipe->pipe_sel.si_note > > $1 =3D {kl_list =3D {slh_first =3D 0xc58df330},=20 > > kl_lock =3D 0xc0493a20 ,=20 > > kl_unlock =3D 0xc0493370 ,=20 > > kl_locked =3D 0xc0493350 , kl_lockarg =3D 0xcabec1= 70 > > } > >=20 > > (kgdb) p/x (knl->kl_list->slh_first) > > $2 =3D 0xc58df330 > >=20 > > (kgdb) p/x *(knl->kl_list->slh_first) > > $3 =3D {kn_link =3D {sle_next =3D 0x0}, kn_selnext =3D {sle_next =3D 0x= 0},=20 > > kn_knlist =3D 0x0, kn_tqe =3D {tqe_next =3D 0xc58de484, tqe_prev =3D = 0xc5e9ab20},=20 > > kn_kq =3D 0x0, kn_kevent =3D {ident =3D 0x0, filter =3D 0x0, flags = =3D 0x0,=20 > > fflags =3D 0x0, data =3D 0x0, udata =3D 0x0}, kn_status =3D 0x20,= =20 > > kn_sfflags =3D 0x0, kn_sdata =3D 0x0, kn_ptr =3D {p_fp =3D 0x0, p_pro= c =3D 0x0,=20 > > p_aio =3D 0x0, p_lio =3D 0x0}, kn_fop =3D 0x0, kn_hook =3D 0x0} > > >=20 > > > I can't think of a way that this can happen, as the only way to get > > > on the knlist is by calling knlist_add, and it should only ever be > > > called from f_attach, which is called in one place, and has to have > > > kn_kq set properly... Are there any custom patches on the system? > >=20 > > This is the stock RELENG_7 as of today. >=20 > Do you have a reproducable test case that demonstrates this? >=20 > You can try the attached patch, but I'm pretty sure that it won't be > triggered... >=20 > --=20 > John-Mark Gurney Voice: +1 415 225 5579 >=20 > "All that I will do, has been done, All that I have, has not." > --- kern_event.c.orig 2007-07-14 14:23:30.000000000 -0700 > +++ kern_event.c 2008-03-02 16:26:59.000000000 -0800 > @@ -1590,6 +1590,8 @@ > KQ_NOTOWNED(kn->kn_kq); > KASSERT((kn->kn_status & (KN_INFLUX|KN_DETACHED)) =3D=3D > (KN_INFLUX|KN_DETACHED), ("knote not KN_INFLUX and KN_DETACHED")); > + KASSERT((kn->kn_status & (KN_MARKER)) =3D=3D 0, > + ("knote has KN_MARKER set")); > if (!islocked) > knl->kl_lock(knl->kl_lockarg); > SLIST_INSERT_HEAD(&knl->kl_list, kn, kn_selnext); When the pipe_peer is being closed, and the pipeclose() has set the pipe_present =3D 0 and enters knlist_cleardel(), the PIPE_MTX may be dropped when knl->kl_list cannot be cleared due to influx knotes. If the following often encountered code fragment if (!(kn->kn_status & KN_DETACHED)) kn->kn_fop->f_detach(kn); knote_drop(kn, td); [1] is executed while the knlist lock is dropped, then the knote memory is freed by the knote_drop() [1] without knote being removed from the knlist, since the filt_pipedetach() contains the following: if (kn->kn_filter =3D=3D EVFILT_WRITE) { if (!cpipe->pipe_peer->pipe_present) { PIPE_UNLOCK(cpipe); return; Now, the memory may be reused in the zone, for instance, for the marker knote. And, we ending up the marker on the knlist. Does this look plausible ? Thanks. --KRG9W6ukdHdpcXku Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkgOBXUACgkQC3+MBN1Mb4gUnACguIt9+YTd4sDynH4vFHyNGZiF v7gAn0ctqZ1OYLH1fE4H31Qjpflh8wqT =sZTX -----END PGP SIGNATURE----- --KRG9W6ukdHdpcXku--