From owner-svn-src-head@FreeBSD.ORG Fri Jun 26 05:57:47 2009 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15635106566C; Fri, 26 Jun 2009 05:57:47 +0000 (UTC) (envelope-from mat.macy@gmail.com) Received: from mail-ew0-f213.google.com (mail-ew0-f213.google.com [209.85.219.213]) by mx1.freebsd.org (Postfix) with ESMTP id 180618FC20; Fri, 26 Jun 2009 05:57:45 +0000 (UTC) (envelope-from mat.macy@gmail.com) Received: by ewy9 with SMTP id 9so1534844ewy.43 for ; Thu, 25 Jun 2009 22:57:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=rrkr8bxS7vzCVYgtxiCyeuIWh7uX5rKPijnV9iedJKY=; b=R8VY1/gfzQ7FKiHReBQDVFVw/NRCKjuWOX2H+hbVwnbg634/7ED9upRFSpfss85swP tSn5Hr6AtLeAXWZ0HL9syAmy3lwP8UpFjWc3pGux5h03aTbfXpfwCcXlJNeWYZk3OBBK zxAmUjawDwvGCS8Qg2wY8vTgkrfo0kGhGkOuA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=mVBET529Ws1kbAmQleEsRAqIQaiL9NA1lA6txAgDlwongV79v0SUWUriXCcGsTTPDz msPCo4+8FGytc1G6sKDKEFtn/cc1+Zo5e78jA2oZzIRTmUVSGkuIYRvYTHlCDbF1FT2w h5ykZYhZI0tjoJ+UPlBgtvzzXkucwAbmr4Zdw= MIME-Version: 1.0 Sender: mat.macy@gmail.com Received: by 10.216.11.200 with SMTP id 50mr1001384wex.183.1245995865099; Thu, 25 Jun 2009 22:57:45 -0700 (PDT) In-Reply-To: <4A42D629.8060806@cs.duke.edu> References: <200906242109.n5OL9uVb029380@svn.freebsd.org> <4A42983C.6050307@freebsd.org> <4A42D629.8060806@cs.duke.edu> Date: Thu, 25 Jun 2009 22:57:45 -0700 X-Google-Sender-Auth: 811bb01f70decaa4 Message-ID: <3c1674c90906252257q7fe4c725m3840b2bb3c8aeb51@mail.gmail.com> From: Kip Macy To: Andrew Gallatin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, Sam Leffler , src-committers@freebsd.org Subject: Re: svn commit: r194909 - head/sys/dev/mxge X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jun 2009 05:57:47 -0000 This is my bug. Hold off for a day or two. -Kip On Wed, Jun 24, 2009 at 6:43 PM, Andrew Gallatin wrot= e: > Sam Leffler wrote: > >> There's something else wrong. =A0This is just covering up the real bug. > > I'm pretty sure the "real bug" is in bpf, but I'm not sure its a bug, > and I suspect there are probably other, similar, bugs lurking when > you try to tear down a busy interface. > > What I was doing was: > > - point a packet generator offering 1.5Mpps at the NIC > > - in a tight shell loop, do > > while (1) > =A0 =A0 =A0 =A0tcpdump -ni mxge0 host 172.31.0.1 > end > > - in another shell loop: > > while (1) > =A0 =A0 =A0 =A0ifconfig mxge0 192.168.1.22 up > =A0 =A0 =A0 =A0sleep 1 > =A0 =A0 =A0 =A0kldunload if_mxge > end > > Before the commit, with the old order: > > =A0 =A0 =A0 lock() > =A0 =A0 =A0 close() > =A0 =A0 =A0 unlock() > =A0 =A0 =A0 ether_ifdetach() > > I'd see either an exhaustion of mbufs because tcpdump snuck in after > I'd closed the device and re-opened it on me (so I never closed it > again, resulting in leaked mbufs), or a panic. > > I then moved the ether_ifdetach() to the new position: > =A0 =A0 =A0 ether_ifdetach() > =A0 =A0 =A0 lock() > =A0 =A0 =A0 close() > =A0 =A0 =A0 unlock() > > This worked great until I started the packet generator, > then it crashed. =A0 The stack I saw (which I don't have > saved, so this is from memory) when I had ether_ifdetach() > first was: > > > panic: mtx_lock() (don't remember exact text) > bpf_mtap() > ether_input() > mxge_rx_done_small() > mxge_clean_rx_done() > mxge_intr() > <...> > > When I looked at the ifp in kgdb, I noticed that all the operations > (if_input(), if_output(), etc) pointed to ifdead_* > The machine I'm using for this is a MacPro, and I can't get ddb > to work on the USB based console, so I'm working purely from dumps. > I don't know how to get a stack of another process in kgdb on > amd64, so that's all the information I have. > > My assumption is that my interrupt thread was running when > ether_ifdetach() called bpfdetach(), and was starting bpf_mtap() > while bpfdetach() was destroying the bpf_if. =A0There doesn't > seem to be anything to prevent bpfdetach() from racing with > bpf_mtap(). > > By calling my close() routine (with a dying flag so nothing can > sneak in before detach), I'm assured that my NIC is quiescent, > and cannot be calling into the stack while the interface is being > torn down. =A0I'd prefer to leave my commit as-is because: > > 1) it works, and fixes a bug > 2) it can be MFC'ed as is > 3) it just feels wrong to be blasting packets up into the stack > =A0 while detaching. =A0With this NIC, the best way to make it > =A0 quiescent is to call close(). =A0There's an interrupt handshake > =A0 done with the NIC to ensure its is quiescent, so doing something > =A0 like disabling its interrupt could leave the things in a weird state. > > Drew > --=20 When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke