From owner-freebsd-stable@FreeBSD.ORG  Wed Jan 23 14:49:45 2008
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id F09FE16A417;
	Wed, 23 Jan 2008 14:49:45 +0000 (UTC) (envelope-from jhb@freebsd.org)
Received: from elvis.mu.org (elvis.mu.org [192.203.228.196])
	by mx1.freebsd.org (Postfix) with ESMTP id D0AAF13C448;
	Wed, 23 Jan 2008 14:49:45 +0000 (UTC) (envelope-from jhb@freebsd.org)
Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net
	[66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id 079301A4D8B;
	Wed, 23 Jan 2008 06:45:53 -0800 (PST)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-stable@freebsd.org
Date: Wed, 23 Jan 2008 09:28:22 -0500
User-Agent: KMail/1.9.7
References: <004b01c85c4e$e1c44540$a54ccfc0$@muni.cz>
	<479513FA.6020802@FreeBSD.org>
In-Reply-To: <479513FA.6020802@FreeBSD.org>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200801230928.23047.jhb@freebsd.org>
Cc: Petr Holub <hopet@ics.muni.cz>, Kris Kennaway <kris@freebsd.org>,
	rwatson@freebsd.org, stable@freebsd.org
Subject: Re: 6.3-RELEASE panic
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 23 Jan 2008 14:49:46 -0000

On Monday 21 January 2008 04:51:54 pm Kris Kennaway wrote:
> Petr Holub wrote:
> > Hi,
> > 
> > I've just updated the 6.2-RELEASE to 6.3-RELEASE using freebsd-update
> > as described in daemonology blog.
> > 
> > While removing the old packages using
> > pkg_delete -af
> > I've tried to stop all the deamons from /usr/local/etc/rc.d and
> > got the following panic (hand transcribed from a photo - I don't have that
> > machine enabled for remote debugging). Panic seems to be deterministic
> > when stopping those scripts (verified by subsequent attempts while
> > pkg_delete was not running).
> 
> > (kgdb) bt
> > #0  0xc06a46a6 in doadump ()
> > #1  0xc06a4b76 in boot ()
> > #2  0xc06a4e0c in panic ()
> > #3  0xc090d1b4 in trap_fatal ()
> > #4  0xc090cf1b in trap_pfault ()
> > #5  0xc090cb59 in trap ()
> > #6  0xc08f9fea in calltrap ()
> > #7  0xc073fa6f in in_delmulti ()
> > #8  0xc0748e15 in ip_freemoptions ()
> > #9  0xc07414cc in in_pcbdetach ()
> > #10 0xc075a0ee in udp_detach ()
> > #11 0xc06de0b8 in soclose ()
> > #12 0xc06cd83b in soo_close ()
> > #13 0xc0683ffc in fdrop_locked ()
> > #14 0xc0683f25 in fdrop ()
> > #15 0xc0682553 in closef ()
> > #16 0xc067f8e7 in kern_close ()
> > #17 0xc067f6d8 in close ()
> > #18 0xc090d4cb in syscall ()
> > #19 0xc08fa03f in Xint0x80_syscall ()
> > #20 0x00000033 in ?? ()
> > Previous frame inner to this frame (corrupt stack?)
> 
> Can you obtain a trace against the kernel.symbols?

I've been seeing this panic (and several variations of) for quite a while
on 6.x.  It appears that a socket is being double-closed somehow.  I
usually see it during exit1() when a process' file descriptor table is
being freed.  I've spent a lot of time looking for a fd reference count
leak or some such but haven't found one yet. :(  I've also seen panics
with vnodes having a ref cnt underflow in vrele or vput, so I've wondered
if it's a fd-level bug that affects both vnodes and sockets rather than
separate socket and vnode bugs.

-- 
John Baldwin