From owner-freebsd-stable@FreeBSD.ORG Wed Jan 23 16:36:16 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 362F316A419 for ; Wed, 23 Jan 2008 16:36:16 +0000 (UTC) (envelope-from fbsd-ml@scrapper.ca) Received: from pd2mo1so.prod.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 0B25713C478 for ; Wed, 23 Jan 2008 16:36:15 +0000 (UTC) (envelope-from fbsd-ml@scrapper.ca) Received: from pd4mr4so.prod.shaw.ca (pd4mr4so-qfe3.prod.shaw.ca [10.0.141.215]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JV300HTWVF1Q6E0@l-daemon> for freebsd-stable@freebsd.org; Wed, 23 Jan 2008 09:35:25 -0700 (MST) Received: from pn2ml1so.prod.shaw.ca ([10.0.121.145]) by pd4mr4so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JV3005RNVF0R0F0@pd4mr4so.prod.shaw.ca> for freebsd-stable@freebsd.org; Wed, 23 Jan 2008 09:35:25 -0700 (MST) Received: from proven.lan ([24.85.241.34]) by l-daemon (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JV300C37VEXHB80@l-daemon> for freebsd-stable@freebsd.org; Wed, 23 Jan 2008 09:35:23 -0700 (MST) Received: from proven.lan (localhost [127.0.0.1]) by proven.lan (8.14.2/8.14.2) with ESMTP id m0NGZLv9064709 for ; Wed, 23 Jan 2008 08:35:21 -0800 (PST envelope-from fbsd-ml@scrapper.ca) Received: from localhost (localhost [[UNIX: localhost]]) by proven.lan (8.14.2/8.14.2/Submit) id m0NGZLlW064708 for freebsd-stable@freebsd.org; Wed, 23 Jan 2008 08:35:21 -0800 (PST envelope-from fbsd-ml@scrapper.ca) Date: Wed, 23 Jan 2008 08:35:20 -0800 From: Norbert Papke In-reply-to: <200801230928.23047.jhb@freebsd.org> To: freebsd-stable@freebsd.org Message-id: <200801230835.21188.fbsd-ml@scrapper.ca> Organization: Archaeological Filing MIME-version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: 7bit Content-disposition: inline References: <004b01c85c4e$e1c44540$a54ccfc0$@muni.cz> <479513FA.6020802@FreeBSD.org> <200801230928.23047.jhb@freebsd.org> X-Authentication-warning: proven.lan: npapke set sender to fbsd-ml@scrapper.ca using -f User-Agent: KMail/1.9.7 Subject: Re: 6.3-RELEASE panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2008 16:36:16 -0000 On January 23, 2008, John Baldwin wrote: > On Monday 21 January 2008 04:51:54 pm Kris Kennaway wrote: > > Petr Holub wrote: > > > Hi, > > > > > > I've just updated the 6.2-RELEASE to 6.3-RELEASE using freebsd-update > > > as described in daemonology blog. > > > > > > While removing the old packages using > > > pkg_delete -af > > > I've tried to stop all the deamons from /usr/local/etc/rc.d and > > > got the following panic (hand transcribed from a photo - I don't have > > > that machine enabled for remote debugging). Panic seems to be > > > deterministic when stopping those scripts (verified by subsequent > > > attempts while pkg_delete was not running). > > > > > > (kgdb) bt > > > #0 0xc06a46a6 in doadump () > > > #1 0xc06a4b76 in boot () > > > #2 0xc06a4e0c in panic () > > > #3 0xc090d1b4 in trap_fatal () > > > #4 0xc090cf1b in trap_pfault () > > > #5 0xc090cb59 in trap () > > > #6 0xc08f9fea in calltrap () > > > #7 0xc073fa6f in in_delmulti () > > > #8 0xc0748e15 in ip_freemoptions () > > > #9 0xc07414cc in in_pcbdetach () > > > #10 0xc075a0ee in udp_detach () > > > #11 0xc06de0b8 in soclose () > > > #12 0xc06cd83b in soo_close () > > > #13 0xc0683ffc in fdrop_locked () > > > #14 0xc0683f25 in fdrop () > > > #15 0xc0682553 in closef () > > > #16 0xc067f8e7 in kern_close () > > > #17 0xc067f6d8 in close () > > > #18 0xc090d4cb in syscall () > > > #19 0xc08fa03f in Xint0x80_syscall () > > > #20 0x00000033 in ?? () > > > Previous frame inner to this frame (corrupt stack?) > > > > Can you obtain a trace against the kernel.symbols? > > I've been seeing this panic (and several variations of) for quite a while > on 6.x. It appears that a socket is being double-closed somehow. I > usually see it during exit1() when a process' file descriptor table is > being freed. I've spent a lot of time looking for a fd reference count > leak or some such but haven't found one yet. :( I've also seen panics > with vnodes having a ref cnt underflow in vrele or vput, so I've wondered > if it's a fd-level bug that affects both vnodes and sockets rather than > separate socket and vnode bugs. For comparison, here is the callstack from the panic reported in kern/116077. The PR has symbolic information. #0 doadump () #1 0xc055293e in boot #2 0xc0552c95 in panic #3 0xc06e6232 in trap_fatal #4 0xc06e5f3b in trap_pfault #5 0xc06e5b51 in trap #6 0xc06d0b1a in calltrap () #7 0xc05e3e6f in in_delmulti #8 0xc05ed8e9 in ip_freemoptions #9 0xc05e5d6c in in_pcbdetach #10 0xc05fee96 in udp_detach #11 0xc058e710 in soclose #12 0xc057db3f in soo_close #13 0xc052fd9c in fdrop_locked #14 0xc052fcc5 in fdrop #15 0xc052e263 in closef #16 0xc052d253 in fdfree #17 0xc0536b4b in exit1 #18 0xc05366b0 in sys_exit #19 0xc06e6577 in syscall #20 0xc06d0b6f in Xint0x80_syscall () #21 0x00000033 in ?? () It is also triggered when a multi-cast client shuts down. The patch in the PR fixes this issue for me. I have been running with it since November. Cheers.