From owner-freebsd-stable@FreeBSD.ORG Fri Sep 22 20:36:56 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A854B16A407 for ; Fri, 22 Sep 2006 20:36:56 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1620243D6D for ; Fri, 22 Sep 2006 20:36:56 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id F0EA21A3C1C; Fri, 22 Sep 2006 13:36:55 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E5AF651379; Fri, 22 Sep 2006 16:36:54 -0400 (EDT) Date: Fri, 22 Sep 2006 16:36:54 -0400 From: Kris Kennaway To: Vivek Khera Message-ID: <20060922203654.GA65693@xor.obsecurity.org> References: <917B087C-5E13-4D7F-94FA-95CB0E5C1884@khera.org> <20060922190328.GA64849@xor.obsecurity.org> <555B84D2-520F-44D6-84D6-CF9CE7EE47C7@khera.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZPt4rx8FFjLCG7dd" Content-Disposition: inline In-Reply-To: <555B84D2-520F-44D6-84D6-CF9CE7EE47C7@khera.org> User-Agent: Mutt/1.4.2.2i Cc: stable@freebsd.org Subject: Re: ffs snapshot lockup X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Sep 2006 20:36:56 -0000 --ZPt4rx8FFjLCG7dd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 22, 2006 at 04:24:41PM -0400, Vivek Khera wrote: >=20 > On Sep 22, 2006, at 3:03 PM, Kris Kennaway wrote: >=20 > >>I know this is very un-useful as a bug report, but I'm putting this > >>out in case anyone else has noticed this. It has never happened to > >>us before when we were running 5.4 and then 6.1 on this box. Nothing > >>else changed on the system during this time. > > > >Yep, we'd need additional debugging to proceed. >=20 > How does one debug a system where the disk sub-system is frozen? I =20 > have serial console with DDB/KDB built into the kernel and debugging =20 > symbols too. Is that the only option here? Perhaps, but you can probably also take a dump still. It's a filesystem deadlock, not a disk device problem. > We're gonna try the full level 0 backup again monday and if it locks =20 > up I'll try to figure it out, but could use some hints as to where to =20 > poke in the kernel. Start by enabling INVARIANTS, INVARIANT_SUPPORT, DEBUG_LOCKS and DEBUG_VFS_LOCKS, then run 'show lockedvnods' and 'alltrace' in DDB (spammy, need that serial console), or at least trace the running processes (show allpcpu) and those listed in lockedvnods. Then call doadump and save the core+kernel.debug when you reboot. Kris --ZPt4rx8FFjLCG7dd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFFFElmWry0BWjoQKURAmtXAKDfFNOcZNyyRkIzOgg1eG6VuO6QCgCg0qbv FM9A/9+XsdnqwaVOm+pDtQA= =aDub -----END PGP SIGNATURE----- --ZPt4rx8FFjLCG7dd--