From owner-freebsd-stable@FreeBSD.ORG  Mon Jan 12 15:16:50 2009
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 45DD51065674;
	Mon, 12 Jan 2009 15:16:50 +0000 (UTC)
	(envelope-from petefrench@ticketswitch.com)
Received: from constantine.ticketswitch.com (constantine.ticketswitch.com
	[IPv6:2002:57e0:1d4e:1::3])
	by mx1.freebsd.org (Postfix) with ESMTP id 07D588FC13;
	Mon, 12 Jan 2009 15:16:49 +0000 (UTC)
	(envelope-from petefrench@ticketswitch.com)
Received: from dilbert.rattatosk ([10.64.50.6] helo=dilbert.ticketswitch.com)
	by constantine.ticketswitch.com with esmtps (TLSv1:AES256-SHA:256)
	(Exim 4.69 (FreeBSD)) (envelope-from <petefrench@ticketswitch.com>)
	id 1LMOWu-000Cm3-Cj; Mon, 12 Jan 2009 15:16:40 +0000
Received: from petefrench by dilbert.ticketswitch.com with local (Exim 4.69
	(FreeBSD)) (envelope-from <petefrench@ticketswitch.com>)
	id 1LMOWu-0001t4-BH; Mon, 12 Jan 2009 15:16:40 +0000
To: rwatson@FreeBSD.org
In-Reply-To: <alpine.BSF.2.00.0901121458140.16794@fledge.watson.org>
Message-Id: <E1LMOWu-0001t4-BH@dilbert.ticketswitch.com>
From: Pete French <petefrench@ticketswitch.com>
Date: Mon, 12 Jan 2009 15:16:40 +0000
Cc: freebsd-stable@freebsd.org, drosih@rpi.edu, rblayzor.bulk@inoc.net
Subject: Re: Big problems with 7.1 locking up :-(
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 12 Jan 2009 15:16:51 -0000

> I'm not sure if you've done this already, but the normal suggestions apply: 
> have you compiled with INVARIANTS/WITNESS/DDB/KDB/BREAK_TO_DEBUGGER, and do 
> any results / panics / etc result?  Sometimes these debugging tools are able 
> to convert hangs into panics, which gives us much more ability to debug them. 

I did, but it turns out I had an incorrect option in there which made the
data I got not relevent. I now have another machine running a kernel
with the following config:

include         GENERIC
ident           DEBUG

options         KDB
options         DDB
options         SW_WATCHDOG
options         DEBUG_VFS_LOCKS
options         MUTEX_DEBUG
options         WITNESS
options         LOCK_PROFILING
options         INVARIANTS
options         INVARIANT_SUPPORT
options         DIAGNOSTIC

Those should enable me to get some useful output I hope.

> If it still hangs rather than panicking, are you able to break into the 
> debugger on the console?  If you're using a video console and not able to get 
> to the debugger, would it be possible to configure a serial console and use 

I cant add a sserial console - I am remote enough from most of
these machines (Slough) and very remote from the test box (its in the USA!)
so I cant get to them physicly. But I do have iLo which lets me use the
console and gives me a bit of access to the front. I will check for NMI.

Just had another lockup here - my working day has become a succession of
running round rebooting servers though iLo at the moment.

Will get back to you when the debug one has crashed - I could possibly
give you direct access to the iLo console on that if you need it ?

-pete.