From owner-freebsd-questions@FreeBSD.ORG  Thu May 31 15:11:24 2012
Return-Path: <owner-freebsd-questions@FreeBSD.ORG>
Delivered-To: freebsd-questions@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id C30CE106564A;
	Thu, 31 May 2012 15:11:24 +0000 (UTC) (envelope-from feld@feld.me)
Received: from feld.me (unknown [IPv6:2607:f4e0:100:300::2])
	by mx1.freebsd.org (Postfix) with ESMTP id 830AF8FC18;
	Thu, 31 May 2012 15:11:24 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me;
	s=blargle; 
	h=In-Reply-To:Message-Id:From:Mime-Version:Date:References:Subject:Cc:To:Content-Type;
	bh=booLkkYv/+ARJZmxyxTU6ygy89uLJr3ito2uITNWjsM=; 
	b=KIF6PAjOgu3AKTKgZrUOW2jV+3ZXF/hvgaI4/V/SKXOBggbw404nMNV6wNgwn+ScU+b5KHYTo2gHzf9gNkjoXuyYDG1ZpjZsD8pQHl3uuE7HLO3xqspaiMJ+bVfp0v0x;
Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org)
	by feld.me with esmtp (Exim 4.77 (FreeBSD))
	(envelope-from <feld@feld.me>)
	id 1Sa71x-000OnE-Fg; Thu, 31 May 2012 10:11:24 -0500
Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4)
	with esmtpa id 1338477071-26372-26371/5/12; Thu, 31 May 2012 15:11:11
	+0000
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
To: freebsd-hackers@freebsd.org, freebsd-questions@FreeBSD.org
References: <op.wbwe9s0k34t2sn@tech304> <201205301317.07345.jhb@freebsd.org>
	<op.we40bn0z34t2sn@tech304> <201205311048.45813.jhb@freebsd.org>
Date: Thu, 31 May 2012 10:11:11 -0500
Mime-Version: 1.0
From: Mark Felder <feld@feld.me>
Message-Id: <op.we6hsx0m34t2sn@tech304>
In-Reply-To: <201205311048.45813.jhb@freebsd.org>
User-Agent: Opera Mail/11.64 (FreeBSD)
X-SA-Score: -1.5
Cc: John Baldwin <jhb@freebsd.org>
Subject: Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 31 May 2012 15:11:24 -0000

So when this hang happens, there never is a real panic. It just sits in a  
state which I describe as like being in a deadlock. How would I go about  
getting a crashdump if it never panics? Is it possible to do the dump over  
a network or something because I don't believe it can write through the  
controller at all.

Also, thank you for the KTR_SCHED tip. This is the type of info I was  
looking for. Unfortunately I've only ever seen this crash once on a kernel  
with debugging enabled. The machine which is currently prepared to do this  
work used to crash a few times a week and now it has 70 days uptime...  
however, it is an example of a machine with mpt0 and em0 sharing an IRQ so  
I might be able to trigger it using Dane's method.

$ vmstat -i
interrupt                          total       rate
irq1: atkbd0                         392          0
irq6: fdc0                             9          0
irq14: ata0                           34          0
irq18: em0 mpt0               1189748491        218
cpu0: timer                   2174263198        400
Total                         3364012124        619


I'm doing my best to get you guys the info you need, but this is one heck  
of a Heisenbug...