From owner-freebsd-hackers@FreeBSD.ORG Tue Nov 5 18:23:24 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 160F5BA; Tue, 5 Nov 2013 18:23:24 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E3A022C31; Tue, 5 Nov 2013 18:23:23 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 026B3B988; Tue, 5 Nov 2013 13:23:23 -0500 (EST) From: John Baldwin To: freebsd-current@freebsd.org Subject: Re: Automated submission of kernel panic reports: sysutils/panicmail Date: Tue, 5 Nov 2013 12:27:14 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <527779ED.9040303@freebsd.org> <527797D8.5040404@freebsd.org> <527811B7.5090102@freebsd.org> In-Reply-To: <527811B7.5090102@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201311051227.14157.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 05 Nov 2013 13:23:23 -0500 (EST) Cc: FreeBSD Hackers , Alfred Perlstein , Jordan Hubbard , Colin Percival X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2013 18:23:24 -0000 On Monday, November 04, 2013 4:29:27 pm Colin Percival wrote: > On 11/04/13 04:49, Alfred Perlstein wrote: > > Colin, have you had a few minutes to check out the crash reporting facilities in > > FreeNAS? > > Yes. > > > The reason I ask is that: > > > > 1) we would like to share code. > > 2) we have this running for a few months now and have a huge corpus of information. > > 3) we are building a nice UI (screenshots attached) over it, we have a couple of > > thousands of lines of code we can share for this. > > Once I have a useful number of panics collected, I was hoping to take the best > pieces from FreeNAS's processing, from the SoC project, and from the processing > I've been doing of automatic panic reports from EC2 instances. > > > We send a minimal set of information: kernel stack trace, ddb buffer and > > hardware. Just enough to get some very, very handy stuff. > > I'm currently sending the dump header and what I get from kgdb 'bt'. If I find > that I'm missing something important, I can always add it to a new version of > the panicmail port. ;-) One of my previous employers maintained a database of panics and I added ways to recognize "known panics" and tag them. I ended up relying a lot on stack trace details from specific OS versions to mark a panic as an instance of a specific bug. Also, you may have very different stack traces even on the same build version for a single bug. In the case of my employer we had a constrained set of kernel configs and specific build versions to work with. It might be harder to correctly match panics in the wild what with patched trees and random kernel configs. -- John Baldwin