From owner-freebsd-current@FreeBSD.ORG Mon Nov 4 12:49:24 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 521468F5; Mon, 4 Nov 2013 12:49:24 +0000 (UTC) (envelope-from alfred@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 048852351; Mon, 4 Nov 2013 12:49:24 +0000 (UTC) Received: from Alfreds-MacBook-Air.local (c-76-21-10-192.hsd1.ca.comcast.net [76.21.10.192]) by elvis.mu.org (Postfix) with ESMTPSA id C463E1A3C19; Mon, 4 Nov 2013 04:49:23 -0800 (PST) Message-ID: <527797D8.5040404@freebsd.org> Date: Mon, 04 Nov 2013 04:49:28 -0800 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: Colin Percival , FreeBSD Hackers , FreeBSD current , Jordan Hubbard Subject: Re: Automated submission of kernel panic reports: sysutils/panicmail References: <527779ED.9040303@freebsd.org> In-Reply-To: <527779ED.9040303@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Nov 2013 12:49:24 -0000 On 11/4/13, 2:41 AM, Colin Percival wrote: > Hi all, > > After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have > now added sysutils/panicmail to the FreeBSD ports tree. If you install this > and add > panicmail_enable="YES" > to your /etc/rc.conf, a panic report will be generated and sent to root@ for > you to review and submit (via email). You can skip the reviewing step and > submit panics automatically by setting panicmail_autosubmit="YES". > > The panics submitted are encrypted to an RSA key which I hold in order to keep > them secure in transit; and I intend to keep the raw panic reports confidential > except to the minimum extent necessary for other developers to help me process > the incoming reports. > > If I receive enough panic reports to be useful, I hope to provide developers > with aggregate statistics. This may include: > > * regular email reports listing the "top panics", to help guide developers > towards the most fertile areas for stability improvements; > > * email to specific developers alerting them to recurring panics in code they > maintain (especially if it becomes clear that the panic has been recently > introduced); and > > * guidance to re@ and secteam@ about how often a particular panic occurs if > an errata notice is being considered > > as well as other yet-to-be-imagined reports of a similarly aggregate and > anonymized nature. > > So please install the sysutils/panicmail port and enable it in rc.conf! This > all depends on getting useful data, and I can't do that without your help. > Colin, have you had a few minutes to check out the crash reporting facilities in FreeNAS? The reason I ask is that: 1) we would like to share code. 2) we have this running for a few months now and have a huge corpus of information. 3) we are building a nice UI (screenshots attached) over it, we have a couple of thousands of lines of code we can share for this. Our scripts can be found here: 1) A startup script that sends us the crashes on system start: https://github.com/freenas/freenas/blob/master/nanobsd/Files/etc/rc.d/ix_textdump 2) A script to submit data at boot OR from command line that sends more comprehensive system information "ixdiagnose": https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/ixdiagnose 3) A very simple script to upload that report: https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/crashuploader We send a minimal set of information: kernel stack trace, ddb buffer and hardware. Just enough to get some very, very handy stuff. I can share with you offline the crash server code, it's django and relatively straight forward. The screenshots can also be seen at: http://people.freebsd.org/~alfred/crashreporter/ We could modify our framework for FreeBSD to do so by checking for a sentinel file depending on the host type and only auto-sending if we see that. -Alfred