From owner-freebsd-hackers@freebsd.org Thu Jul 12 16:09:06 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51EA610484B5 for ; Thu, 12 Jul 2018 16:09:06 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DD9B870B9B for ; Thu, 12 Jul 2018 16:09:05 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id A193410484B1; Thu, 12 Jul 2018 16:09:05 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A14310484AC for ; Thu, 12 Jul 2018 16:09:05 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F1C270B97; Thu, 12 Jul 2018 16:09:05 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-2.local (ralph.baldwin.cx [66.234.199.215]) by mail.baldwin.cx (Postfix) with ESMTPSA id A1F5210A87D; Thu, 12 Jul 2018 12:09:03 -0400 (EDT) Subject: Re: crashinfo doesn't support compressed crashdumps - which way forward? To: Mark Johnston , Alexander Leidinger References: <20180712141409.Horde.NVSzLsvqE-2PCSDqhhfb0Xd@webmail.leidinger.net> <20180712143118.GA15892@raichu> Cc: hackers@freebsd.org From: John Baldwin Message-ID: <6858738d-7eef-cc0f-10bd-900fb822fecd@FreeBSD.org> Date: Thu, 12 Jul 2018 09:09:02 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <20180712143118.GA15892@raichu> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Thu, 12 Jul 2018 12:09:04 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2018 16:09:06 -0000 On 7/12/18 7:31 AM, Mark Johnston wrote: > On Thu, Jul 12, 2018 at 02:14:09PM +0200, Alexander Leidinger wrote: >> Hi, >> >> the crashinfo script doesn't know how to handle compressed coredumps. >> What would be acceptable behavior (ordered by my preferrence)? >> 1) decompress in /var/crash and then proceed normally (already >> implemented locally) >> 2) decompress to CRASHTMPDIR:/var/tmp/xxx and delete when finished >> 3) keep it like it is >> 4) teach tools to understand compressed dumps (gzip / zstd) >> >> Implicitly there is the question what what is the purpose of >> compressing crashdumps, to have more RAM than space in dumpdev (which >> is valid in my case), or to save space in /var/crash (which I don't >> care much about). > > I think jhb has a patch which implements 2). I do not have strong > feelings on which is the right way forward, but I mildly prefer 2) to > 1). It looks like crashinfo can be disabled in rc.conf, so users who > are space-constrained in /var/crash can take the additional step of > setting crashinfo_enable=NO. FWIW, when I committed the compression > support, my use-case involved both a small dump device and a small > /var filesystem. Yes, here's the patch for 2 which has been tested. 4) is pretty hard to do in practice as you have to basically decompress into RAM when reading the core to do anything useful as opposed to a format that only compressed certain parts (e.g. if the page tables were not compressed only the payload in a minidump, and if it were compressed on some kind of block boundaries so that you could locate a given block and decompress it when reading specific data). Coming up with such a format would be more useful but requires more work. Index: usr.sbin/crashinfo/crashinfo.sh =================================================================== --- usr.sbin/crashinfo/crashinfo.sh (revision 335896) +++ usr.sbin/crashinfo/crashinfo.sh (working copy) @@ -38,6 +38,13 @@ exit 1 } +# Remove an uncompressed copy of a dump +cleanup() +{ + + [ -e $VMCORE ] && rm -f $VMCORE +} + # Find a gdb binary to use and save the value in GDB. find_gdb() { @@ -133,7 +140,7 @@ # Figure out the crash directory and number from the vmcore name. CRASHDIR=`dirname $1` - DUMPNR=$(expr $(basename $1) : 'vmcore\.\([0-9]*\)$') + DUMPNR=$(expr $(basename $1) : 'vmcore\.\([0-9]*\)') if [ -z "$DUMPNR" ]; then echo "Unable to determine dump number from vmcore file $1." exit 1 @@ -174,8 +181,16 @@ fi if [ ! -e $VMCORE ]; then - echo "$VMCORE not found" - exit 1 + if [ -e $VMCORE.gz ]; then + trap cleanup EXIT HUP INT QUIT TERM + gzcat $VMCORE.gz > $VMCORE + elif [ -e $VMCORE.zst ]; then + trap cleanup EXIT HUP INT QUIT TERM + zstdcat $VMCORE.zst > $VMCORE + else + echo "$VMCORE not found" + exit 1 + fi fi if [ ! -e $INFO ]; then -- John Baldwin