From owner-freebsd-hackers@freebsd.org Thu Jul 12 14:51:52 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC96E10413DC for ; Thu, 12 Jul 2018 14:51:52 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 64D7D9578C for ; Thu, 12 Jul 2018 14:51:52 +0000 (UTC) (envelope-from ler@lerctr.org) Received: by mailman.ysv.freebsd.org (Postfix) id 2610B10413DB; Thu, 12 Jul 2018 14:51:52 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03F6910413D7 for ; Thu, 12 Jul 2018 14:51:52 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:bb:dcff:fe50:d900]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.lerctr.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B5E595783; Thu, 12 Jul 2018 14:51:51 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=pEv3fyjQwacH6q1Ux8QLsvdHwogNG5UMmHxd3iVdqPo=; b=Ypee5oXYm4UxrlZj6c/q+Jzoqi wASb1Zp82iQXp5ZY/soOEPcjuS/GNrRq2OK2voa/R9E+7ipLX+OHiVBCIPo8xWuwJz8hkdI6FV8WW wiX5nbOiA1dVXC8dIe+Ev+YapY89MaHRmM3briFlmkRiEbt6ovHtGGp3SVsfEQeceDWQ=; Received: from [2600:1700:210:b18f:5d16:74d5:75bf:c686] (port=49314 helo=ler-imac.local) by thebighonker.lerctr.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fdcwf-000Dg8-S0; Thu, 12 Jul 2018 09:51:49 -0500 Date: Thu, 12 Jul 2018 09:51:49 -0500 From: Larry Rosenman To: Alan Somers Cc: Mark Johnston , Alexander Leidinger , "freebsd-hackers@freebsd.org" Subject: Re: crashinfo doesn't support compressed crashdumps - which way forward? Message-ID: <20180712145149.a6rt7kzaobze65gv@ler-imac.local> References: <20180712141409.Horde.NVSzLsvqE-2PCSDqhhfb0Xd@webmail.leidinger.net> <20180712143118.GA15892@raichu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="o7sdm5hpvbycbezs" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180622 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2018 14:51:53 -0000 --o7sdm5hpvbycbezs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 12, 2018 at 08:50:08AM -0600, Alan Somers wrote: > On Thu, Jul 12, 2018 at 8:31 AM, Mark Johnston wrote: >=20 > > On Thu, Jul 12, 2018 at 02:14:09PM +0200, Alexander Leidinger wrote: > > > Hi, > > > > > > the crashinfo script doesn't know how to handle compressed coredumps. > > > What would be acceptable behavior (ordered by my preferrence)? > > > 1) decompress in /var/crash and then proceed normally (already > > > implemented locally) > > > 2) decompress to CRASHTMPDIR:/var/tmp/xxx and delete when finished > > > 3) keep it like it is > > > 4) teach tools to understand compressed dumps (gzip / zstd) > > >=20 > I briefly looked into this option, but it's harder than it sounds. There > are several libraries that would need to be modified, and I think some of > them work by mmap(2)ping the core file rather than by fread(2)ing it. I > don't know of anyway to mmap a compressed file. >=20 >=20 > > > > > > Implicitly there is the question what what is the purpose of > > > compressing crashdumps, to have more RAM than space in dumpdev (which > > > is valid in my case), or to save space in /var/crash (which I don't > > > care much about). > > >=20 > Compressed crashdumps are also great for systems with slow dumpdevs. They > greatly speed up the dumping process. >=20 >=20 > > > > I think jhb has a patch which implements 2). I do not have strong > > feelings on which is the right way forward, but I mildly prefer 2) to > > 1). It looks like crashinfo can be disabled in rc.conf, so users who > > are space-constrained in /var/crash can take the additional step of > > setting crashinfo_enable=3DNO. FWIW, when I committed the compression > > support, my use-case involved both a small dump device and a small > > /var filesystem. I have jhb's patch applied and it works great. borg.lerctr.org /usr/src $ svn diff Index: usr.sbin/crashinfo/crashinfo.sh =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- usr.sbin/crashinfo/crashinfo.sh (revision 336184) +++ usr.sbin/crashinfo/crashinfo.sh (working copy) @@ -38,6 +38,13 @@ exit 1 } +# Remove an uncompressed copy of a dump +cleanup() +{ + + [ -e $VMCORE ] && rm -f $VMCORE +} + # Find a gdb binary to use and save the value in GDB. find_gdb() { @@ -133,7 +140,7 @@ # Figure out the crash directory and number from the vmcore name. CRASHDIR=3D`dirname $1` - DUMPNR=3D$(expr $(basename $1) : 'vmcore\.\([0-9]*\)$') + DUMPNR=3D$(expr $(basename $1) : 'vmcore\.\([0-9]*\)') if [ -z "$DUMPNR" ]; then echo "Unable to determine dump number from vmcore file $1." exit 1 @@ -174,8 +181,16 @@ fi if [ ! -e $VMCORE ]; then - echo "$VMCORE not found" - exit 1 + if [ -e $VMCORE.gz ]; then + trap cleanup EXIT HUP INT QUIT TERM + gzcat $VMCORE.gz > $VMCORE + elif [ -e $VMCORE.zst ]; then + trap cleanup EXIT HUP INT QUIT TERM + zstdcat $VMCORE.zst > $VMCORE + else + echo "$VMCORE not found" + exit 1 + fi fi if [ ! -e $INFO ]; then borg.lerctr.org /usr/src $ --=20 Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Drive, Round Rock, TX 78665-2106 --o7sdm5hpvbycbezs Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQHABAABCgCqFiEEHjgknedhWzvJgwVzaXyZsatIp30FAltHawUsFIAAAAAAFQAO cGthLWFkZHJlc3NAZ251cGcub3JnbGVyQGxlcmN0ci5vcmdfFIAAAAAALgAoaXNz dWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDFFMzgy NDlERTc2MTVCM0JDOTgzMDU3MzY5N0M5OUIxQUI0OEE3N0QACgkQaXyZsatIp32m MQf/ZVSsnAqswOAvjWXSjnvwBszsjDm/mvnd1UbTX0cFlEMkcSCLF9csjThegEL1 ElTdV6iv5GhMYZ8mnvVyQMCUtm01ekK9HjdhgTgtljHGINBOUI/wkJI/s6BS049A 9LM4EbGKbx2SoPNfHs8KgAWbAfnQF6UaKMTbnK4FzSYj2tcBtTKvQqXpEns2hvmx IOtgLMa9Hdsr1RpkQzN9Vsq6DqQSOwN8aFBReSQtHUV1/nduYnFKw7Hw/Do/1hTz re3WHsvFA6b3vdlBHrDsws55qmJNTcdoA5GoQNqtaDCCLTWojWIw2IPv/GGj3eT8 tUZv5FiOhN9SEyKceYrK2mi14A== =YRvO -----END PGP SIGNATURE----- --o7sdm5hpvbycbezs--