From owner-freebsd-current@FreeBSD.ORG Thu Dec 20 13:32:53 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9CDF816A418 for ; Thu, 20 Dec 2007 13:32:53 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.185]) by mx1.freebsd.org (Postfix) with ESMTP id EABA613C465 for ; Thu, 20 Dec 2007 13:32:52 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so1951565nfb.33 for ; Thu, 20 Dec 2007 05:32:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:message-id; bh=6xE7+coxtivywsrg/Du8HyBEPLVYKzkoMposCWiY9Vs=; b=VMWX2YY4Y8nBGIWsX9lcsbV4Fggzf2ENAIRLFlr28FDssx/RMOfFh2gEqyYteiPF0yw7y5VsGaTgl+/OR+SueviBsLUxCm00l/r/3SC5iyXSlUTTNiYXtebYQQykFVD/onznJkE1YD+SRaRX+q8wnnQUsL42GdknGkGbnsLfI5g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:message-id; b=l6ahJQMcwNG/sGF57GXD9rvkZ7/sEFpCouIn5whXyvo1ttz8MxjzMgRL0ZoDy41iWIQbACnAdeF2UsiiVHEgAXy29ApapY/ASGaUt8Y6nZ9zBgT/h1cKpeDQhmxRmPYYgr9BO7Xb8yIhTtISve8udopboSWnDqdwXwO7weiEDJE= Received: by 10.78.160.4 with SMTP id i4mr10057108hue.35.1198157569269; Thu, 20 Dec 2007 05:32:49 -0800 (PST) Received: from orion ( [89.162.141.1]) by mx.google.com with ESMTPS id 2sm186084nfv.2007.12.20.05.32.45 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 20 Dec 2007 05:32:47 -0800 (PST) From: Nikolay Pavlov To: freebsd-current@freebsd.org Date: Thu, 20 Dec 2007 15:32:40 +0200 User-Agent: KMail/1.9.6 (enterprise 0.20070907.709405) References: <20071218120359.E15521@fledge.watson.org> In-Reply-To: <20071218120359.E15521@fledge.watson.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1211070.4xZmMiWbSL"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200712201532.41123.qpadla@gmail.com> Cc: arch@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: DDB scripting, output capture, and textdumps X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: qpadla@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 13:32:53 -0000 --nextPart1211070.4xZmMiWbSL Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Tuesday 18 December 2007 14:10:46 Robert Watson wrote: > Dear all: > > I've been hacking on-and-off for a while on a side project to improve > our kernel debugging facilities. Primarily, my concern has been to > address three problems: > > - The complications of employing kernel core dumps for debugging, > including the large size of dumps making them unwieldy to distribute > or store for any extended period (even with minidumps), the requirement > to have relatively synchronized kernel source in order to use the dumps, > the need to have a kernel with debugging symbols, and the problems with > fsck causing sufficient swap use to invalidate dumps before they can be > extracted. > > - The decreasing likelihood that notebooks will ship with serial ports > that can be used for interactive debugging using DDB. Making > end-users type in stack traces is cruel, photos are a pain, and X11 > rules out both. > > - The fact that a great many problems are most easily diagnosed using > utility routines present in DDB, but not as easily using kgdb for > offline analysis. I find that for many bugs I analyze, simply > looking at the DDB output is sufficient to identify the source of the > problem. > > An idea I punted around a bit at BSDCan earlier this year (or perhaps it > was at EuroBSDCon the previous year) was an idea of a "textdump" -- that > is, a new type of kernel dump based on capturing automatically extracted > debugging information generated by DDB. The result would be an ASCII > text file that could be filed as a bug report, perhaps even > automatically. > > To this end, I have implemented three new facilities for use with DDB: > > (1) DDB output capture. The output of DDB is stored in a memory buffer, > and can be extracted using a sysctl or textdumps (see below). This > can be turned on and off, both for use manually ("I'll want this > later, but not that") and as part of scripts (see below). > > (2) DDB scripting. A limited number of named scripts can be defined to > run a series of DDB commands. No loops, etc, just simple command > lists. These can be caused to run automatically on entering DDB > for various scenarios, including WITNESS violations and kernel panics. > They can also be run by hand in order to save a bit of typing if you use > DDB in a repetitive way (as I do). > > (3) Textdumps. A new dump type that stores a series of data files > containing various pieces of information, including the DDB capture > buffer, kernel message buffer, kernel configuration (if compiled > into the kernel), panic message, and kernel version string. These are > stored in the ustar format inside the dump partition (aligned to the > end) so can be easily extended, and savecore(8) requires almost no new > logic to deal with them (it just drops numbered tar files in > /var/crash). This makes it straight forward to extend the textdump > format to include new types of information and avoids the issue of how > to safely simultaneously represent information in many different formats > in the same file. > > These are pretty flexible tools, and you can imagine doing the following > sorts of things: > > - Setting the kdb.enter.panic script to automatically turn on output > capture, do full backtraces of all threads, show open file > information, dump UMA stats, and save it all to a textdump and then > reboot. > > - Setting the kdb.enter.witness script to show lock information, > generate a coredump, and reboot. Or, just to automatically do "show > allocks" and drop to the DDB prompt. > > - Adding a flag to rc.conf to automatically submit textdumps via e-mail > to a specific address, perhaps including GNATS or an automated bug > system. These could be unpacked and automatically analyzed, and do to > the compact size, kept for long-term trend analysis or to identify when > a problem started occuring. > > I've produced an initial snapshot of the above, which can be found here: > > http://www.watson.org/~robert/freebsd/20071218-ddb.tgz > > This adds three files to DDB, patches quite a few kernel files (to pass > more information into KDB about why it's being entered, in order to > trigger the right script), enhancements to savecore(8) to know how to > extract textdumps, adds a ddb(8) command line tool so that userspace can > manage DDB scripts from outside the debugger, extensions to the ddb(4) > man page, and a new textdump(4) man page. > > There are a number of known limitations; I've tried to document them at > the top of the pertinent files where I am aware of them. I also regret > to say that to date I've been able to test only on i386, and not other > platforms. I'd welcome any feedback -- I'd like to get these changes > into CVS in the next week or two. It looks like some files is not included in the patch. I have this error: make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP=3D"cc -E" CC=3D"= cc"=20 xargs=20 mkdep -a -f .newdep -O -pipe -std=3Dc99 -g -Wall -Wredundant-decls -Wneste= d-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winlin= e -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I.= -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter= -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ng= atm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys= /gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERN= EL_OPTION_HEADERS -include=20 opt_global.h -fno-common -finline-limit=3D8000 --param=20 inline-unit-growth=3D100 --param=20 large-function-growth=3D1000 -mno-align-long-strings -mpreferred-stack-bou= ndary=3D2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding cc: /usr/src/sys/ddb/db_capture.c: No such file or directory cc: /usr/src/sys/ddb/db_script.c: No such file or directory cc: /usr/src/sys/ddb/db_textdump.c: No such file or directory mkdep: compile failed *** Error code 1 Stop in /usr/obj/usr/src/sys/GENERIC. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. root@orion-vm:/usr/src# ls -la /usr/src/sys/ddb/ = = =20 total 424 drwxr-xr-x 2 root wheel 1024 Dec 19 16:49 ./ drwxr-xr-x 53 root wheel 1024 Oct 29 08:24 ../ =2Drw-r--r-- 1 root wheel 2591 Dec 4 2005 db_access.c =2Drw-r--r-- 1 root wheel 1431 Jan 6 2005 db_access.h =2Drw-r--r-- 1 root wheel 7737 Jan 6 2005 db_break.c =2Drw-r--r-- 1 root wheel 2098 Jan 6 2005 db_break.h =2Drw-r--r-- 1 root wheel 16579 Dec 19 16:49 db_command.c =2Drw-r--r-- 1 root wheel 15705 Jan 17 2007 db_command.c.orig =2Drw-r--r-- 1 root wheel 1633 Dec 19 16:49 db_command.h =2Drw-r--r-- 1 root wheel 1588 Jan 6 2005 db_command.h.orig =2Drw-r--r-- 1 root wheel 7270 Oct 27 20:19 db_examine.c =2Drw-r--r-- 1 root wheel 4811 Jan 6 2005 db_expr.c =2Drw-r--r-- 1 root wheel 7981 Dec 19 16:49 db_input.c =2Drw-r--r-- 1 root wheel 7931 Jan 6 2005 db_input.c.orig =2Drw-r--r-- 1 root wheel 5978 Dec 19 16:49 db_lex.c =2Drw-r--r-- 1 root wheel 5304 Jan 6 2005 db_lex.c.orig =2Drw-r--r-- 1 root wheel 1951 Dec 19 16:49 db_lex.h =2Drw-r--r-- 1 root wheel 1861 Jan 6 2005 db_lex.h.orig =2Drw-r--r-- 1 root wheel 5976 Dec 19 16:49 db_main.c =2Drw-r--r-- 1 root wheel 5787 Nov 6 2006 db_main.c.orig =2Drw-r--r-- 1 root wheel 6889 Dec 19 16:49 db_output.c =2Drw-r--r-- 1 root wheel 6639 Oct 10 2006 db_output.c.orig =2Drw-r--r-- 1 root wheel 1444 Oct 8 2006 db_output.h =2Drw-r--r-- 1 root wheel 2013 Apr 14 2005 db_print.c =2Drw-r--r-- 1 root wheel 10956 Nov 13 13:43 db_ps.c =2Drw-r--r-- 1 root wheel 8810 Apr 14 2005 db_run.c =2Drw-r--r-- 1 root wheel 7815 Jun 16 2006 db_sym.c =2Drw-r--r-- 1 root wheel 3591 Jan 6 2005 db_sym.h =2Drw-r--r-- 1 root wheel 5143 Jan 17 2007 db_thread.c =2Drw-r--r-- 1 root wheel 3402 Jan 6 2005 db_variables.c =2Drw-r--r-- 1 root wheel 1854 Jan 6 2005 db_variables.h =2Drw-r--r-- 1 root wheel 7218 Nov 17 2006 db_watch.c =2Drw-r--r-- 1 root wheel 1509 Jan 6 2005 db_watch.h =2Drw-r--r-- 1 root wheel 2264 Jan 6 2005 db_write_cmd.c =2Drw-r--r-- 1 root wheel 7467 Dec 19 16:49 ddb.h =2Drw-r--r-- 1 root wheel 5776 Jul 12 2006 ddb.h.orig =2D-=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 =2D Best regards, Nikolay Pavlov. <<<----------------------------------- = =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 --nextPart1211070.4xZmMiWbSL Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQBHam75/2R6KvEYGaIRAo+GAJ996lCyHp0+dyN5rW7dWbBRugjQFgCg39e1 +bfdaDTErGPe2aJWZCW0BTw= =CE18 -----END PGP SIGNATURE----- --nextPart1211070.4xZmMiWbSL--