Date: Thu, 20 Dec 2007 15:32:40 +0200 From: Nikolay Pavlov <qpadla@gmail.com> To: freebsd-current@freebsd.org Cc: arch@freebsd.org, Robert Watson <rwatson@freebsd.org>, current@freebsd.org Subject: Re: DDB scripting, output capture, and textdumps Message-ID: <200712201532.41123.qpadla@gmail.com> In-Reply-To: <20071218120359.E15521@fledge.watson.org> References: <20071218120359.E15521@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--nextPart1211070.4xZmMiWbSL
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
On Tuesday 18 December 2007 14:10:46 Robert Watson wrote:
> Dear all:
>
> I've been hacking on-and-off for a while on a side project to improve
> our kernel debugging facilities. Primarily, my concern has been to
> address three problems:
>
> - The complications of employing kernel core dumps for debugging,
> including the large size of dumps making them unwieldy to distribute
> or store for any extended period (even with minidumps), the requirement
> to have relatively synchronized kernel source in order to use the dumps,
> the need to have a kernel with debugging symbols, and the problems with
> fsck causing sufficient swap use to invalidate dumps before they can be
> extracted.
>
> - The decreasing likelihood that notebooks will ship with serial ports
> that can be used for interactive debugging using DDB. Making
> end-users type in stack traces is cruel, photos are a pain, and X11
> rules out both.
>
> - The fact that a great many problems are most easily diagnosed using
> utility routines present in DDB, but not as easily using kgdb for
> offline analysis. I find that for many bugs I analyze, simply
> looking at the DDB output is sufficient to identify the source of the
> problem.
>
> An idea I punted around a bit at BSDCan earlier this year (or perhaps it
> was at EuroBSDCon the previous year) was an idea of a "textdump" -- that
> is, a new type of kernel dump based on capturing automatically extracted
> debugging information generated by DDB. The result would be an ASCII
> text file that could be filed as a bug report, perhaps even
> automatically.
>
> To this end, I have implemented three new facilities for use with DDB:
>
> (1) DDB output capture. The output of DDB is stored in a memory buffer,
> and can be extracted using a sysctl or textdumps (see below). This
> can be turned on and off, both for use manually ("I'll want this
> later, but not that") and as part of scripts (see below).
>
> (2) DDB scripting. A limited number of named scripts can be defined to
> run a series of DDB commands. No loops, etc, just simple command
> lists. These can be caused to run automatically on entering DDB
> for various scenarios, including WITNESS violations and kernel panics.
> They can also be run by hand in order to save a bit of typing if you use
> DDB in a repetitive way (as I do).
>
> (3) Textdumps. A new dump type that stores a series of data files
> containing various pieces of information, including the DDB capture
> buffer, kernel message buffer, kernel configuration (if compiled
> into the kernel), panic message, and kernel version string. These are
> stored in the ustar format inside the dump partition (aligned to the
> end) so can be easily extended, and savecore(8) requires almost no new
> logic to deal with them (it just drops numbered tar files in
> /var/crash). This makes it straight forward to extend the textdump
> format to include new types of information and avoids the issue of how
> to safely simultaneously represent information in many different formats
> in the same file.
>
> These are pretty flexible tools, and you can imagine doing the following
> sorts of things:
>
> - Setting the kdb.enter.panic script to automatically turn on output
> capture, do full backtraces of all threads, show open file
> information, dump UMA stats, and save it all to a textdump and then
> reboot.
>
> - Setting the kdb.enter.witness script to show lock information,
> generate a coredump, and reboot. Or, just to automatically do "show
> allocks" and drop to the DDB prompt.
>
> - Adding a flag to rc.conf to automatically submit textdumps via e-mail
> to a specific address, perhaps including GNATS or an automated bug
> system. These could be unpacked and automatically analyzed, and do to
> the compact size, kept for long-term trend analysis or to identify when
> a problem started occuring.
>
> I've produced an initial snapshot of the above, which can be found here:
>
> http://www.watson.org/~robert/freebsd/20071218-ddb.tgz
>
> This adds three files to DDB, patches quite a few kernel files (to pass
> more information into KDB about why it's being entered, in order to
> trigger the right script), enhancements to savecore(8) to know how to
> extract textdumps, adds a ddb(8) command line tool so that userspace can
> manage DDB scripts from outside the debugger, extensions to the ddb(4)
> man page, and a new textdump(4) man page.
>
> There are a number of known limitations; I've tried to document them at
> the top of the pertinent files where I am aware of them. I also regret
> to say that to date I've been able to test only on i386, and not other
> platforms. I'd welcome any feedback -- I'd like to get these changes
> into CVS in the next week or two.
It looks like some files is not included in the patch.
I have this error:
make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP=3D"cc -E" CC=3D"=
cc"=20
xargs=20
mkdep -a -f .newdep -O -pipe -std=3Dc99 -g -Wall -Wredundant-decls -Wneste=
d-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winlin=
e -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I.=
-I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter=
-I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ng=
atm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys=
/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERN=
EL_OPTION_HEADERS -include=20
opt_global.h -fno-common -finline-limit=3D8000 --param=20
inline-unit-growth=3D100 --param=20
large-function-growth=3D1000 -mno-align-long-strings -mpreferred-stack-bou=
ndary=3D2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding
cc: /usr/src/sys/ddb/db_capture.c: No such file or directory
cc: /usr/src/sys/ddb/db_script.c: No such file or directory
cc: /usr/src/sys/ddb/db_textdump.c: No such file or directory
mkdep: compile failed
*** Error code 1
Stop in /usr/obj/usr/src/sys/GENERIC.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
root@orion-vm:/usr/src# ls -la /usr/src/sys/ddb/ =
=
=20
total 424
drwxr-xr-x 2 root wheel 1024 Dec 19 16:49 ./
drwxr-xr-x 53 root wheel 1024 Oct 29 08:24 ../
=2Drw-r--r-- 1 root wheel 2591 Dec 4 2005 db_access.c
=2Drw-r--r-- 1 root wheel 1431 Jan 6 2005 db_access.h
=2Drw-r--r-- 1 root wheel 7737 Jan 6 2005 db_break.c
=2Drw-r--r-- 1 root wheel 2098 Jan 6 2005 db_break.h
=2Drw-r--r-- 1 root wheel 16579 Dec 19 16:49 db_command.c
=2Drw-r--r-- 1 root wheel 15705 Jan 17 2007 db_command.c.orig
=2Drw-r--r-- 1 root wheel 1633 Dec 19 16:49 db_command.h
=2Drw-r--r-- 1 root wheel 1588 Jan 6 2005 db_command.h.orig
=2Drw-r--r-- 1 root wheel 7270 Oct 27 20:19 db_examine.c
=2Drw-r--r-- 1 root wheel 4811 Jan 6 2005 db_expr.c
=2Drw-r--r-- 1 root wheel 7981 Dec 19 16:49 db_input.c
=2Drw-r--r-- 1 root wheel 7931 Jan 6 2005 db_input.c.orig
=2Drw-r--r-- 1 root wheel 5978 Dec 19 16:49 db_lex.c
=2Drw-r--r-- 1 root wheel 5304 Jan 6 2005 db_lex.c.orig
=2Drw-r--r-- 1 root wheel 1951 Dec 19 16:49 db_lex.h
=2Drw-r--r-- 1 root wheel 1861 Jan 6 2005 db_lex.h.orig
=2Drw-r--r-- 1 root wheel 5976 Dec 19 16:49 db_main.c
=2Drw-r--r-- 1 root wheel 5787 Nov 6 2006 db_main.c.orig
=2Drw-r--r-- 1 root wheel 6889 Dec 19 16:49 db_output.c
=2Drw-r--r-- 1 root wheel 6639 Oct 10 2006 db_output.c.orig
=2Drw-r--r-- 1 root wheel 1444 Oct 8 2006 db_output.h
=2Drw-r--r-- 1 root wheel 2013 Apr 14 2005 db_print.c
=2Drw-r--r-- 1 root wheel 10956 Nov 13 13:43 db_ps.c
=2Drw-r--r-- 1 root wheel 8810 Apr 14 2005 db_run.c
=2Drw-r--r-- 1 root wheel 7815 Jun 16 2006 db_sym.c
=2Drw-r--r-- 1 root wheel 3591 Jan 6 2005 db_sym.h
=2Drw-r--r-- 1 root wheel 5143 Jan 17 2007 db_thread.c
=2Drw-r--r-- 1 root wheel 3402 Jan 6 2005 db_variables.c
=2Drw-r--r-- 1 root wheel 1854 Jan 6 2005 db_variables.h
=2Drw-r--r-- 1 root wheel 7218 Nov 17 2006 db_watch.c
=2Drw-r--r-- 1 root wheel 1509 Jan 6 2005 db_watch.h
=2Drw-r--r-- 1 root wheel 2264 Jan 6 2005 db_write_cmd.c
=2Drw-r--r-- 1 root wheel 7467 Dec 19 16:49 ddb.h
=2Drw-r--r-- 1 root wheel 5776 Jul 12 2006 ddb.h.orig
=2D-=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20
=2D Best regards, Nikolay Pavlov. <<<----------------------------------- =
=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20
--nextPart1211070.4xZmMiWbSL
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQBHam75/2R6KvEYGaIRAo+GAJ996lCyHp0+dyN5rW7dWbBRugjQFgCg39e1
+bfdaDTErGPe2aJWZCW0BTw=
=CE18
-----END PGP SIGNATURE-----
--nextPart1211070.4xZmMiWbSL--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200712201532.41123.qpadla>