Date: Tue, 20 Feb 2007 07:21:39 -0800 From: John Utz <john@utzweb.net> To: Robert Watson <rwatson@FreeBSD.org> Cc: current@FreeBSD.org Subject: Re: sbflush() panics, new socket-related DDB commands (was: cvs commit: src/sys/conf files src/sys/kern uipc_debug.c (fwd)) Message-ID: <20070220072139.78d85ca5@minime.local> In-Reply-To: <20070215095248.P55760@fledge.watson.org> References: <20070215095248.P55760@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 15 Feb 2007 10:12:59 +0000 (GMT) Robert Watson <rwatson@FreeBSD.org> wrote: > > Dear All, > > Of late we've seen an increase in the number of reports of panics in > sbflush(). These occur due to an invariant check when closing > sockets, which in effect confirms that the amount of actual data in a > socket buffer matches the socket buffer's cached length from that > data. Typically, these panics are a symptom of mbuf chain/mbuf > packet queue corruption, and in the past have been symptomatic > primarily of device driver bugs (i.e., the device driver touches mbuf > fields after passing the mbuf into the stack, etc). However, > debugging these problems is very difficult, as sbflush() is called > only when the socket is closed, and the corruption frequently occurs > much earlier. It's possibly worth pointing out that i can get either the same or a remarkably similar panic 100% reproducibly by using usbaudio. As soon as mpg321 finishes the song and closes the device i get a panic. 100% reproducible. the panic does *not* occur if i use the snd_ich stuff instead of usbaudio. i last synced at the following: minime# uname -a FreeBSD minime.local 7.0-CURRENT FreeBSD 7.0-CURRENT #20: Mon Feb 12 07:49:10 PST 2007 spaz@minime.local:/usr/src/sys/i386/compile/MINIME i386 I havent reported it previously because i thought it was my own little weird problem and i've been attempting to get a more specific failure point beyond 'device close' > If you are experiencing these panics, please compile your kernel with > "options SOCKBUF_DEBUG", which adds more frequent invariant checks of > socket buffer consistency. These checks may have a substantial > performance impact, but unless we can catch the problem(s) closer to > when it(they) occur(s), the chances of tracking the source are low. > We could be looking at one or more bugs in some combination of the > socket, protocol, and network interface code leading to a recent > increase. Perhaps socket buffer auto-sizing has increased the > chances of it occuring, or perhaps we just have increase parallelism > in test hardware. Dont think it's network code because it get's tickled by closing usbaudio. > I've added some new commands to DDB to make it easier to understand > the state of the system following panic. While the same information > can be extracted with kgdb, core dumps are not reliably available in > all environments, and certain other information (such as lock > information) is most easily obtained using DDB. The commands are: > > show socket <addr> > show sockbuf <addr> > show protosw <addr> > show domain <addr> > > Only the first two of these are likely to be used directly, and they > invoke protosw and domain printing as required. It would be helpful > if people debugging network panics involving sockets could use the > above (and especially show socket) on socket arguments in stack > traces. For example, in an sbflush() panic, the first argument to > soclose() in the stack trace is the socket pointer, so is ideal to > pass to "show socket" :-). > > FYI, in general, the first argument to sofoo() functions is the > socket pointer. Likewise, the first argument to sbfoo() functions is > the socket buffer pointer. This is not the case of soo_foo() > functions, which take a file descriptor, or filt_sofoo() functions, > which accept a knote. In general, if you print the socket contents > with "show socket", then it isn't necessary to print socket buffers > separately; "show socket" prints not just the contents of the socket > buffers, but also the pointers to them so that those pointers can be > compared with other arguments in the stack. I will endeavor to resync and rebuild and see if i can provide a more specific failure point with the usb stuff, but i will not be able to tackle it before the weekend because this laptop is my only computer (and i run -current on it, suprisingly well for the most part!) > Robert N M Watson > Computer Laboratory > University of Cambridge > > ---------- Forwarded message ---------- > Date: Thu, 15 Feb 2007 01:28:23 +0000 (UTC) > From: Robert Watson <rwatson@FreeBSD.org> > To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, > cvs-all@FreeBSD.org Subject: cvs commit: src/sys/conf files > src/sys/kern uipc_debug.c > > rwatson 2007-02-15 01:28:22 UTC > > FreeBSD src repository > > Modified files: > sys/conf files > Added files: > sys/kern uipc_debug.c > Log: > Teach DDB how to print sockets, socket buffers, protosw's, and > domain structures given pointers to them. > > Revision Changes Path > 1.1177 +1 -0 src/sys/conf/files > 1.1 +530 -0 src/sys/kern/uipc_debug.c (new) > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to > "freebsd-current-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070220072139.78d85ca5>