From owner-freebsd-current@FreeBSD.ORG Thu Feb 15 10:13:00 2007 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B000F16A401 for ; Thu, 15 Feb 2007 10:13:00 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 666FB13C461 for ; Thu, 15 Feb 2007 10:13:00 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 165B547F3F for ; Thu, 15 Feb 2007 05:13:00 -0500 (EST) Date: Thu, 15 Feb 2007 10:12:59 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: current@FreeBSD.org Message-ID: <20070215095248.P55760@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: sbflush() panics, new socket-related DDB commands (was: cvs commit: src/sys/conf files src/sys/kern uipc_debug.c (fwd)) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 10:13:00 -0000 Dear All, Of late we've seen an increase in the number of reports of panics in sbflush(). These occur due to an invariant check when closing sockets, which in effect confirms that the amount of actual data in a socket buffer matches the socket buffer's cached length from that data. Typically, these panics are a symptom of mbuf chain/mbuf packet queue corruption, and in the past have been symptomatic primarily of device driver bugs (i.e., the device driver touches mbuf fields after passing the mbuf into the stack, etc). However, debugging these problems is very difficult, as sbflush() is called only when the socket is closed, and the corruption frequently occurs much earlier. If you are experiencing these panics, please compile your kernel with "options SOCKBUF_DEBUG", which adds more frequent invariant checks of socket buffer consistency. These checks may have a substantial performance impact, but unless we can catch the problem(s) closer to when it(they) occur(s), the chances of tracking the source are low. We could be looking at one or more bugs in some combination of the socket, protocol, and network interface code leading to a recent increase. Perhaps socket buffer auto-sizing has increased the chances of it occuring, or perhaps we just have increase parallelism in test hardware. I've added some new commands to DDB to make it easier to understand the state of the system following panic. While the same information can be extracted with kgdb, core dumps are not reliably available in all environments, and certain other information (such as lock information) is most easily obtained using DDB. The commands are: show socket show sockbuf show protosw show domain Only the first two of these are likely to be used directly, and they invoke protosw and domain printing as required. It would be helpful if people debugging network panics involving sockets could use the above (and especially show socket) on socket arguments in stack traces. For example, in an sbflush() panic, the first argument to soclose() in the stack trace is the socket pointer, so is ideal to pass to "show socket" :-). FYI, in general, the first argument to sofoo() functions is the socket pointer. Likewise, the first argument to sbfoo() functions is the socket buffer pointer. This is not the case of soo_foo() functions, which take a file descriptor, or filt_sofoo() functions, which accept a knote. In general, if you print the socket contents with "show socket", then it isn't necessary to print socket buffers separately; "show socket" prints not just the contents of the socket buffers, but also the pointers to them so that those pointers can be compared with other arguments in the stack. Robert N M Watson Computer Laboratory University of Cambridge ---------- Forwarded message ---------- Date: Thu, 15 Feb 2007 01:28:23 +0000 (UTC) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/conf files src/sys/kern uipc_debug.c rwatson 2007-02-15 01:28:22 UTC FreeBSD src repository Modified files: sys/conf files Added files: sys/kern uipc_debug.c Log: Teach DDB how to print sockets, socket buffers, protosw's, and domain structures given pointers to them. Revision Changes Path 1.1177 +1 -0 src/sys/conf/files 1.1 +530 -0 src/sys/kern/uipc_debug.c (new)